Jul 28 15:33:37.999654 ip-10-0-131-121 systemd[1]: Starting Kubernetes Kubelet...
Jul 28 15:33:40.076868 ip-10-0-131-121 kubenswrapper[2568]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 28 15:33:40.076868 ip-10-0-131-121 kubenswrapper[2568]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Jul 28 15:33:40.076868 ip-10-0-131-121 kubenswrapper[2568]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 28 15:33:40.076868 ip-10-0-131-121 kubenswrapper[2568]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Jul 28 15:33:40.076868 ip-10-0-131-121 kubenswrapper[2568]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 28 15:33:40.077527 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:33:40.076950    2568 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Jul 28 15:33:40.083713 ip-10-0-131-121 kubenswrapper[2568]: W0728 15:33:40.083682    2568 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 28 15:33:40.083713 ip-10-0-131-121 kubenswrapper[2568]: W0728 15:33:40.083706    2568 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 28 15:33:40.083713 ip-10-0-131-121 kubenswrapper[2568]: W0728 15:33:40.083711    2568 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 28 15:33:40.083713 ip-10-0-131-121 kubenswrapper[2568]: W0728 15:33:40.083715    2568 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 28 15:33:40.083713 ip-10-0-131-121 kubenswrapper[2568]: W0728 15:33:40.083719    2568 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 28 15:33:40.084047 ip-10-0-131-121 kubenswrapper[2568]: W0728 15:33:40.083723    2568 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 28 15:33:40.084047 ip-10-0-131-121 kubenswrapper[2568]: W0728 15:33:40.083727    2568 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 28 15:33:40.084047 ip-10-0-131-121 kubenswrapper[2568]: W0728 15:33:40.083731    2568 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 28 15:33:40.084047 ip-10-0-131-121 kubenswrapper[2568]: W0728 15:33:40.083735    2568 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 28 15:33:40.084047 ip-10-0-131-121 kubenswrapper[2568]: W0728 15:33:40.083738    2568 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 28 15:33:40.084047 ip-10-0-131-121 kubenswrapper[2568]: W0728 15:33:40.083742    2568 feature_gate.go:328] unrecognized feature gate: Example2
Jul 28 15:33:40.084047 ip-10-0-131-121 kubenswrapper[2568]: W0728 15:33:40.083746    2568 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 28 15:33:40.084047 ip-10-0-131-121 kubenswrapper[2568]: W0728 15:33:40.083749    2568 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 28 15:33:40.084047 ip-10-0-131-121 kubenswrapper[2568]: W0728 15:33:40.083753    2568 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 28 15:33:40.084047 ip-10-0-131-121 kubenswrapper[2568]: W0728 15:33:40.083756    2568 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 28 15:33:40.084047 ip-10-0-131-121 kubenswrapper[2568]: W0728 15:33:40.083759    2568 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 28 15:33:40.084047 ip-10-0-131-121 kubenswrapper[2568]: W0728 15:33:40.083763    2568 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 28 15:33:40.084047 ip-10-0-131-121 kubenswrapper[2568]: W0728 15:33:40.083767    2568 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 28 15:33:40.084047 ip-10-0-131-121 kubenswrapper[2568]: W0728 15:33:40.083770    2568 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 28 15:33:40.084047 ip-10-0-131-121 kubenswrapper[2568]: W0728 15:33:40.083801    2568 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 28 15:33:40.084047 ip-10-0-131-121 kubenswrapper[2568]: W0728 15:33:40.083805    2568 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 28 15:33:40.084047 ip-10-0-131-121 kubenswrapper[2568]: W0728 15:33:40.083809    2568 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 28 15:33:40.084047 ip-10-0-131-121 kubenswrapper[2568]: W0728 15:33:40.083813    2568 feature_gate.go:328] unrecognized feature gate: Example
Jul 28 15:33:40.084047 ip-10-0-131-121 kubenswrapper[2568]: W0728 15:33:40.083816    2568 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 28 15:33:40.084047 ip-10-0-131-121 kubenswrapper[2568]: W0728 15:33:40.083820    2568 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 28 15:33:40.084996 ip-10-0-131-121 kubenswrapper[2568]: W0728 15:33:40.083824    2568 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 28 15:33:40.084996 ip-10-0-131-121 kubenswrapper[2568]: W0728 15:33:40.083828    2568 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 28 15:33:40.084996 ip-10-0-131-121 kubenswrapper[2568]: W0728 15:33:40.083834    2568 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 28 15:33:40.084996 ip-10-0-131-121 kubenswrapper[2568]: W0728 15:33:40.083840    2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 28 15:33:40.084996 ip-10-0-131-121 kubenswrapper[2568]: W0728 15:33:40.083844    2568 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 28 15:33:40.084996 ip-10-0-131-121 kubenswrapper[2568]: W0728 15:33:40.083848    2568 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 28 15:33:40.084996 ip-10-0-131-121 kubenswrapper[2568]: W0728 15:33:40.083852    2568 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 28 15:33:40.084996 ip-10-0-131-121 kubenswrapper[2568]: W0728 15:33:40.083856    2568 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 28 15:33:40.084996 ip-10-0-131-121 kubenswrapper[2568]: W0728 15:33:40.083860    2568 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 28 15:33:40.084996 ip-10-0-131-121 kubenswrapper[2568]: W0728 15:33:40.083864    2568 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 28 15:33:40.084996 ip-10-0-131-121 kubenswrapper[2568]: W0728 15:33:40.083868    2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 28 15:33:40.084996 ip-10-0-131-121 kubenswrapper[2568]: W0728 15:33:40.083872    2568 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 28 15:33:40.084996 ip-10-0-131-121 kubenswrapper[2568]: W0728 15:33:40.083876    2568 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 28 15:33:40.084996 ip-10-0-131-121 kubenswrapper[2568]: W0728 15:33:40.083881    2568 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 28 15:33:40.084996 ip-10-0-131-121 kubenswrapper[2568]: W0728 15:33:40.083885    2568 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 28 15:33:40.084996 ip-10-0-131-121 kubenswrapper[2568]: W0728 15:33:40.083889    2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 28 15:33:40.084996 ip-10-0-131-121 kubenswrapper[2568]: W0728 15:33:40.083893    2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 28 15:33:40.084996 ip-10-0-131-121 kubenswrapper[2568]: W0728 15:33:40.083897    2568 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 28 15:33:40.084996 ip-10-0-131-121 kubenswrapper[2568]: W0728 15:33:40.083902    2568 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 28 15:33:40.085828 ip-10-0-131-121 kubenswrapper[2568]: W0728 15:33:40.083906    2568 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 28 15:33:40.085828 ip-10-0-131-121 kubenswrapper[2568]: W0728 15:33:40.083911    2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 28 15:33:40.085828 ip-10-0-131-121 kubenswrapper[2568]: W0728 15:33:40.083916    2568 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 28 15:33:40.085828 ip-10-0-131-121 kubenswrapper[2568]: W0728 15:33:40.083920    2568 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 28 15:33:40.085828 ip-10-0-131-121 kubenswrapper[2568]: W0728 15:33:40.083924    2568 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 28 15:33:40.085828 ip-10-0-131-121 kubenswrapper[2568]: W0728 15:33:40.083929    2568 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 28 15:33:40.085828 ip-10-0-131-121 kubenswrapper[2568]: W0728 15:33:40.083935    2568 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 28 15:33:40.085828 ip-10-0-131-121 kubenswrapper[2568]: W0728 15:33:40.083939    2568 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 28 15:33:40.085828 ip-10-0-131-121 kubenswrapper[2568]: W0728 15:33:40.083943    2568 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 28 15:33:40.085828 ip-10-0-131-121 kubenswrapper[2568]: W0728 15:33:40.083948    2568 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 28 15:33:40.085828 ip-10-0-131-121 kubenswrapper[2568]: W0728 15:33:40.083955    2568 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 28 15:33:40.085828 ip-10-0-131-121 kubenswrapper[2568]: W0728 15:33:40.083961    2568 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 28 15:33:40.085828 ip-10-0-131-121 kubenswrapper[2568]: W0728 15:33:40.083968    2568 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 28 15:33:40.085828 ip-10-0-131-121 kubenswrapper[2568]: W0728 15:33:40.083973    2568 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 28 15:33:40.085828 ip-10-0-131-121 kubenswrapper[2568]: W0728 15:33:40.083978    2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 28 15:33:40.085828 ip-10-0-131-121 kubenswrapper[2568]: W0728 15:33:40.083983    2568 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 28 15:33:40.085828 ip-10-0-131-121 kubenswrapper[2568]: W0728 15:33:40.083988    2568 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 28 15:33:40.085828 ip-10-0-131-121 kubenswrapper[2568]: W0728 15:33:40.083994    2568 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 28 15:33:40.085828 ip-10-0-131-121 kubenswrapper[2568]: W0728 15:33:40.083998    2568 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 28 15:33:40.086625 ip-10-0-131-121 kubenswrapper[2568]: W0728 15:33:40.084003    2568 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 28 15:33:40.086625 ip-10-0-131-121 kubenswrapper[2568]: W0728 15:33:40.084007    2568 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 28 15:33:40.086625 ip-10-0-131-121 kubenswrapper[2568]: W0728 15:33:40.084012    2568 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 28 15:33:40.086625 ip-10-0-131-121 kubenswrapper[2568]: W0728 15:33:40.084016    2568 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 28 15:33:40.086625 ip-10-0-131-121 kubenswrapper[2568]: W0728 15:33:40.084021    2568 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 28 15:33:40.086625 ip-10-0-131-121 kubenswrapper[2568]: W0728 15:33:40.084025    2568 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 28 15:33:40.086625 ip-10-0-131-121 kubenswrapper[2568]: W0728 15:33:40.084031    2568 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 28 15:33:40.086625 ip-10-0-131-121 kubenswrapper[2568]: W0728 15:33:40.084035    2568 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 28 15:33:40.086625 ip-10-0-131-121 kubenswrapper[2568]: W0728 15:33:40.084039    2568 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 28 15:33:40.086625 ip-10-0-131-121 kubenswrapper[2568]: W0728 15:33:40.084044    2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 28 15:33:40.086625 ip-10-0-131-121 kubenswrapper[2568]: W0728 15:33:40.084049    2568 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 28 15:33:40.086625 ip-10-0-131-121 kubenswrapper[2568]: W0728 15:33:40.084053    2568 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 28 15:33:40.086625 ip-10-0-131-121 kubenswrapper[2568]: W0728 15:33:40.084057    2568 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 28 15:33:40.086625 ip-10-0-131-121 kubenswrapper[2568]: W0728 15:33:40.084061    2568 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 28 15:33:40.086625 ip-10-0-131-121 kubenswrapper[2568]: W0728 15:33:40.084066    2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 28 15:33:40.086625 ip-10-0-131-121 kubenswrapper[2568]: W0728 15:33:40.084071    2568 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 28 15:33:40.086625 ip-10-0-131-121 kubenswrapper[2568]: W0728 15:33:40.084075    2568 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 28 15:33:40.086625 ip-10-0-131-121 kubenswrapper[2568]: W0728 15:33:40.084080    2568 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 28 15:33:40.086625 ip-10-0-131-121 kubenswrapper[2568]: W0728 15:33:40.084084    2568 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 28 15:33:40.086625 ip-10-0-131-121 kubenswrapper[2568]: W0728 15:33:40.084088    2568 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 28 15:33:40.087495 ip-10-0-131-121 kubenswrapper[2568]: W0728 15:33:40.084095    2568 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 28 15:33:40.087495 ip-10-0-131-121 kubenswrapper[2568]: W0728 15:33:40.084100    2568 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 28 15:33:40.087495 ip-10-0-131-121 kubenswrapper[2568]: W0728 15:33:40.084104    2568 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 28 15:33:40.087495 ip-10-0-131-121 kubenswrapper[2568]: W0728 15:33:40.084109    2568 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 28 15:33:40.087495 ip-10-0-131-121 kubenswrapper[2568]: W0728 15:33:40.084113    2568 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 28 15:33:40.087495 ip-10-0-131-121 kubenswrapper[2568]: W0728 15:33:40.084117    2568 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 28 15:33:40.087495 ip-10-0-131-121 kubenswrapper[2568]: W0728 15:33:40.084121    2568 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 28 15:33:40.087495 ip-10-0-131-121 kubenswrapper[2568]: W0728 15:33:40.084129    2568 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 28 15:33:40.087495 ip-10-0-131-121 kubenswrapper[2568]: W0728 15:33:40.084133    2568 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 28 15:33:40.087495 ip-10-0-131-121 kubenswrapper[2568]: W0728 15:33:40.084137    2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 28 15:33:40.087495 ip-10-0-131-121 kubenswrapper[2568]: W0728 15:33:40.084142    2568 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 28 15:33:40.087495 ip-10-0-131-121 kubenswrapper[2568]: W0728 15:33:40.084146    2568 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 28 15:33:40.087495 ip-10-0-131-121 kubenswrapper[2568]: W0728 15:33:40.084151    2568 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 28 15:33:40.087495 ip-10-0-131-121 kubenswrapper[2568]: W0728 15:33:40.084155    2568 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 28 15:33:40.087495 ip-10-0-131-121 kubenswrapper[2568]: W0728 15:33:40.084159    2568 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 28 15:33:40.087495 ip-10-0-131-121 kubenswrapper[2568]: W0728 15:33:40.084163    2568 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 28 15:33:40.087495 ip-10-0-131-121 kubenswrapper[2568]: W0728 15:33:40.085361    2568 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 28 15:33:40.087495 ip-10-0-131-121 kubenswrapper[2568]: W0728 15:33:40.085372    2568 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 28 15:33:40.087495 ip-10-0-131-121 kubenswrapper[2568]: W0728 15:33:40.085377    2568 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 28 15:33:40.088395 ip-10-0-131-121 kubenswrapper[2568]: W0728 15:33:40.085381    2568 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 28 15:33:40.088395 ip-10-0-131-121 kubenswrapper[2568]: W0728 15:33:40.085385    2568 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 28 15:33:40.088395 ip-10-0-131-121 kubenswrapper[2568]: W0728 15:33:40.085390    2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 28 15:33:40.088395 ip-10-0-131-121 kubenswrapper[2568]: W0728 15:33:40.085394    2568 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 28 15:33:40.088395 ip-10-0-131-121 kubenswrapper[2568]: W0728 15:33:40.085400    2568 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 28 15:33:40.088395 ip-10-0-131-121 kubenswrapper[2568]: W0728 15:33:40.085405    2568 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 28 15:33:40.088395 ip-10-0-131-121 kubenswrapper[2568]: W0728 15:33:40.085409    2568 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 28 15:33:40.088395 ip-10-0-131-121 kubenswrapper[2568]: W0728 15:33:40.085414    2568 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 28 15:33:40.088395 ip-10-0-131-121 kubenswrapper[2568]: W0728 15:33:40.085418    2568 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 28 15:33:40.088395 ip-10-0-131-121 kubenswrapper[2568]: W0728 15:33:40.085422    2568 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 28 15:33:40.088395 ip-10-0-131-121 kubenswrapper[2568]: W0728 15:33:40.085426    2568 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 28 15:33:40.088395 ip-10-0-131-121 kubenswrapper[2568]: W0728 15:33:40.085431    2568 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 28 15:33:40.088395 ip-10-0-131-121 kubenswrapper[2568]: W0728 15:33:40.085435    2568 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 28 15:33:40.088395 ip-10-0-131-121 kubenswrapper[2568]: W0728 15:33:40.085439    2568 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 28 15:33:40.088395 ip-10-0-131-121 kubenswrapper[2568]: W0728 15:33:40.085446    2568 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 28 15:33:40.088395 ip-10-0-131-121 kubenswrapper[2568]: W0728 15:33:40.085450    2568 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 28 15:33:40.088395 ip-10-0-131-121 kubenswrapper[2568]: W0728 15:33:40.085457    2568 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 28 15:33:40.088395 ip-10-0-131-121 kubenswrapper[2568]: W0728 15:33:40.085463    2568 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 28 15:33:40.088395 ip-10-0-131-121 kubenswrapper[2568]: W0728 15:33:40.085467    2568 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 28 15:33:40.088963 ip-10-0-131-121 kubenswrapper[2568]: W0728 15:33:40.085472    2568 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 28 15:33:40.088963 ip-10-0-131-121 kubenswrapper[2568]: W0728 15:33:40.085480    2568 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 28 15:33:40.088963 ip-10-0-131-121 kubenswrapper[2568]: W0728 15:33:40.085485    2568 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 28 15:33:40.088963 ip-10-0-131-121 kubenswrapper[2568]: W0728 15:33:40.085490    2568 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 28 15:33:40.088963 ip-10-0-131-121 kubenswrapper[2568]: W0728 15:33:40.085495    2568 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 28 15:33:40.088963 ip-10-0-131-121 kubenswrapper[2568]: W0728 15:33:40.085499    2568 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 28 15:33:40.088963 ip-10-0-131-121 kubenswrapper[2568]: W0728 15:33:40.085504    2568 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 28 15:33:40.088963 ip-10-0-131-121 kubenswrapper[2568]: W0728 15:33:40.085508    2568 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 28 15:33:40.088963 ip-10-0-131-121 kubenswrapper[2568]: W0728 15:33:40.085512    2568 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 28 15:33:40.088963 ip-10-0-131-121 kubenswrapper[2568]: W0728 15:33:40.085516    2568 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 28 15:33:40.088963 ip-10-0-131-121 kubenswrapper[2568]: W0728 15:33:40.085521    2568 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 28 15:33:40.088963 ip-10-0-131-121 kubenswrapper[2568]: W0728 15:33:40.085525    2568 feature_gate.go:328] unrecognized feature gate: Example2
Jul 28 15:33:40.088963 ip-10-0-131-121 kubenswrapper[2568]: W0728 15:33:40.085529    2568 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 28 15:33:40.088963 ip-10-0-131-121 kubenswrapper[2568]: W0728 15:33:40.085534    2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 28 15:33:40.088963 ip-10-0-131-121 kubenswrapper[2568]: W0728 15:33:40.085538    2568 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 28 15:33:40.088963 ip-10-0-131-121 kubenswrapper[2568]: W0728 15:33:40.085542    2568 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 28 15:33:40.088963 ip-10-0-131-121 kubenswrapper[2568]: W0728 15:33:40.085546    2568 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 28 15:33:40.088963 ip-10-0-131-121 kubenswrapper[2568]: W0728 15:33:40.085550    2568 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 28 15:33:40.088963 ip-10-0-131-121 kubenswrapper[2568]: W0728 15:33:40.085557    2568 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 28 15:33:40.089522 ip-10-0-131-121 kubenswrapper[2568]: W0728 15:33:40.085563    2568 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 28 15:33:40.089522 ip-10-0-131-121 kubenswrapper[2568]: W0728 15:33:40.085570    2568 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 28 15:33:40.089522 ip-10-0-131-121 kubenswrapper[2568]: W0728 15:33:40.085575    2568 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 28 15:33:40.089522 ip-10-0-131-121 kubenswrapper[2568]: W0728 15:33:40.085581    2568 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 28 15:33:40.089522 ip-10-0-131-121 kubenswrapper[2568]: W0728 15:33:40.085585    2568 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 28 15:33:40.089522 ip-10-0-131-121 kubenswrapper[2568]: W0728 15:33:40.085590    2568 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 28 15:33:40.089522 ip-10-0-131-121 kubenswrapper[2568]: W0728 15:33:40.085595    2568 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 28 15:33:40.089522 ip-10-0-131-121 kubenswrapper[2568]: W0728 15:33:40.085601    2568 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 28 15:33:40.089522 ip-10-0-131-121 kubenswrapper[2568]: W0728 15:33:40.085605    2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 28 15:33:40.089522 ip-10-0-131-121 kubenswrapper[2568]: W0728 15:33:40.085611    2568 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 28 15:33:40.089522 ip-10-0-131-121 kubenswrapper[2568]: W0728 15:33:40.085615    2568 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 28 15:33:40.089522 ip-10-0-131-121 kubenswrapper[2568]: W0728 15:33:40.085620    2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 28 15:33:40.089522 ip-10-0-131-121 kubenswrapper[2568]: W0728 15:33:40.085624    2568 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 28 15:33:40.089522 ip-10-0-131-121 kubenswrapper[2568]: W0728 15:33:40.085628    2568 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 28 15:33:40.089522 ip-10-0-131-121 kubenswrapper[2568]: W0728 15:33:40.085634    2568 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 28 15:33:40.089522 ip-10-0-131-121 kubenswrapper[2568]: W0728 15:33:40.085638    2568 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 28 15:33:40.089522 ip-10-0-131-121 kubenswrapper[2568]: W0728 15:33:40.085643    2568 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 28 15:33:40.089522 ip-10-0-131-121 kubenswrapper[2568]: W0728 15:33:40.085648    2568 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 28 15:33:40.089522 ip-10-0-131-121 kubenswrapper[2568]: W0728 15:33:40.085652    2568 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 28 15:33:40.090172 ip-10-0-131-121 kubenswrapper[2568]: W0728 15:33:40.085657    2568 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 28 15:33:40.090172 ip-10-0-131-121 kubenswrapper[2568]: W0728 15:33:40.085661    2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 28 15:33:40.090172 ip-10-0-131-121 kubenswrapper[2568]: W0728 15:33:40.085665    2568 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 28 15:33:40.090172 ip-10-0-131-121 kubenswrapper[2568]: W0728 15:33:40.085670    2568 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 28 15:33:40.090172 ip-10-0-131-121 kubenswrapper[2568]: W0728 15:33:40.085674    2568 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 28 15:33:40.090172 ip-10-0-131-121 kubenswrapper[2568]: W0728 15:33:40.085678    2568 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 28 15:33:40.090172 ip-10-0-131-121 kubenswrapper[2568]: W0728 15:33:40.085682    2568 feature_gate.go:328] unrecognized feature gate: Example
Jul 28 15:33:40.090172 ip-10-0-131-121 kubenswrapper[2568]: W0728 15:33:40.085687    2568 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 28 15:33:40.090172 ip-10-0-131-121 kubenswrapper[2568]: W0728 15:33:40.085691    2568 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 28 15:33:40.090172 ip-10-0-131-121 kubenswrapper[2568]: W0728 15:33:40.085696    2568 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 28 15:33:40.090172 ip-10-0-131-121 kubenswrapper[2568]: W0728 15:33:40.085700    2568 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 28 15:33:40.090172 ip-10-0-131-121 kubenswrapper[2568]: W0728 15:33:40.085704    2568 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 28 15:33:40.090172 ip-10-0-131-121 kubenswrapper[2568]: W0728 15:33:40.085710    2568 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 28 15:33:40.090172 ip-10-0-131-121 kubenswrapper[2568]: W0728 15:33:40.085714    2568 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 28 15:33:40.090172 ip-10-0-131-121 kubenswrapper[2568]: W0728 15:33:40.085719    2568 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 28 15:33:40.090172 ip-10-0-131-121 kubenswrapper[2568]: W0728 15:33:40.085722    2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 28 15:33:40.090172 ip-10-0-131-121 kubenswrapper[2568]: W0728 15:33:40.085727    2568 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 28 15:33:40.090172 ip-10-0-131-121 kubenswrapper[2568]: W0728 15:33:40.085731    2568 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 28 15:33:40.090172 ip-10-0-131-121 kubenswrapper[2568]: W0728 15:33:40.085735    2568 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 28 15:33:40.090172 ip-10-0-131-121 kubenswrapper[2568]: W0728 15:33:40.085739    2568 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 28 15:33:40.090740 ip-10-0-131-121 kubenswrapper[2568]: W0728 15:33:40.085744    2568 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 28 15:33:40.090740 ip-10-0-131-121 kubenswrapper[2568]: W0728 15:33:40.085748    2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 28 15:33:40.090740 ip-10-0-131-121 kubenswrapper[2568]: W0728 15:33:40.085752    2568 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 28 15:33:40.090740 ip-10-0-131-121 kubenswrapper[2568]: W0728 15:33:40.085757    2568 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 28 15:33:40.090740 ip-10-0-131-121 kubenswrapper[2568]: W0728 15:33:40.085761    2568 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 28 15:33:40.090740 ip-10-0-131-121 kubenswrapper[2568]: W0728 15:33:40.085765    2568 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 28 15:33:40.090740 ip-10-0-131-121 kubenswrapper[2568]: W0728 15:33:40.085769    2568 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 28 15:33:40.090740 ip-10-0-131-121 kubenswrapper[2568]: W0728 15:33:40.085791    2568 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 28 15:33:40.090740 ip-10-0-131-121 kubenswrapper[2568]: W0728 15:33:40.085795    2568 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 28 15:33:40.090740 ip-10-0-131-121 kubenswrapper[2568]: W0728 15:33:40.085800    2568 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 28 15:33:40.090740 ip-10-0-131-121 kubenswrapper[2568]: W0728 15:33:40.085805    2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 28 15:33:40.090740 ip-10-0-131-121 kubenswrapper[2568]: W0728 15:33:40.085810    2568 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 28 15:33:40.090740 ip-10-0-131-121 kubenswrapper[2568]: W0728 15:33:40.085814    2568 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 28 15:33:40.090740 ip-10-0-131-121 kubenswrapper[2568]: W0728 15:33:40.085818    2568 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 28 15:33:40.090740 ip-10-0-131-121 kubenswrapper[2568]: W0728 15:33:40.085822    2568 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 28 15:33:40.090740 ip-10-0-131-121 kubenswrapper[2568]: W0728 15:33:40.085826    2568 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 28 15:33:40.090740 ip-10-0-131-121 kubenswrapper[2568]: W0728 15:33:40.085831    2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 28 15:33:40.090740 ip-10-0-131-121 kubenswrapper[2568]: W0728 15:33:40.085834    2568 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 28 15:33:40.090740 ip-10-0-131-121 kubenswrapper[2568]: W0728 15:33:40.085839    2568 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 28 15:33:40.090740 ip-10-0-131-121 kubenswrapper[2568]: W0728 15:33:40.086455    2568 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 28 15:33:40.091302 ip-10-0-131-121 kubenswrapper[2568]: W0728 15:33:40.086463    2568 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 28 15:33:40.091302 ip-10-0-131-121 kubenswrapper[2568]: W0728 15:33:40.086468    2568 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 28 15:33:40.091302 ip-10-0-131-121 kubenswrapper[2568]: W0728 15:33:40.086472    2568 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 28 15:33:40.091302 ip-10-0-131-121 kubenswrapper[2568]: W0728 15:33:40.086476    2568 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 28 15:33:40.091302 ip-10-0-131-121 kubenswrapper[2568]: W0728 15:33:40.086480    2568 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 28 15:33:40.091302 ip-10-0-131-121 kubenswrapper[2568]: W0728 15:33:40.086485    2568 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 28 15:33:40.091302 ip-10-0-131-121 kubenswrapper[2568]: W0728 15:33:40.086489    2568 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 28 15:33:40.091302 ip-10-0-131-121 kubenswrapper[2568]: W0728 15:33:40.086494    2568 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 28 15:33:40.091302 ip-10-0-131-121 kubenswrapper[2568]: W0728 15:33:40.086498    2568 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 28 15:33:40.091302 ip-10-0-131-121 kubenswrapper[2568]: W0728 15:33:40.086502    2568 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 28 15:33:40.091302 ip-10-0-131-121 kubenswrapper[2568]: W0728 15:33:40.086506    2568 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 28 15:33:40.091302 ip-10-0-131-121 kubenswrapper[2568]: W0728 15:33:40.086511    2568 feature_gate.go:328] unrecognized feature gate: Example2
Jul 28 15:33:40.091302 ip-10-0-131-121 kubenswrapper[2568]: W0728 15:33:40.086516    2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 28 15:33:40.091302 ip-10-0-131-121 kubenswrapper[2568]: W0728 15:33:40.086520    2568 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 28 15:33:40.091302 ip-10-0-131-121 kubenswrapper[2568]: W0728 15:33:40.086524    2568 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 28 15:33:40.091302 ip-10-0-131-121 kubenswrapper[2568]: W0728 15:33:40.086530    2568 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 28 15:33:40.091302 ip-10-0-131-121 kubenswrapper[2568]: W0728 15:33:40.086535    2568 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 28 15:33:40.091302 ip-10-0-131-121 kubenswrapper[2568]: W0728 15:33:40.086539    2568 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 28 15:33:40.091302 ip-10-0-131-121 kubenswrapper[2568]: W0728 15:33:40.086543    2568 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 28 15:33:40.091302 ip-10-0-131-121 kubenswrapper[2568]: W0728 15:33:40.086550    2568 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 28 15:33:40.091869 ip-10-0-131-121 kubenswrapper[2568]: W0728 15:33:40.086554    2568 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 28 15:33:40.091869 ip-10-0-131-121 kubenswrapper[2568]: W0728 15:33:40.086560    2568 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 28 15:33:40.091869 ip-10-0-131-121 kubenswrapper[2568]: W0728 15:33:40.086564    2568 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 28 15:33:40.091869 ip-10-0-131-121 kubenswrapper[2568]: W0728 15:33:40.086569    2568 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 28 15:33:40.091869 ip-10-0-131-121 kubenswrapper[2568]: W0728 15:33:40.086573    2568 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 28 15:33:40.091869 ip-10-0-131-121 kubenswrapper[2568]: W0728 15:33:40.086580    2568 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 28 15:33:40.091869 ip-10-0-131-121 kubenswrapper[2568]: W0728 15:33:40.086586    2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 28 15:33:40.091869 ip-10-0-131-121 kubenswrapper[2568]: W0728 15:33:40.086590    2568 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 28 15:33:40.091869 ip-10-0-131-121 kubenswrapper[2568]: W0728 15:33:40.086595    2568 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 28 15:33:40.091869 ip-10-0-131-121 kubenswrapper[2568]: W0728 15:33:40.086599    2568 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 28 15:33:40.091869 ip-10-0-131-121 kubenswrapper[2568]: W0728 15:33:40.086603    2568 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 28 15:33:40.091869 ip-10-0-131-121 kubenswrapper[2568]: W0728 15:33:40.086607    2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 28 15:33:40.091869 ip-10-0-131-121 kubenswrapper[2568]: W0728 15:33:40.086612    2568 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 28 15:33:40.091869 ip-10-0-131-121 kubenswrapper[2568]: W0728 15:33:40.086616    2568 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 28 15:33:40.091869 ip-10-0-131-121 kubenswrapper[2568]: W0728 15:33:40.086620    2568 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 28 15:33:40.091869 ip-10-0-131-121 kubenswrapper[2568]: W0728 15:33:40.086624    2568 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 28 15:33:40.091869 ip-10-0-131-121 kubenswrapper[2568]: W0728 15:33:40.086628    2568 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 28 15:33:40.091869 ip-10-0-131-121 kubenswrapper[2568]: W0728 15:33:40.086632    2568 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 28 15:33:40.091869 ip-10-0-131-121 kubenswrapper[2568]: W0728 15:33:40.086636    2568 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 28 15:33:40.091869 ip-10-0-131-121 kubenswrapper[2568]: W0728 15:33:40.086640    2568 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 28 15:33:40.092368 ip-10-0-131-121 kubenswrapper[2568]: W0728 15:33:40.086644    2568 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 28 15:33:40.092368 ip-10-0-131-121 kubenswrapper[2568]: W0728 15:33:40.086648    2568 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 28 15:33:40.092368 ip-10-0-131-121 kubenswrapper[2568]: W0728 15:33:40.086652    2568 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 28 15:33:40.092368 ip-10-0-131-121 kubenswrapper[2568]: W0728 15:33:40.086656    2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 28 15:33:40.092368 ip-10-0-131-121 kubenswrapper[2568]: W0728 15:33:40.086660    2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 28 15:33:40.092368 ip-10-0-131-121 kubenswrapper[2568]: W0728 15:33:40.086665    2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 28 15:33:40.092368 ip-10-0-131-121 kubenswrapper[2568]: W0728 15:33:40.086669    2568 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 28 15:33:40.092368 ip-10-0-131-121 kubenswrapper[2568]: W0728 15:33:40.086673    2568 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 28 15:33:40.092368 ip-10-0-131-121 kubenswrapper[2568]: W0728 15:33:40.086680    2568 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 28 15:33:40.092368 ip-10-0-131-121 kubenswrapper[2568]: W0728 15:33:40.086685    2568 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 28 15:33:40.092368 ip-10-0-131-121 kubenswrapper[2568]: W0728 15:33:40.086689    2568 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 28 15:33:40.092368 ip-10-0-131-121 kubenswrapper[2568]: W0728 15:33:40.086695    2568 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 28 15:33:40.092368 ip-10-0-131-121 kubenswrapper[2568]: W0728 15:33:40.086699    2568 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 28 15:33:40.092368 ip-10-0-131-121 kubenswrapper[2568]: W0728 15:33:40.086706    2568 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 28 15:33:40.092368 ip-10-0-131-121 kubenswrapper[2568]: W0728 15:33:40.086711    2568 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 28 15:33:40.092368 ip-10-0-131-121 kubenswrapper[2568]: W0728 15:33:40.086715    2568 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 28 15:33:40.092368 ip-10-0-131-121 kubenswrapper[2568]: W0728 15:33:40.086720    2568 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 28 15:33:40.092368 ip-10-0-131-121 kubenswrapper[2568]: W0728 15:33:40.086724    2568 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 28 15:33:40.092368 ip-10-0-131-121 kubenswrapper[2568]: W0728 15:33:40.086731    2568 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 28 15:33:40.092902 ip-10-0-131-121 kubenswrapper[2568]: W0728 15:33:40.086736    2568 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 28 15:33:40.092902 ip-10-0-131-121 kubenswrapper[2568]: W0728 15:33:40.086741    2568 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 28 15:33:40.092902 ip-10-0-131-121 kubenswrapper[2568]: W0728 15:33:40.086746    2568 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 28 15:33:40.092902 ip-10-0-131-121 kubenswrapper[2568]: W0728 15:33:40.086753    2568 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 28 15:33:40.092902 ip-10-0-131-121 kubenswrapper[2568]: W0728 15:33:40.086758    2568 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 28 15:33:40.092902 ip-10-0-131-121 kubenswrapper[2568]: W0728 15:33:40.086762    2568 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 28 15:33:40.092902 ip-10-0-131-121 kubenswrapper[2568]: W0728 15:33:40.086789    2568 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 28 15:33:40.092902 ip-10-0-131-121 kubenswrapper[2568]: W0728 15:33:40.086793    2568 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 28 15:33:40.092902 ip-10-0-131-121 kubenswrapper[2568]: W0728 15:33:40.086798    2568 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 28 15:33:40.092902 ip-10-0-131-121 kubenswrapper[2568]: W0728 15:33:40.086802    2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 28 15:33:40.092902 ip-10-0-131-121 kubenswrapper[2568]: W0728 15:33:40.086806    2568 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 28 15:33:40.092902 ip-10-0-131-121 kubenswrapper[2568]: W0728 15:33:40.086810    2568 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 28 15:33:40.092902 ip-10-0-131-121 kubenswrapper[2568]: W0728 15:33:40.086815    2568 feature_gate.go:328] unrecognized feature gate: Example
Jul 28 15:33:40.092902 ip-10-0-131-121 kubenswrapper[2568]: W0728 15:33:40.086819    2568 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 28 15:33:40.092902 ip-10-0-131-121 kubenswrapper[2568]: W0728 15:33:40.086823    2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 28 15:33:40.092902 ip-10-0-131-121 kubenswrapper[2568]: W0728 15:33:40.086827    2568 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 28 15:33:40.092902 ip-10-0-131-121 kubenswrapper[2568]: W0728 15:33:40.086831    2568 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 28 15:33:40.092902 ip-10-0-131-121 kubenswrapper[2568]: W0728 15:33:40.086836    2568 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 28 15:33:40.092902 ip-10-0-131-121 kubenswrapper[2568]: W0728 15:33:40.086840    2568 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 28 15:33:40.092902 ip-10-0-131-121 kubenswrapper[2568]: W0728 15:33:40.086844    2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 28 15:33:40.093370 ip-10-0-131-121 kubenswrapper[2568]: W0728 15:33:40.086848    2568 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 28 15:33:40.093370 ip-10-0-131-121 kubenswrapper[2568]: W0728 15:33:40.086852    2568 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 28 15:33:40.093370 ip-10-0-131-121 kubenswrapper[2568]: W0728 15:33:40.086862    2568 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 28 15:33:40.093370 ip-10-0-131-121 kubenswrapper[2568]: W0728 15:33:40.086867    2568 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 28 15:33:40.093370 ip-10-0-131-121 kubenswrapper[2568]: W0728 15:33:40.086873    2568 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 28 15:33:40.093370 ip-10-0-131-121 kubenswrapper[2568]: W0728 15:33:40.086877    2568 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 28 15:33:40.093370 ip-10-0-131-121 kubenswrapper[2568]: W0728 15:33:40.086883    2568 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 28 15:33:40.093370 ip-10-0-131-121 kubenswrapper[2568]: W0728 15:33:40.086887    2568 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 28 15:33:40.093370 ip-10-0-131-121 kubenswrapper[2568]: W0728 15:33:40.086891    2568 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 28 15:33:40.093370 ip-10-0-131-121 kubenswrapper[2568]: W0728 15:33:40.086895    2568 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 28 15:33:40.093370 ip-10-0-131-121 kubenswrapper[2568]: W0728 15:33:40.086899    2568 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 28 15:33:40.093370 ip-10-0-131-121 kubenswrapper[2568]: W0728 15:33:40.086903    2568 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 28 15:33:40.093370 ip-10-0-131-121 kubenswrapper[2568]: W0728 15:33:40.086907    2568 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 28 15:33:40.093370 ip-10-0-131-121 kubenswrapper[2568]: W0728 15:33:40.086912    2568 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 28 15:33:40.093370 ip-10-0-131-121 kubenswrapper[2568]: W0728 15:33:40.086916    2568 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 28 15:33:40.093370 ip-10-0-131-121 kubenswrapper[2568]: W0728 15:33:40.086920    2568 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 28 15:33:40.093370 ip-10-0-131-121 kubenswrapper[2568]: W0728 15:33:40.086924    2568 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 28 15:33:40.093370 ip-10-0-131-121 kubenswrapper[2568]: W0728 15:33:40.086928    2568 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 28 15:33:40.093370 ip-10-0-131-121 kubenswrapper[2568]: W0728 15:33:40.086932    2568 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 28 15:33:40.093885 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:33:40.087035    2568 flags.go:64] FLAG: --address="0.0.0.0"
Jul 28 15:33:40.093885 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:33:40.087046    2568 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Jul 28 15:33:40.093885 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:33:40.087057    2568 flags.go:64] FLAG: --anonymous-auth="true"
Jul 28 15:33:40.093885 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:33:40.087064    2568 flags.go:64] FLAG: --application-metrics-count-limit="100"
Jul 28 15:33:40.093885 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:33:40.087071    2568 flags.go:64] FLAG: --authentication-token-webhook="false"
Jul 28 15:33:40.093885 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:33:40.087076    2568 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Jul 28 15:33:40.093885 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:33:40.087082    2568 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Jul 28 15:33:40.093885 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:33:40.087089    2568 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Jul 28 15:33:40.093885 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:33:40.087094    2568 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Jul 28 15:33:40.093885 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:33:40.087099    2568 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Jul 28 15:33:40.093885 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:33:40.087105    2568 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Jul 28 15:33:40.093885 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:33:40.087110    2568 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Jul 28 15:33:40.093885 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:33:40.087115    2568 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Jul 28 15:33:40.093885 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:33:40.087120    2568 flags.go:64] FLAG: --cgroup-root=""
Jul 28 15:33:40.093885 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:33:40.087124    2568 flags.go:64] FLAG: --cgroups-per-qos="true"
Jul 28 15:33:40.093885 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:33:40.087129    2568 flags.go:64] FLAG: --client-ca-file=""
Jul 28 15:33:40.093885 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:33:40.087134    2568 flags.go:64] FLAG: --cloud-provider="external"
Jul 28 15:33:40.093885 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:33:40.087141    2568 flags.go:64] FLAG: --cluster-dns="[]"
Jul 28 15:33:40.093885 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:33:40.087150    2568 flags.go:64] FLAG: --cluster-domain=""
Jul 28 15:33:40.093885 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:33:40.087156    2568 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Jul 28 15:33:40.093885 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:33:40.087161    2568 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Jul 28 15:33:40.093885 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:33:40.087166    2568 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Jul 28 15:33:40.093885 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:33:40.087172    2568 flags.go:64] FLAG: --container-log-max-files="5"
Jul 28 15:33:40.093885 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:33:40.087178    2568 flags.go:64] FLAG: --container-log-max-size="10Mi"
Jul 28 15:33:40.094545 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:33:40.087183    2568 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Jul 28 15:33:40.094545 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:33:40.087188    2568 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Jul 28 15:33:40.094545 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:33:40.087194    2568 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Jul 28 15:33:40.094545 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:33:40.087198    2568 flags.go:64] FLAG: --contention-profiling="false"
Jul 28 15:33:40.094545 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:33:40.087203    2568 flags.go:64] FLAG: --cpu-cfs-quota="true"
Jul 28 15:33:40.094545 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:33:40.087208    2568 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Jul 28 15:33:40.094545 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:33:40.087213    2568 flags.go:64] FLAG: --cpu-manager-policy="none"
Jul 28 15:33:40.094545 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:33:40.087218    2568 flags.go:64] FLAG: --cpu-manager-policy-options=""
Jul 28 15:33:40.094545 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:33:40.087224    2568 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Jul 28 15:33:40.094545 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:33:40.087230    2568 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Jul 28 15:33:40.094545 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:33:40.087234    2568 flags.go:64] FLAG: --enable-debugging-handlers="true"
Jul 28 15:33:40.094545 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:33:40.087239    2568 flags.go:64] FLAG: --enable-load-reader="false"
Jul 28 15:33:40.094545 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:33:40.087244    2568 flags.go:64] FLAG: --enable-server="true"
Jul 28 15:33:40.094545 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:33:40.087249    2568 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Jul 28 15:33:40.094545 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:33:40.087256    2568 flags.go:64] FLAG: --event-burst="100"
Jul 28 15:33:40.094545 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:33:40.087262    2568 flags.go:64] FLAG: --event-qps="50"
Jul 28 15:33:40.094545 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:33:40.087267    2568 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Jul 28 15:33:40.094545 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:33:40.087272    2568 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Jul 28 15:33:40.094545 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:33:40.087276    2568 flags.go:64] FLAG: --eviction-hard=""
Jul 28 15:33:40.094545 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:33:40.087283    2568 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Jul 28 15:33:40.094545 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:33:40.087287    2568 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Jul 28 15:33:40.094545 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:33:40.087292    2568 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Jul 28 15:33:40.094545 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:33:40.087297    2568 flags.go:64] FLAG: --eviction-soft=""
Jul 28 15:33:40.094545 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:33:40.087302    2568 flags.go:64] FLAG: --eviction-soft-grace-period=""
Jul 28 15:33:40.094545 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:33:40.087307    2568 flags.go:64] FLAG: --exit-on-lock-contention="false"
Jul 28 15:33:40.095197 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:33:40.087314    2568 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Jul 28 15:33:40.095197 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:33:40.087319    2568 flags.go:64] FLAG: --experimental-mounter-path=""
Jul 28 15:33:40.095197 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:33:40.087327    2568 flags.go:64] FLAG: --fail-cgroupv1="false"
Jul 28 15:33:40.095197 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:33:40.087331    2568 flags.go:64] FLAG: --fail-swap-on="true"
Jul 28 15:33:40.095197 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:33:40.087336    2568 flags.go:64] FLAG: --feature-gates=""
Jul 28 15:33:40.095197 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:33:40.087342    2568 flags.go:64] FLAG: --file-check-frequency="20s"
Jul 28 15:33:40.095197 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:33:40.087347    2568 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Jul 28 15:33:40.095197 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:33:40.087352    2568 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Jul 28 15:33:40.095197 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:33:40.087357    2568 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Jul 28 15:33:40.095197 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:33:40.087362    2568 flags.go:64] FLAG: --healthz-port="10248"
Jul 28 15:33:40.095197 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:33:40.087368    2568 flags.go:64] FLAG: --help="false"
Jul 28 15:33:40.095197 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:33:40.087372    2568 flags.go:64] FLAG: --hostname-override="ip-10-0-131-121.ec2.internal"
Jul 28 15:33:40.095197 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:33:40.087377    2568 flags.go:64] FLAG: --housekeeping-interval="10s"
Jul 28 15:33:40.095197 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:33:40.087382    2568 flags.go:64] FLAG: --http-check-frequency="20s"
Jul 28 15:33:40.095197 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:33:40.087387    2568 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Jul 28 15:33:40.095197 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:33:40.087392    2568 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Jul 28 15:33:40.095197 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:33:40.087398    2568 flags.go:64] FLAG: --image-gc-high-threshold="85"
Jul 28 15:33:40.095197 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:33:40.087403    2568 flags.go:64] FLAG: --image-gc-low-threshold="80"
Jul 28 15:33:40.095197 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:33:40.087408    2568 flags.go:64] FLAG: --image-service-endpoint=""
Jul 28 15:33:40.095197 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:33:40.087412    2568 flags.go:64] FLAG: --kernel-memcg-notification="false"
Jul 28 15:33:40.095197 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:33:40.087417    2568 flags.go:64] FLAG: --kube-api-burst="100"
Jul 28 15:33:40.095197 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:33:40.087421    2568 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Jul 28 15:33:40.095197 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:33:40.087426    2568 flags.go:64] FLAG: --kube-api-qps="50"
Jul 28 15:33:40.095197 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:33:40.087431    2568 flags.go:64] FLAG: --kube-reserved=""
Jul 28 15:33:40.095802 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:33:40.087436    2568 flags.go:64] FLAG: --kube-reserved-cgroup=""
Jul 28 15:33:40.095802 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:33:40.087440    2568 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Jul 28 15:33:40.095802 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:33:40.087445    2568 flags.go:64] FLAG: --kubelet-cgroups=""
Jul 28 15:33:40.095802 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:33:40.087450    2568 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Jul 28 15:33:40.095802 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:33:40.087454    2568 flags.go:64] FLAG: --lock-file=""
Jul 28 15:33:40.095802 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:33:40.087459    2568 flags.go:64] FLAG: --log-cadvisor-usage="false"
Jul 28 15:33:40.095802 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:33:40.087464    2568 flags.go:64] FLAG: --log-flush-frequency="5s"
Jul 28 15:33:40.095802 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:33:40.087469    2568 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Jul 28 15:33:40.095802 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:33:40.087479    2568 flags.go:64] FLAG: --log-json-split-stream="false"
Jul 28 15:33:40.095802 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:33:40.087484    2568 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Jul 28 15:33:40.095802 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:33:40.087490    2568 flags.go:64] FLAG: --log-text-split-stream="false"
Jul 28 15:33:40.095802 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:33:40.087495    2568 flags.go:64] FLAG: --logging-format="text"
Jul 28 15:33:40.095802 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:33:40.087499    2568 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Jul 28 15:33:40.095802 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:33:40.087504    2568 flags.go:64] FLAG: --make-iptables-util-chains="true"
Jul 28 15:33:40.095802 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:33:40.087509    2568 flags.go:64] FLAG: --manifest-url=""
Jul 28 15:33:40.095802 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:33:40.087514    2568 flags.go:64] FLAG: --manifest-url-header=""
Jul 28 15:33:40.095802 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:33:40.087521    2568 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Jul 28 15:33:40.095802 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:33:40.087526    2568 flags.go:64] FLAG: --max-open-files="1000000"
Jul 28 15:33:40.095802 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:33:40.087533    2568 flags.go:64] FLAG: --max-pods="110"
Jul 28 15:33:40.095802 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:33:40.087538    2568 flags.go:64] FLAG: --maximum-dead-containers="-1"
Jul 28 15:33:40.095802 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:33:40.087543    2568 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Jul 28 15:33:40.095802 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:33:40.087548    2568 flags.go:64] FLAG: --memory-manager-policy="None"
Jul 28 15:33:40.095802 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:33:40.087553    2568 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Jul 28 15:33:40.095802 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:33:40.087557    2568 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Jul 28 15:33:40.095802 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:33:40.087562    2568 flags.go:64] FLAG: --node-ip="0.0.0.0"
Jul 28 15:33:40.096415 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:33:40.087567    2568 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Jul 28 15:33:40.096415 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:33:40.087579    2568 flags.go:64] FLAG: --node-status-max-images="50"
Jul 28 15:33:40.096415 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:33:40.087584    2568 flags.go:64] FLAG: --node-status-update-frequency="10s"
Jul 28 15:33:40.096415 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:33:40.087589    2568 flags.go:64] FLAG: --oom-score-adj="-999"
Jul 28 15:33:40.096415 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:33:40.087594    2568 flags.go:64] FLAG: --pod-cidr=""
Jul 28 15:33:40.096415 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:33:40.087598    2568 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:7505336a406394b8c1c77d14d890a9f628aa4cfff172312179cb9af8c28c30bb"
Jul 28 15:33:40.096415 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:33:40.087607    2568 flags.go:64] FLAG: --pod-manifest-path=""
Jul 28 15:33:40.096415 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:33:40.087612    2568 flags.go:64] FLAG: --pod-max-pids="-1"
Jul 28 15:33:40.096415 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:33:40.087617    2568 flags.go:64] FLAG: --pods-per-core="0"
Jul 28 15:33:40.096415 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:33:40.087622    2568 flags.go:64] FLAG: --port="10250"
Jul 28 15:33:40.096415 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:33:40.087627    2568 flags.go:64] FLAG: --protect-kernel-defaults="false"
Jul 28 15:33:40.096415 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:33:40.087632    2568 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-05f144d8153a40878"
Jul 28 15:33:40.096415 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:33:40.087637    2568 flags.go:64] FLAG: --qos-reserved=""
Jul 28 15:33:40.096415 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:33:40.087642    2568 flags.go:64] FLAG: --read-only-port="10255"
Jul 28 15:33:40.096415 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:33:40.087646    2568 flags.go:64] FLAG: --register-node="true"
Jul 28 15:33:40.096415 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:33:40.087651    2568 flags.go:64] FLAG: --register-with-taints=""
Jul 28 15:33:40.096415 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:33:40.087657    2568 flags.go:64] FLAG: --registry-burst="10"
Jul 28 15:33:40.096415 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:33:40.087663    2568 flags.go:64] FLAG: --registry-qps="5"
Jul 28 15:33:40.096415 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:33:40.087668    2568 flags.go:64] FLAG: --reserved-cpus=""
Jul 28 15:33:40.096415 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:33:40.087672    2568 flags.go:64] FLAG: --reserved-memory=""
Jul 28 15:33:40.096415 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:33:40.087678    2568 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Jul 28 15:33:40.096415 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:33:40.087683    2568 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Jul 28 15:33:40.096415 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:33:40.087688    2568 flags.go:64] FLAG: --rotate-certificates="false"
Jul 28 15:33:40.096415 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:33:40.087693    2568 flags.go:64] FLAG: --rotate-server-certificates="false"
Jul 28 15:33:40.096415 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:33:40.087697    2568 flags.go:64] FLAG: --runonce="false"
Jul 28 15:33:40.097062 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:33:40.087702    2568 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Jul 28 15:33:40.097062 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:33:40.087707    2568 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Jul 28 15:33:40.097062 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:33:40.087713    2568 flags.go:64] FLAG: --seccomp-default="false"
Jul 28 15:33:40.097062 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:33:40.087718    2568 flags.go:64] FLAG: --serialize-image-pulls="true"
Jul 28 15:33:40.097062 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:33:40.087722    2568 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Jul 28 15:33:40.097062 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:33:40.087727    2568 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Jul 28 15:33:40.097062 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:33:40.087732    2568 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Jul 28 15:33:40.097062 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:33:40.087736    2568 flags.go:64] FLAG: --storage-driver-password="root"
Jul 28 15:33:40.097062 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:33:40.087741    2568 flags.go:64] FLAG: --storage-driver-secure="false"
Jul 28 15:33:40.097062 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:33:40.087746    2568 flags.go:64] FLAG: --storage-driver-table="stats"
Jul 28 15:33:40.097062 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:33:40.087750    2568 flags.go:64] FLAG: --storage-driver-user="root"
Jul 28 15:33:40.097062 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:33:40.087755    2568 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Jul 28 15:33:40.097062 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:33:40.087760    2568 flags.go:64] FLAG: --sync-frequency="1m0s"
Jul 28 15:33:40.097062 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:33:40.087764    2568 flags.go:64] FLAG: --system-cgroups=""
Jul 28 15:33:40.097062 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:33:40.087769    2568 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Jul 28 15:33:40.097062 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:33:40.087794    2568 flags.go:64] FLAG: --system-reserved-cgroup=""
Jul 28 15:33:40.097062 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:33:40.087798    2568 flags.go:64] FLAG: --tls-cert-file=""
Jul 28 15:33:40.097062 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:33:40.087803    2568 flags.go:64] FLAG: --tls-cipher-suites="[]"
Jul 28 15:33:40.097062 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:33:40.087809    2568 flags.go:64] FLAG: --tls-min-version=""
Jul 28 15:33:40.097062 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:33:40.087814    2568 flags.go:64] FLAG: --tls-private-key-file=""
Jul 28 15:33:40.097062 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:33:40.087818    2568 flags.go:64] FLAG: --topology-manager-policy="none"
Jul 28 15:33:40.097062 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:33:40.087823    2568 flags.go:64] FLAG: --topology-manager-policy-options=""
Jul 28 15:33:40.097062 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:33:40.087828    2568 flags.go:64] FLAG: --topology-manager-scope="container"
Jul 28 15:33:40.097062 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:33:40.087833    2568 flags.go:64] FLAG: --v="2"
Jul 28 15:33:40.097062 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:33:40.087839    2568 flags.go:64] FLAG: --version="false"
Jul 28 15:33:40.097696 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:33:40.087847    2568 flags.go:64] FLAG: --vmodule=""
Jul 28 15:33:40.097696 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:33:40.087853    2568 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Jul 28 15:33:40.097696 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:33:40.087859    2568 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Jul 28 15:33:40.097696 ip-10-0-131-121 kubenswrapper[2568]: W0728 15:33:40.088077    2568 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 28 15:33:40.097696 ip-10-0-131-121 kubenswrapper[2568]: W0728 15:33:40.088085    2568 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 28 15:33:40.097696 ip-10-0-131-121 kubenswrapper[2568]: W0728 15:33:40.088091    2568 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 28 15:33:40.097696 ip-10-0-131-121 kubenswrapper[2568]: W0728 15:33:40.088096    2568 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 28 15:33:40.097696 ip-10-0-131-121 kubenswrapper[2568]: W0728 15:33:40.088102    2568 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 28 15:33:40.097696 ip-10-0-131-121 kubenswrapper[2568]: W0728 15:33:40.088106    2568 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 28 15:33:40.097696 ip-10-0-131-121 kubenswrapper[2568]: W0728 15:33:40.088111    2568 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 28 15:33:40.097696 ip-10-0-131-121 kubenswrapper[2568]: W0728 15:33:40.088116    2568 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 28 15:33:40.097696 ip-10-0-131-121 kubenswrapper[2568]: W0728 15:33:40.088121    2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 28 15:33:40.097696 ip-10-0-131-121 kubenswrapper[2568]: W0728 15:33:40.088125    2568 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 28 15:33:40.097696 ip-10-0-131-121 kubenswrapper[2568]: W0728 15:33:40.088130    2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 28 15:33:40.097696 ip-10-0-131-121 kubenswrapper[2568]: W0728 15:33:40.088134    2568 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 28 15:33:40.097696 ip-10-0-131-121 kubenswrapper[2568]: W0728 15:33:40.088139    2568 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 28 15:33:40.097696 ip-10-0-131-121 kubenswrapper[2568]: W0728 15:33:40.088143    2568 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 28 15:33:40.097696 ip-10-0-131-121 kubenswrapper[2568]: W0728 15:33:40.088147    2568 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 28 15:33:40.097696 ip-10-0-131-121 kubenswrapper[2568]: W0728 15:33:40.088152    2568 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 28 15:33:40.097696 ip-10-0-131-121 kubenswrapper[2568]: W0728 15:33:40.088156    2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 28 15:33:40.098213 ip-10-0-131-121 kubenswrapper[2568]: W0728 15:33:40.088161    2568 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 28 15:33:40.098213 ip-10-0-131-121 kubenswrapper[2568]: W0728 15:33:40.088168    2568 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 28 15:33:40.098213 ip-10-0-131-121 kubenswrapper[2568]: W0728 15:33:40.088174    2568 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 28 15:33:40.098213 ip-10-0-131-121 kubenswrapper[2568]: W0728 15:33:40.088180    2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 28 15:33:40.098213 ip-10-0-131-121 kubenswrapper[2568]: W0728 15:33:40.088185    2568 feature_gate.go:328] unrecognized feature gate: Example2
Jul 28 15:33:40.098213 ip-10-0-131-121 kubenswrapper[2568]: W0728 15:33:40.088191    2568 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 28 15:33:40.098213 ip-10-0-131-121 kubenswrapper[2568]: W0728 15:33:40.088195    2568 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 28 15:33:40.098213 ip-10-0-131-121 kubenswrapper[2568]: W0728 15:33:40.088200    2568 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 28 15:33:40.098213 ip-10-0-131-121 kubenswrapper[2568]: W0728 15:33:40.088205    2568 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 28 15:33:40.098213 ip-10-0-131-121 kubenswrapper[2568]: W0728 15:33:40.088209    2568 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 28 15:33:40.098213 ip-10-0-131-121 kubenswrapper[2568]: W0728 15:33:40.088213    2568 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 28 15:33:40.098213 ip-10-0-131-121 kubenswrapper[2568]: W0728 15:33:40.088218    2568 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 28 15:33:40.098213 ip-10-0-131-121 kubenswrapper[2568]: W0728 15:33:40.088222    2568 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 28 15:33:40.098213 ip-10-0-131-121 kubenswrapper[2568]: W0728 15:33:40.088227    2568 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 28 15:33:40.098213 ip-10-0-131-121 kubenswrapper[2568]: W0728 15:33:40.088232    2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 28 15:33:40.098213 ip-10-0-131-121 kubenswrapper[2568]: W0728 15:33:40.088236    2568 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 28 15:33:40.098213 ip-10-0-131-121 kubenswrapper[2568]: W0728 15:33:40.088240    2568 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 28 15:33:40.098213 ip-10-0-131-121 kubenswrapper[2568]: W0728 15:33:40.088245    2568 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 28 15:33:40.098213 ip-10-0-131-121 kubenswrapper[2568]: W0728 15:33:40.088249    2568 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 28 15:33:40.098678 ip-10-0-131-121 kubenswrapper[2568]: W0728 15:33:40.088253    2568 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 28 15:33:40.098678 ip-10-0-131-121 kubenswrapper[2568]: W0728 15:33:40.088257    2568 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 28 15:33:40.098678 ip-10-0-131-121 kubenswrapper[2568]: W0728 15:33:40.088261    2568 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 28 15:33:40.098678 ip-10-0-131-121 kubenswrapper[2568]: W0728 15:33:40.088265    2568 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 28 15:33:40.098678 ip-10-0-131-121 kubenswrapper[2568]: W0728 15:33:40.088269    2568 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 28 15:33:40.098678 ip-10-0-131-121 kubenswrapper[2568]: W0728 15:33:40.088273    2568 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 28 15:33:40.098678 ip-10-0-131-121 kubenswrapper[2568]: W0728 15:33:40.088277    2568 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 28 15:33:40.098678 ip-10-0-131-121 kubenswrapper[2568]: W0728 15:33:40.088282    2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 28 15:33:40.098678 ip-10-0-131-121 kubenswrapper[2568]: W0728 15:33:40.088286    2568 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 28 15:33:40.098678 ip-10-0-131-121 kubenswrapper[2568]: W0728 15:33:40.088290    2568 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 28 15:33:40.098678 ip-10-0-131-121 kubenswrapper[2568]: W0728 15:33:40.088294    2568 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 28 15:33:40.098678 ip-10-0-131-121 kubenswrapper[2568]: W0728 15:33:40.088298    2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 28 15:33:40.098678 ip-10-0-131-121 kubenswrapper[2568]: W0728 15:33:40.088303    2568 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 28 15:33:40.098678 ip-10-0-131-121 kubenswrapper[2568]: W0728 15:33:40.088307    2568 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 28 15:33:40.098678 ip-10-0-131-121 kubenswrapper[2568]: W0728 15:33:40.088311    2568 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 28 15:33:40.098678 ip-10-0-131-121 kubenswrapper[2568]: W0728 15:33:40.088315    2568 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 28 15:33:40.098678 ip-10-0-131-121 kubenswrapper[2568]: W0728 15:33:40.088320    2568 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 28 15:33:40.098678 ip-10-0-131-121 kubenswrapper[2568]: W0728 15:33:40.088325    2568 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 28 15:33:40.098678 ip-10-0-131-121 kubenswrapper[2568]: W0728 15:33:40.088329    2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 28 15:33:40.098678 ip-10-0-131-121 kubenswrapper[2568]: W0728 15:33:40.088333    2568 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 28 15:33:40.099178 ip-10-0-131-121 kubenswrapper[2568]: W0728 15:33:40.088337    2568 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 28 15:33:40.099178 ip-10-0-131-121 kubenswrapper[2568]: W0728 15:33:40.088341    2568 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 28 15:33:40.099178 ip-10-0-131-121 kubenswrapper[2568]: W0728 15:33:40.088346    2568 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 28 15:33:40.099178 ip-10-0-131-121 kubenswrapper[2568]: W0728 15:33:40.088350    2568 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 28 15:33:40.099178 ip-10-0-131-121 kubenswrapper[2568]: W0728 15:33:40.088354    2568 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 28 15:33:40.099178 ip-10-0-131-121 kubenswrapper[2568]: W0728 15:33:40.088358    2568 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 28 15:33:40.099178 ip-10-0-131-121 kubenswrapper[2568]: W0728 15:33:40.088362    2568 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 28 15:33:40.099178 ip-10-0-131-121 kubenswrapper[2568]: W0728 15:33:40.088367    2568 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 28 15:33:40.099178 ip-10-0-131-121 kubenswrapper[2568]: W0728 15:33:40.088372    2568 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 28 15:33:40.099178 ip-10-0-131-121 kubenswrapper[2568]: W0728 15:33:40.088376    2568 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 28 15:33:40.099178 ip-10-0-131-121 kubenswrapper[2568]: W0728 15:33:40.088380    2568 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 28 15:33:40.099178 ip-10-0-131-121 kubenswrapper[2568]: W0728 15:33:40.088385    2568 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 28 15:33:40.099178 ip-10-0-131-121 kubenswrapper[2568]: W0728 15:33:40.088389    2568 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 28 15:33:40.099178 ip-10-0-131-121 kubenswrapper[2568]: W0728 15:33:40.088393    2568 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 28 15:33:40.099178 ip-10-0-131-121 kubenswrapper[2568]: W0728 15:33:40.088397    2568 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 28 15:33:40.099178 ip-10-0-131-121 kubenswrapper[2568]: W0728 15:33:40.088401    2568 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 28 15:33:40.099178 ip-10-0-131-121 kubenswrapper[2568]: W0728 15:33:40.088405    2568 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 28 15:33:40.099178 ip-10-0-131-121 kubenswrapper[2568]: W0728 15:33:40.088409    2568 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 28 15:33:40.099178 ip-10-0-131-121 kubenswrapper[2568]: W0728 15:33:40.088413    2568 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 28 15:33:40.099178 ip-10-0-131-121 kubenswrapper[2568]: W0728 15:33:40.088417    2568 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 28 15:33:40.099178 ip-10-0-131-121 kubenswrapper[2568]: W0728 15:33:40.088421    2568 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 28 15:33:40.099682 ip-10-0-131-121 kubenswrapper[2568]: W0728 15:33:40.088425    2568 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 28 15:33:40.099682 ip-10-0-131-121 kubenswrapper[2568]: W0728 15:33:40.088429    2568 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 28 15:33:40.099682 ip-10-0-131-121 kubenswrapper[2568]: W0728 15:33:40.088437    2568 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 28 15:33:40.099682 ip-10-0-131-121 kubenswrapper[2568]: W0728 15:33:40.088442    2568 feature_gate.go:328] unrecognized feature gate: Example
Jul 28 15:33:40.099682 ip-10-0-131-121 kubenswrapper[2568]: W0728 15:33:40.088446    2568 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 28 15:33:40.099682 ip-10-0-131-121 kubenswrapper[2568]: W0728 15:33:40.088450    2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 28 15:33:40.099682 ip-10-0-131-121 kubenswrapper[2568]: W0728 15:33:40.088454    2568 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 28 15:33:40.099682 ip-10-0-131-121 kubenswrapper[2568]: W0728 15:33:40.088461    2568 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 28 15:33:40.099682 ip-10-0-131-121 kubenswrapper[2568]: W0728 15:33:40.088466    2568 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 28 15:33:40.099682 ip-10-0-131-121 kubenswrapper[2568]: W0728 15:33:40.088473    2568 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 28 15:33:40.099682 ip-10-0-131-121 kubenswrapper[2568]: W0728 15:33:40.088479    2568 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 28 15:33:40.099682 ip-10-0-131-121 kubenswrapper[2568]: W0728 15:33:40.088483    2568 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 28 15:33:40.099682 ip-10-0-131-121 kubenswrapper[2568]: W0728 15:33:40.088488    2568 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 28 15:33:40.099682 ip-10-0-131-121 kubenswrapper[2568]: W0728 15:33:40.088493    2568 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 28 15:33:40.099682 ip-10-0-131-121 kubenswrapper[2568]: W0728 15:33:40.088497    2568 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 28 15:33:40.099682 ip-10-0-131-121 kubenswrapper[2568]: W0728 15:33:40.088501    2568 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 28 15:33:40.099682 ip-10-0-131-121 kubenswrapper[2568]: W0728 15:33:40.088506    2568 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 28 15:33:40.099682 ip-10-0-131-121 kubenswrapper[2568]: W0728 15:33:40.088510    2568 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 28 15:33:40.099682 ip-10-0-131-121 kubenswrapper[2568]: W0728 15:33:40.088514    2568 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 28 15:33:40.100156 ip-10-0-131-121 kubenswrapper[2568]: W0728 15:33:40.088519    2568 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 28 15:33:40.100156 ip-10-0-131-121 kubenswrapper[2568]: W0728 15:33:40.088523    2568 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 28 15:33:40.100156 ip-10-0-131-121 kubenswrapper[2568]: W0728 15:33:40.088530    2568 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 28 15:33:40.100156 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:33:40.089152    2568 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 28 15:33:40.100156 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:33:40.096015    2568 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Jul 28 15:33:40.100156 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:33:40.096032    2568 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Jul 28 15:33:40.100156 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:33:40.096713    2568 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Jul 28 15:33:40.100156 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:33:40.096721    2568 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Jul 28 15:33:40.100156 ip-10-0-131-121 kubenswrapper[2568]: W0728 15:33:40.096800    2568 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 28 15:33:40.100156 ip-10-0-131-121 kubenswrapper[2568]: W0728 15:33:40.096806    2568 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 28 15:33:40.100156 ip-10-0-131-121 kubenswrapper[2568]: W0728 15:33:40.096810    2568 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 28 15:33:40.100156 ip-10-0-131-121 kubenswrapper[2568]: W0728 15:33:40.096815    2568 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 28 15:33:40.100156 ip-10-0-131-121 kubenswrapper[2568]: W0728 15:33:40.096819    2568 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 28 15:33:40.100156 ip-10-0-131-121 kubenswrapper[2568]: W0728 15:33:40.096822    2568 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 28 15:33:40.100517 ip-10-0-131-121 kubenswrapper[2568]: W0728 15:33:40.096825    2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 28 15:33:40.100517 ip-10-0-131-121 kubenswrapper[2568]: W0728 15:33:40.096828    2568 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 28 15:33:40.100517 ip-10-0-131-121 kubenswrapper[2568]: W0728 15:33:40.096831    2568 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 28 15:33:40.100517 ip-10-0-131-121 kubenswrapper[2568]: W0728 15:33:40.096834    2568 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 28 15:33:40.100517 ip-10-0-131-121 kubenswrapper[2568]: W0728 15:33:40.096836    2568 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 28 15:33:40.100517 ip-10-0-131-121 kubenswrapper[2568]: W0728 15:33:40.096839    2568 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 28 15:33:40.100517 ip-10-0-131-121 kubenswrapper[2568]: W0728 15:33:40.096842    2568 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 28 15:33:40.100517 ip-10-0-131-121 kubenswrapper[2568]: W0728 15:33:40.096845    2568 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 28 15:33:40.100517 ip-10-0-131-121 kubenswrapper[2568]: W0728 15:33:40.096848    2568 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 28 15:33:40.100517 ip-10-0-131-121 kubenswrapper[2568]: W0728 15:33:40.096850    2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 28 15:33:40.100517 ip-10-0-131-121 kubenswrapper[2568]: W0728 15:33:40.096853    2568 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 28 15:33:40.100517 ip-10-0-131-121 kubenswrapper[2568]: W0728 15:33:40.096855    2568 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 28 15:33:40.100517 ip-10-0-131-121 kubenswrapper[2568]: W0728 15:33:40.096858    2568 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 28 15:33:40.100517 ip-10-0-131-121 kubenswrapper[2568]: W0728 15:33:40.096862    2568 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 28 15:33:40.100517 ip-10-0-131-121 kubenswrapper[2568]: W0728 15:33:40.096866    2568 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 28 15:33:40.100517 ip-10-0-131-121 kubenswrapper[2568]: W0728 15:33:40.096871    2568 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 28 15:33:40.100517 ip-10-0-131-121 kubenswrapper[2568]: W0728 15:33:40.096875    2568 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 28 15:33:40.100517 ip-10-0-131-121 kubenswrapper[2568]: W0728 15:33:40.096879    2568 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 28 15:33:40.100517 ip-10-0-131-121 kubenswrapper[2568]: W0728 15:33:40.096882    2568 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 28 15:33:40.100517 ip-10-0-131-121 kubenswrapper[2568]: W0728 15:33:40.096885    2568 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 28 15:33:40.101092 ip-10-0-131-121 kubenswrapper[2568]: W0728 15:33:40.096888    2568 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 28 15:33:40.101092 ip-10-0-131-121 kubenswrapper[2568]: W0728 15:33:40.096891    2568 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 28 15:33:40.101092 ip-10-0-131-121 kubenswrapper[2568]: W0728 15:33:40.096894    2568 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 28 15:33:40.101092 ip-10-0-131-121 kubenswrapper[2568]: W0728 15:33:40.096897    2568 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 28 15:33:40.101092 ip-10-0-131-121 kubenswrapper[2568]: W0728 15:33:40.096901    2568 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 28 15:33:40.101092 ip-10-0-131-121 kubenswrapper[2568]: W0728 15:33:40.096903    2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 28 15:33:40.101092 ip-10-0-131-121 kubenswrapper[2568]: W0728 15:33:40.096906    2568 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 28 15:33:40.101092 ip-10-0-131-121 kubenswrapper[2568]: W0728 15:33:40.096910    2568 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 28 15:33:40.101092 ip-10-0-131-121 kubenswrapper[2568]: W0728 15:33:40.096914    2568 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 28 15:33:40.101092 ip-10-0-131-121 kubenswrapper[2568]: W0728 15:33:40.096933    2568 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 28 15:33:40.101092 ip-10-0-131-121 kubenswrapper[2568]: W0728 15:33:40.096937    2568 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 28 15:33:40.101092 ip-10-0-131-121 kubenswrapper[2568]: W0728 15:33:40.096942    2568 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 28 15:33:40.101092 ip-10-0-131-121 kubenswrapper[2568]: W0728 15:33:40.096946    2568 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 28 15:33:40.101092 ip-10-0-131-121 kubenswrapper[2568]: W0728 15:33:40.096950    2568 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 28 15:33:40.101092 ip-10-0-131-121 kubenswrapper[2568]: W0728 15:33:40.096954    2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 28 15:33:40.101092 ip-10-0-131-121 kubenswrapper[2568]: W0728 15:33:40.096958    2568 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 28 15:33:40.101092 ip-10-0-131-121 kubenswrapper[2568]: W0728 15:33:40.096961    2568 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 28 15:33:40.101092 ip-10-0-131-121 kubenswrapper[2568]: W0728 15:33:40.096964    2568 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 28 15:33:40.101092 ip-10-0-131-121 kubenswrapper[2568]: W0728 15:33:40.096967    2568 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 28 15:33:40.101092 ip-10-0-131-121 kubenswrapper[2568]: W0728 15:33:40.096970    2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 28 15:33:40.101572 ip-10-0-131-121 kubenswrapper[2568]: W0728 15:33:40.096972    2568 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 28 15:33:40.101572 ip-10-0-131-121 kubenswrapper[2568]: W0728 15:33:40.096975    2568 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 28 15:33:40.101572 ip-10-0-131-121 kubenswrapper[2568]: W0728 15:33:40.096977    2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 28 15:33:40.101572 ip-10-0-131-121 kubenswrapper[2568]: W0728 15:33:40.096980    2568 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 28 15:33:40.101572 ip-10-0-131-121 kubenswrapper[2568]: W0728 15:33:40.096982    2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 28 15:33:40.101572 ip-10-0-131-121 kubenswrapper[2568]: W0728 15:33:40.096985    2568 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 28 15:33:40.101572 ip-10-0-131-121 kubenswrapper[2568]: W0728 15:33:40.096987    2568 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 28 15:33:40.101572 ip-10-0-131-121 kubenswrapper[2568]: W0728 15:33:40.096990    2568 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 28 15:33:40.101572 ip-10-0-131-121 kubenswrapper[2568]: W0728 15:33:40.096993    2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 28 15:33:40.101572 ip-10-0-131-121 kubenswrapper[2568]: W0728 15:33:40.096995    2568 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 28 15:33:40.101572 ip-10-0-131-121 kubenswrapper[2568]: W0728 15:33:40.096998    2568 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 28 15:33:40.101572 ip-10-0-131-121 kubenswrapper[2568]: W0728 15:33:40.097001    2568 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 28 15:33:40.101572 ip-10-0-131-121 kubenswrapper[2568]: W0728 15:33:40.097004    2568 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 28 15:33:40.101572 ip-10-0-131-121 kubenswrapper[2568]: W0728 15:33:40.097007    2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 28 15:33:40.101572 ip-10-0-131-121 kubenswrapper[2568]: W0728 15:33:40.097009    2568 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 28 15:33:40.101572 ip-10-0-131-121 kubenswrapper[2568]: W0728 15:33:40.097013    2568 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 28 15:33:40.101572 ip-10-0-131-121 kubenswrapper[2568]: W0728 15:33:40.097016    2568 feature_gate.go:328] unrecognized feature gate: Example2
Jul 28 15:33:40.101572 ip-10-0-131-121 kubenswrapper[2568]: W0728 15:33:40.097018    2568 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 28 15:33:40.101572 ip-10-0-131-121 kubenswrapper[2568]: W0728 15:33:40.097022    2568 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 28 15:33:40.101572 ip-10-0-131-121 kubenswrapper[2568]: W0728 15:33:40.097026    2568 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 28 15:33:40.102077 ip-10-0-131-121 kubenswrapper[2568]: W0728 15:33:40.097031    2568 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 28 15:33:40.102077 ip-10-0-131-121 kubenswrapper[2568]: W0728 15:33:40.097035    2568 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 28 15:33:40.102077 ip-10-0-131-121 kubenswrapper[2568]: W0728 15:33:40.097039    2568 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 28 15:33:40.102077 ip-10-0-131-121 kubenswrapper[2568]: W0728 15:33:40.097041    2568 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 28 15:33:40.102077 ip-10-0-131-121 kubenswrapper[2568]: W0728 15:33:40.097045    2568 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 28 15:33:40.102077 ip-10-0-131-121 kubenswrapper[2568]: W0728 15:33:40.097049    2568 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 28 15:33:40.102077 ip-10-0-131-121 kubenswrapper[2568]: W0728 15:33:40.097052    2568 feature_gate.go:328] unrecognized feature gate: Example
Jul 28 15:33:40.102077 ip-10-0-131-121 kubenswrapper[2568]: W0728 15:33:40.097054    2568 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 28 15:33:40.102077 ip-10-0-131-121 kubenswrapper[2568]: W0728 15:33:40.097057    2568 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 28 15:33:40.102077 ip-10-0-131-121 kubenswrapper[2568]: W0728 15:33:40.097060    2568 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 28 15:33:40.102077 ip-10-0-131-121 kubenswrapper[2568]: W0728 15:33:40.097062    2568 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 28 15:33:40.102077 ip-10-0-131-121 kubenswrapper[2568]: W0728 15:33:40.097065    2568 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 28 15:33:40.102077 ip-10-0-131-121 kubenswrapper[2568]: W0728 15:33:40.097067    2568 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 28 15:33:40.102077 ip-10-0-131-121 kubenswrapper[2568]: W0728 15:33:40.097070    2568 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 28 15:33:40.102077 ip-10-0-131-121 kubenswrapper[2568]: W0728 15:33:40.097073    2568 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 28 15:33:40.102077 ip-10-0-131-121 kubenswrapper[2568]: W0728 15:33:40.097075    2568 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 28 15:33:40.102077 ip-10-0-131-121 kubenswrapper[2568]: W0728 15:33:40.097078    2568 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 28 15:33:40.102077 ip-10-0-131-121 kubenswrapper[2568]: W0728 15:33:40.097081    2568 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 28 15:33:40.102077 ip-10-0-131-121 kubenswrapper[2568]: W0728 15:33:40.097083    2568 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 28 15:33:40.102077 ip-10-0-131-121 kubenswrapper[2568]: W0728 15:33:40.097086    2568 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 28 15:33:40.102557 ip-10-0-131-121 kubenswrapper[2568]: W0728 15:33:40.097088    2568 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 28 15:33:40.102557 ip-10-0-131-121 kubenswrapper[2568]: W0728 15:33:40.097091    2568 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 28 15:33:40.102557 ip-10-0-131-121 kubenswrapper[2568]: W0728 15:33:40.097093    2568 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 28 15:33:40.102557 ip-10-0-131-121 kubenswrapper[2568]: W0728 15:33:40.097096    2568 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 28 15:33:40.102557 ip-10-0-131-121 kubenswrapper[2568]: W0728 15:33:40.097101    2568 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 28 15:33:40.102557 ip-10-0-131-121 kubenswrapper[2568]: W0728 15:33:40.097105    2568 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 28 15:33:40.102557 ip-10-0-131-121 kubenswrapper[2568]: W0728 15:33:40.097109    2568 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 28 15:33:40.102557 ip-10-0-131-121 kubenswrapper[2568]: W0728 15:33:40.097114    2568 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 28 15:33:40.102557 ip-10-0-131-121 kubenswrapper[2568]: W0728 15:33:40.097118    2568 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 28 15:33:40.102557 ip-10-0-131-121 kubenswrapper[2568]: W0728 15:33:40.097120    2568 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 28 15:33:40.102557 ip-10-0-131-121 kubenswrapper[2568]: W0728 15:33:40.097123    2568 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 28 15:33:40.102557 ip-10-0-131-121 kubenswrapper[2568]: W0728 15:33:40.097126    2568 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 28 15:33:40.102557 ip-10-0-131-121 kubenswrapper[2568]: W0728 15:33:40.097128    2568 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 28 15:33:40.102557 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:33:40.097133    2568 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 28 15:33:40.102932 ip-10-0-131-121 kubenswrapper[2568]: W0728 15:33:40.097286    2568 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 28 15:33:40.102932 ip-10-0-131-121 kubenswrapper[2568]: W0728 15:33:40.097293    2568 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 28 15:33:40.102932 ip-10-0-131-121 kubenswrapper[2568]: W0728 15:33:40.097296    2568 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 28 15:33:40.102932 ip-10-0-131-121 kubenswrapper[2568]: W0728 15:33:40.097299    2568 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 28 15:33:40.102932 ip-10-0-131-121 kubenswrapper[2568]: W0728 15:33:40.097302    2568 feature_gate.go:328] unrecognized feature gate: Example2
Jul 28 15:33:40.102932 ip-10-0-131-121 kubenswrapper[2568]: W0728 15:33:40.097306    2568 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 28 15:33:40.102932 ip-10-0-131-121 kubenswrapper[2568]: W0728 15:33:40.097309    2568 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 28 15:33:40.102932 ip-10-0-131-121 kubenswrapper[2568]: W0728 15:33:40.097311    2568 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 28 15:33:40.102932 ip-10-0-131-121 kubenswrapper[2568]: W0728 15:33:40.097314    2568 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 28 15:33:40.102932 ip-10-0-131-121 kubenswrapper[2568]: W0728 15:33:40.097317    2568 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 28 15:33:40.102932 ip-10-0-131-121 kubenswrapper[2568]: W0728 15:33:40.097319    2568 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 28 15:33:40.102932 ip-10-0-131-121 kubenswrapper[2568]: W0728 15:33:40.097322    2568 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 28 15:33:40.102932 ip-10-0-131-121 kubenswrapper[2568]: W0728 15:33:40.097325    2568 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 28 15:33:40.102932 ip-10-0-131-121 kubenswrapper[2568]: W0728 15:33:40.097328    2568 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 28 15:33:40.102932 ip-10-0-131-121 kubenswrapper[2568]: W0728 15:33:40.097332    2568 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 28 15:33:40.102932 ip-10-0-131-121 kubenswrapper[2568]: W0728 15:33:40.097335    2568 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 28 15:33:40.102932 ip-10-0-131-121 kubenswrapper[2568]: W0728 15:33:40.097339    2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 28 15:33:40.102932 ip-10-0-131-121 kubenswrapper[2568]: W0728 15:33:40.097344    2568 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 28 15:33:40.102932 ip-10-0-131-121 kubenswrapper[2568]: W0728 15:33:40.097349    2568 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 28 15:33:40.102932 ip-10-0-131-121 kubenswrapper[2568]: W0728 15:33:40.097353    2568 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 28 15:33:40.103410 ip-10-0-131-121 kubenswrapper[2568]: W0728 15:33:40.097356    2568 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 28 15:33:40.103410 ip-10-0-131-121 kubenswrapper[2568]: W0728 15:33:40.097360    2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 28 15:33:40.103410 ip-10-0-131-121 kubenswrapper[2568]: W0728 15:33:40.097362    2568 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 28 15:33:40.103410 ip-10-0-131-121 kubenswrapper[2568]: W0728 15:33:40.097371    2568 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 28 15:33:40.103410 ip-10-0-131-121 kubenswrapper[2568]: W0728 15:33:40.097375    2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 28 15:33:40.103410 ip-10-0-131-121 kubenswrapper[2568]: W0728 15:33:40.097378    2568 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 28 15:33:40.103410 ip-10-0-131-121 kubenswrapper[2568]: W0728 15:33:40.097382    2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 28 15:33:40.103410 ip-10-0-131-121 kubenswrapper[2568]: W0728 15:33:40.097385    2568 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 28 15:33:40.103410 ip-10-0-131-121 kubenswrapper[2568]: W0728 15:33:40.097388    2568 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 28 15:33:40.103410 ip-10-0-131-121 kubenswrapper[2568]: W0728 15:33:40.097391    2568 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 28 15:33:40.103410 ip-10-0-131-121 kubenswrapper[2568]: W0728 15:33:40.097393    2568 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 28 15:33:40.103410 ip-10-0-131-121 kubenswrapper[2568]: W0728 15:33:40.097396    2568 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 28 15:33:40.103410 ip-10-0-131-121 kubenswrapper[2568]: W0728 15:33:40.097398    2568 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 28 15:33:40.103410 ip-10-0-131-121 kubenswrapper[2568]: W0728 15:33:40.097401    2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 28 15:33:40.103410 ip-10-0-131-121 kubenswrapper[2568]: W0728 15:33:40.097403    2568 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 28 15:33:40.103410 ip-10-0-131-121 kubenswrapper[2568]: W0728 15:33:40.097406    2568 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 28 15:33:40.103410 ip-10-0-131-121 kubenswrapper[2568]: W0728 15:33:40.097408    2568 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 28 15:33:40.103410 ip-10-0-131-121 kubenswrapper[2568]: W0728 15:33:40.097411    2568 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 28 15:33:40.103410 ip-10-0-131-121 kubenswrapper[2568]: W0728 15:33:40.097413    2568 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 28 15:33:40.103932 ip-10-0-131-121 kubenswrapper[2568]: W0728 15:33:40.097416    2568 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 28 15:33:40.103932 ip-10-0-131-121 kubenswrapper[2568]: W0728 15:33:40.097421    2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 28 15:33:40.103932 ip-10-0-131-121 kubenswrapper[2568]: W0728 15:33:40.097424    2568 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 28 15:33:40.103932 ip-10-0-131-121 kubenswrapper[2568]: W0728 15:33:40.097429    2568 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 28 15:33:40.103932 ip-10-0-131-121 kubenswrapper[2568]: W0728 15:33:40.097433    2568 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 28 15:33:40.103932 ip-10-0-131-121 kubenswrapper[2568]: W0728 15:33:40.097436    2568 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 28 15:33:40.103932 ip-10-0-131-121 kubenswrapper[2568]: W0728 15:33:40.097440    2568 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 28 15:33:40.103932 ip-10-0-131-121 kubenswrapper[2568]: W0728 15:33:40.097443    2568 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 28 15:33:40.103932 ip-10-0-131-121 kubenswrapper[2568]: W0728 15:33:40.097446    2568 feature_gate.go:328] unrecognized feature gate: Example
Jul 28 15:33:40.103932 ip-10-0-131-121 kubenswrapper[2568]: W0728 15:33:40.097449    2568 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 28 15:33:40.103932 ip-10-0-131-121 kubenswrapper[2568]: W0728 15:33:40.097451    2568 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 28 15:33:40.103932 ip-10-0-131-121 kubenswrapper[2568]: W0728 15:33:40.097454    2568 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 28 15:33:40.103932 ip-10-0-131-121 kubenswrapper[2568]: W0728 15:33:40.097456    2568 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 28 15:33:40.103932 ip-10-0-131-121 kubenswrapper[2568]: W0728 15:33:40.097459    2568 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 28 15:33:40.103932 ip-10-0-131-121 kubenswrapper[2568]: W0728 15:33:40.097461    2568 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 28 15:33:40.103932 ip-10-0-131-121 kubenswrapper[2568]: W0728 15:33:40.097463    2568 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 28 15:33:40.103932 ip-10-0-131-121 kubenswrapper[2568]: W0728 15:33:40.097466    2568 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 28 15:33:40.103932 ip-10-0-131-121 kubenswrapper[2568]: W0728 15:33:40.097470    2568 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 28 15:33:40.103932 ip-10-0-131-121 kubenswrapper[2568]: W0728 15:33:40.097473    2568 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 28 15:33:40.104389 ip-10-0-131-121 kubenswrapper[2568]: W0728 15:33:40.097476    2568 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 28 15:33:40.104389 ip-10-0-131-121 kubenswrapper[2568]: W0728 15:33:40.097478    2568 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 28 15:33:40.104389 ip-10-0-131-121 kubenswrapper[2568]: W0728 15:33:40.097481    2568 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 28 15:33:40.104389 ip-10-0-131-121 kubenswrapper[2568]: W0728 15:33:40.097484    2568 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 28 15:33:40.104389 ip-10-0-131-121 kubenswrapper[2568]: W0728 15:33:40.097486    2568 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 28 15:33:40.104389 ip-10-0-131-121 kubenswrapper[2568]: W0728 15:33:40.097489    2568 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 28 15:33:40.104389 ip-10-0-131-121 kubenswrapper[2568]: W0728 15:33:40.097491    2568 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 28 15:33:40.104389 ip-10-0-131-121 kubenswrapper[2568]: W0728 15:33:40.097494    2568 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 28 15:33:40.104389 ip-10-0-131-121 kubenswrapper[2568]: W0728 15:33:40.097496    2568 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 28 15:33:40.104389 ip-10-0-131-121 kubenswrapper[2568]: W0728 15:33:40.097500    2568 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 28 15:33:40.104389 ip-10-0-131-121 kubenswrapper[2568]: W0728 15:33:40.097505    2568 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 28 15:33:40.104389 ip-10-0-131-121 kubenswrapper[2568]: W0728 15:33:40.097509    2568 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 28 15:33:40.104389 ip-10-0-131-121 kubenswrapper[2568]: W0728 15:33:40.097513    2568 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 28 15:33:40.104389 ip-10-0-131-121 kubenswrapper[2568]: W0728 15:33:40.097516    2568 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 28 15:33:40.104389 ip-10-0-131-121 kubenswrapper[2568]: W0728 15:33:40.097519    2568 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 28 15:33:40.104389 ip-10-0-131-121 kubenswrapper[2568]: W0728 15:33:40.097521    2568 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 28 15:33:40.104389 ip-10-0-131-121 kubenswrapper[2568]: W0728 15:33:40.097524    2568 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 28 15:33:40.104389 ip-10-0-131-121 kubenswrapper[2568]: W0728 15:33:40.097526    2568 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 28 15:33:40.104389 ip-10-0-131-121 kubenswrapper[2568]: W0728 15:33:40.097529    2568 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 28 15:33:40.104389 ip-10-0-131-121 kubenswrapper[2568]: W0728 15:33:40.097531    2568 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 28 15:33:40.105007 ip-10-0-131-121 kubenswrapper[2568]: W0728 15:33:40.097534    2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 28 15:33:40.105007 ip-10-0-131-121 kubenswrapper[2568]: W0728 15:33:40.097536    2568 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 28 15:33:40.105007 ip-10-0-131-121 kubenswrapper[2568]: W0728 15:33:40.097539    2568 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 28 15:33:40.105007 ip-10-0-131-121 kubenswrapper[2568]: W0728 15:33:40.097541    2568 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 28 15:33:40.105007 ip-10-0-131-121 kubenswrapper[2568]: W0728 15:33:40.097543    2568 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 28 15:33:40.105007 ip-10-0-131-121 kubenswrapper[2568]: W0728 15:33:40.097546    2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 28 15:33:40.105007 ip-10-0-131-121 kubenswrapper[2568]: W0728 15:33:40.097548    2568 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 28 15:33:40.105007 ip-10-0-131-121 kubenswrapper[2568]: W0728 15:33:40.097551    2568 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 28 15:33:40.105007 ip-10-0-131-121 kubenswrapper[2568]: W0728 15:33:40.097553    2568 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 28 15:33:40.105007 ip-10-0-131-121 kubenswrapper[2568]: W0728 15:33:40.097556    2568 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 28 15:33:40.105007 ip-10-0-131-121 kubenswrapper[2568]: W0728 15:33:40.097559    2568 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 28 15:33:40.105007 ip-10-0-131-121 kubenswrapper[2568]: W0728 15:33:40.097562    2568 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 28 15:33:40.105007 ip-10-0-131-121 kubenswrapper[2568]: W0728 15:33:40.097564    2568 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 28 15:33:40.105007 ip-10-0-131-121 kubenswrapper[2568]: W0728 15:33:40.097567    2568 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 28 15:33:40.105007 ip-10-0-131-121 kubenswrapper[2568]: W0728 15:33:40.097570    2568 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 28 15:33:40.105007 ip-10-0-131-121 kubenswrapper[2568]: W0728 15:33:40.097572    2568 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 28 15:33:40.105007 ip-10-0-131-121 kubenswrapper[2568]: W0728 15:33:40.097575    2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 28 15:33:40.105007 ip-10-0-131-121 kubenswrapper[2568]: W0728 15:33:40.097581    2568 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 28 15:33:40.105007 ip-10-0-131-121 kubenswrapper[2568]: W0728 15:33:40.097586    2568 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 28 15:33:40.105007 ip-10-0-131-121 kubenswrapper[2568]: W0728 15:33:40.097591    2568 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 28 15:33:40.105481 ip-10-0-131-121 kubenswrapper[2568]: W0728 15:33:40.097595    2568 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 28 15:33:40.105481 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:33:40.097599    2568 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 28 15:33:40.105481 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:33:40.097733    2568 server.go:962] "Client rotation is on, will bootstrap in background"
Jul 28 15:33:40.105481 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:33:40.099792    2568 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Jul 28 15:33:40.105481 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:33:40.101340    2568 server.go:1019] "Starting client certificate rotation"
Jul 28 15:33:40.105481 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:33:40.101442    2568 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 28 15:33:40.105481 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:33:40.101477    2568 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 28 15:33:40.105481 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:33:40.103415    2568 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 28 15:33:40.105481 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:33:40.104551    2568 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 28 15:33:40.115571 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:33:40.115550    2568 log.go:25] "Validated CRI v1 runtime API"
Jul 28 15:33:40.117554 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:33:40.117535    2568 log.go:25] "Validated CRI v1 image API"
Jul 28 15:33:40.118428 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:33:40.118406    2568 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Jul 28 15:33:40.122591 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:33:40.122562    2568 fs.go:135] Filesystem UUIDs: map[181d6476-ba79-4cf4-baf7-20b3ec7932cb:/dev/nvme0n1p4 6f443fc6-5fe9-40e7-aa62-5beace16d5a3:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2]
Jul 28 15:33:40.122672 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:33:40.122591    2568 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Jul 28 15:33:40.126962 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:33:40.126829    2568 manager.go:216] Machine: {Timestamp:2026-07-28 15:33:40.126314011 +0000 UTC m=+0.311509171 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3104387 MemoryCapacity:33164496896 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec26d5d56308e73a6f18d0259eb81ffd SystemUUID:ec26d5d5-6308-e73a-6f18-d0259eb81ffd BootID:05f3d200-7463-4fe3-8dce-85b69f467cc6 Filesystems:[{Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582246400 Type:vfs Inodes:4048400 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632902656 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582250496 Type:vfs Inodes:1048576 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:c9:79:cb:08:71 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:c9:79:cb:08:71 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:be:c2:11:6f:4b:cb Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164496896 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Jul 28 15:33:40.126962 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:33:40.126947    2568 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Jul 28 15:33:40.127138 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:33:40.127054    2568 manager.go:232] Version: {KernelVersion:5.14.0-570.129.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260717-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Jul 28 15:33:40.129721 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:33:40.129691    2568 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Jul 28 15:33:40.129923 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:33:40.129726    2568 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-131-121.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Jul 28 15:33:40.130479 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:33:40.130467    2568 topology_manager.go:138] "Creating topology manager with none policy"
Jul 28 15:33:40.130544 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:33:40.130485    2568 container_manager_linux.go:309] "Creating device plugin manager"
Jul 28 15:33:40.130544 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:33:40.130496    2568 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 28 15:33:40.131225 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:33:40.131213    2568 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 28 15:33:40.132452 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:33:40.132439    2568 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Jul 28 15:33:40.134638 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:33:40.134620    2568 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 28 15:33:40.136287 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:33:40.136253    2568 state_mem.go:36] "Initialized new in-memory state store"
Jul 28 15:33:40.136428 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:33:40.136417    2568 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Jul 28 15:33:40.136485 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:33:40.136458    2568 kubelet.go:486] "Attempting to sync node with API server"
Jul 28 15:33:40.136485 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:33:40.136481    2568 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Jul 28 15:33:40.136567 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:33:40.136503    2568 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Jul 28 15:33:40.136567 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:33:40.136518    2568 kubelet.go:398] "Adding apiserver pod source"
Jul 28 15:33:40.136567 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:33:40.136545    2568 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Jul 28 15:33:40.137019 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:33:40.137004    2568 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 28 15:33:40.137100 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:33:40.137029    2568 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 28 15:33:40.139980 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:33:40.139964    2568 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.10-2.rhaos4.21.git83ee25b.el9" apiVersion="v1"
Jul 28 15:33:40.141701 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:33:40.141679    2568 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Jul 28 15:33:40.141801 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:33:40.141727    2568 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Jul 28 15:33:40.142435 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:33:40.142412    2568 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Jul 28 15:33:40.142518 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:33:40.142446    2568 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Jul 28 15:33:40.142518 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:33:40.142456    2568 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Jul 28 15:33:40.142518 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:33:40.142465    2568 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Jul 28 15:33:40.142518 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:33:40.142479    2568 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Jul 28 15:33:40.142518 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:33:40.142489    2568 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Jul 28 15:33:40.142518 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:33:40.142501    2568 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Jul 28 15:33:40.142518 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:33:40.142509    2568 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Jul 28 15:33:40.142518 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:33:40.142520    2568 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Jul 28 15:33:40.142866 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:33:40.142529    2568 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Jul 28 15:33:40.142866 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:33:40.142564    2568 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Jul 28 15:33:40.142866 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:33:40.142573    2568 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Jul 28 15:33:40.143464 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:33:40.143448    2568 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Jul 28 15:33:40.143558 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:33:40.143530    2568 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Jul 28 15:33:40.146634 ip-10-0-131-121 kubenswrapper[2568]: E0728 15:33:40.146609    2568 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Jul 28 15:33:40.146716 ip-10-0-131-121 kubenswrapper[2568]: E0728 15:33:40.146637    2568 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-131-121.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Jul 28 15:33:40.146835 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:33:40.146821    2568 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-131-121.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Jul 28 15:33:40.147052 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:33:40.147040    2568 server.go:1268] "Started kubelet"
Jul 28 15:33:40.147161 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:33:40.147111    2568 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Jul 28 15:33:40.147625 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:33:40.147110    2568 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Jul 28 15:33:40.147625 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:33:40.147339    2568 server_v1.go:49] "podresources" method="list" useActivePods=true
Jul 28 15:33:40.147979 ip-10-0-131-121 systemd[1]: Started Kubernetes Kubelet.
Jul 28 15:33:40.149083 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:33:40.149070    2568 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Jul 28 15:33:40.149985 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:33:40.149973    2568 server.go:310] "Adding debug handlers to kubelet server"
Jul 28 15:33:40.154831 ip-10-0-131-121 kubenswrapper[2568]: E0728 15:33:40.154813    2568 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Jul 28 15:33:40.155264 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:33:40.155249    2568 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Jul 28 15:33:40.155264 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:33:40.155256    2568 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Jul 28 15:33:40.155358 ip-10-0-131-121 kubenswrapper[2568]: E0728 15:33:40.155324    2568 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-121.ec2.internal\" not found"
Jul 28 15:33:40.155736 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:33:40.155718    2568 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Jul 28 15:33:40.155829 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:33:40.155796    2568 volume_manager.go:311] "The desired_state_of_world populator starts"
Jul 28 15:33:40.155829 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:33:40.155815    2568 volume_manager.go:313] "Starting Kubelet Volume Manager"
Jul 28 15:33:40.155971 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:33:40.155947    2568 reconstruct.go:97] "Volume reconstruction finished"
Jul 28 15:33:40.155971 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:33:40.155955    2568 reconciler.go:29] "Reconciler: start to sync state"
Jul 28 15:33:40.156140 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:33:40.156125    2568 factory.go:55] Registering systemd factory
Jul 28 15:33:40.156218 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:33:40.156156    2568 factory.go:223] Registration of the systemd container factory successfully
Jul 28 15:33:40.156376 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:33:40.156360    2568 factory.go:153] Registering CRI-O factory
Jul 28 15:33:40.156430 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:33:40.156378    2568 factory.go:223] Registration of the crio container factory successfully
Jul 28 15:33:40.156467 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:33:40.156432    2568 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Jul 28 15:33:40.156467 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:33:40.156457    2568 factory.go:103] Registering Raw factory
Jul 28 15:33:40.156565 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:33:40.156474    2568 manager.go:1196] Started watching for new ooms in manager
Jul 28 15:33:40.156914 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:33:40.156897    2568 manager.go:349] Starting recovery of all containers
Jul 28 15:33:40.157903 ip-10-0-131-121 kubenswrapper[2568]: E0728 15:33:40.156961    2568 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-131-121.ec2.internal.18c67dfb332087bc  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-131-121.ec2.internal,UID:ip-10-0-131-121.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-131-121.ec2.internal,},FirstTimestamp:2026-07-28 15:33:40.147013564 +0000 UTC m=+0.332208724,LastTimestamp:2026-07-28 15:33:40.147013564 +0000 UTC m=+0.332208724,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-131-121.ec2.internal,}"
Jul 28 15:33:40.166611 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:33:40.166595    2568 manager.go:354] Recovery completed
Jul 28 15:33:40.168670 ip-10-0-131-121 kubenswrapper[2568]: E0728 15:33:40.168639    2568 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-131-121.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Jul 28 15:33:40.168804 ip-10-0-131-121 kubenswrapper[2568]: E0728 15:33:40.168762    2568 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Jul 28 15:33:40.172506 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:33:40.172492    2568 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 28 15:33:40.174395 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:33:40.174372    2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-121.ec2.internal" event="NodeHasSufficientMemory"
Jul 28 15:33:40.174480 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:33:40.174410    2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-121.ec2.internal" event="NodeHasNoDiskPressure"
Jul 28 15:33:40.174480 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:33:40.174420    2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-121.ec2.internal" event="NodeHasSufficientPID"
Jul 28 15:33:40.174981 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:33:40.174967    2568 cpu_manager.go:222] "Starting CPU manager" policy="none"
Jul 28 15:33:40.174981 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:33:40.174976    2568 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 28 15:33:40.175084 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:33:40.174982    2568 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Jul 28 15:33:40.175084 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:33:40.175001    2568 state_mem.go:36] "Initialized new in-memory state store"
Jul 28 15:33:40.175721 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:33:40.175703    2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-121.ec2.internal" event="NodeHasSufficientMemory"
Jul 28 15:33:40.175768 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:33:40.175740    2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-121.ec2.internal" event="NodeHasNoDiskPressure"
Jul 28 15:33:40.175768 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:33:40.175753    2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-121.ec2.internal" event="NodeHasSufficientPID"
Jul 28 15:33:40.177271 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:33:40.177253    2568 policy_none.go:49] "None policy: Start"
Jul 28 15:33:40.177336 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:33:40.177278    2568 memory_manager.go:187] "Starting memorymanager" policy="None"
Jul 28 15:33:40.177336 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:33:40.177294    2568 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Jul 28 15:33:40.179042 ip-10-0-131-121 kubenswrapper[2568]: E0728 15:33:40.178966    2568 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-131-121.ec2.internal.18c67dfb34c258a3  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-131-121.ec2.internal,UID:ip-10-0-131-121.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-131-121.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-131-121.ec2.internal,},FirstTimestamp:2026-07-28 15:33:40.174395555 +0000 UTC m=+0.359590718,LastTimestamp:2026-07-28 15:33:40.174395555 +0000 UTC m=+0.359590718,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-131-121.ec2.internal,}"
Jul 28 15:33:40.179322 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:33:40.179309    2568 policy_none.go:47] "Start"
Jul 28 15:33:40.191787 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:33:40.191629    2568 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-rbzwv"
Jul 28 15:33:40.192885 ip-10-0-131-121 kubenswrapper[2568]: E0728 15:33:40.192813    2568 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-131-121.ec2.internal.18c67dfb34c2a4f1  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-131-121.ec2.internal,UID:ip-10-0-131-121.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-131-121.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-131-121.ec2.internal,},FirstTimestamp:2026-07-28 15:33:40.174415089 +0000 UTC m=+0.359610249,LastTimestamp:2026-07-28 15:33:40.174415089 +0000 UTC m=+0.359610249,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-131-121.ec2.internal,}"
Jul 28 15:33:40.209597 ip-10-0-131-121 kubenswrapper[2568]: E0728 15:33:40.209463    2568 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-131-121.ec2.internal.18c67dfb34c2c9e0  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-131-121.ec2.internal,UID:ip-10-0-131-121.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-131-121.ec2.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-131-121.ec2.internal,},FirstTimestamp:2026-07-28 15:33:40.174424544 +0000 UTC m=+0.359619704,LastTimestamp:2026-07-28 15:33:40.174424544 +0000 UTC m=+0.359619704,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-131-121.ec2.internal,}"
Jul 28 15:33:40.210837 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:33:40.210818    2568 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-rbzwv"
Jul 28 15:33:40.213500 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:33:40.213486    2568 manager.go:337] "Starting Device Plugin manager"
Jul 28 15:33:40.213577 ip-10-0-131-121 kubenswrapper[2568]: E0728 15:33:40.213563    2568 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Jul 28 15:33:40.213610 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:33:40.213580    2568 server.go:87] "Starting device plugin registration server"
Jul 28 15:33:40.213883 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:33:40.213869    2568 eviction_manager.go:189] "Eviction manager: starting control loop"
Jul 28 15:33:40.213965 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:33:40.213883    2568 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Jul 28 15:33:40.214072 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:33:40.213973    2568 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Jul 28 15:33:40.214072 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:33:40.214050    2568 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Jul 28 15:33:40.214072 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:33:40.214057    2568 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Jul 28 15:33:40.214650 ip-10-0-131-121 kubenswrapper[2568]: E0728 15:33:40.214630    2568 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Jul 28 15:33:40.214735 ip-10-0-131-121 kubenswrapper[2568]: E0728 15:33:40.214684    2568 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-131-121.ec2.internal\" not found"
Jul 28 15:33:40.287699 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:33:40.287659    2568 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Jul 28 15:33:40.289862 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:33:40.288868    2568 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Jul 28 15:33:40.289862 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:33:40.288904    2568 status_manager.go:244] "Starting to sync pod status with apiserver"
Jul 28 15:33:40.289862 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:33:40.288928    2568 kubelet.go:2448] "Starting kubelet main sync loop"
Jul 28 15:33:40.289862 ip-10-0-131-121 kubenswrapper[2568]: E0728 15:33:40.288965    2568 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Jul 28 15:33:40.292524 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:33:40.292506    2568 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 28 15:33:40.314364 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:33:40.314340    2568 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 28 15:33:40.315629 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:33:40.315607    2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-121.ec2.internal" event="NodeHasSufficientMemory"
Jul 28 15:33:40.315694 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:33:40.315646    2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-121.ec2.internal" event="NodeHasNoDiskPressure"
Jul 28 15:33:40.315694 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:33:40.315661    2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-121.ec2.internal" event="NodeHasSufficientPID"
Jul 28 15:33:40.315694 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:33:40.315686    2568 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-131-121.ec2.internal"
Jul 28 15:33:40.329291 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:33:40.329268    2568 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-131-121.ec2.internal"
Jul 28 15:33:40.329356 ip-10-0-131-121 kubenswrapper[2568]: E0728 15:33:40.329301    2568 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-131-121.ec2.internal\": node \"ip-10-0-131-121.ec2.internal\" not found"
Jul 28 15:33:40.346850 ip-10-0-131-121 kubenswrapper[2568]: E0728 15:33:40.346823    2568 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-121.ec2.internal\" not found"
Jul 28 15:33:40.389263 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:33:40.389225    2568 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-121.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-131-121.ec2.internal"]
Jul 28 15:33:40.389414 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:33:40.389304    2568 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 28 15:33:40.390283 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:33:40.390268    2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-121.ec2.internal" event="NodeHasSufficientMemory"
Jul 28 15:33:40.390344 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:33:40.390298    2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-121.ec2.internal" event="NodeHasNoDiskPressure"
Jul 28 15:33:40.390344 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:33:40.390310    2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-121.ec2.internal" event="NodeHasSufficientPID"
Jul 28 15:33:40.392616 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:33:40.392601    2568 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 28 15:33:40.392741 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:33:40.392724    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-121.ec2.internal"
Jul 28 15:33:40.392804 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:33:40.392769    2568 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 28 15:33:40.393367 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:33:40.393350    2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-121.ec2.internal" event="NodeHasSufficientMemory"
Jul 28 15:33:40.393461 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:33:40.393376    2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-121.ec2.internal" event="NodeHasNoDiskPressure"
Jul 28 15:33:40.393461 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:33:40.393378    2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-121.ec2.internal" event="NodeHasSufficientMemory"
Jul 28 15:33:40.393461 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:33:40.393390    2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-121.ec2.internal" event="NodeHasSufficientPID"
Jul 28 15:33:40.393461 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:33:40.393401    2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-121.ec2.internal" event="NodeHasNoDiskPressure"
Jul 28 15:33:40.393461 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:33:40.393412    2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-121.ec2.internal" event="NodeHasSufficientPID"
Jul 28 15:33:40.396295 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:33:40.396278    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-121.ec2.internal"
Jul 28 15:33:40.396352 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:33:40.396309    2568 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 28 15:33:40.397288 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:33:40.397271    2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-121.ec2.internal" event="NodeHasSufficientMemory"
Jul 28 15:33:40.397396 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:33:40.397299    2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-121.ec2.internal" event="NodeHasNoDiskPressure"
Jul 28 15:33:40.397396 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:33:40.397308    2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-121.ec2.internal" event="NodeHasSufficientPID"
Jul 28 15:33:40.419545 ip-10-0-131-121 kubenswrapper[2568]: E0728 15:33:40.419513    2568 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-131-121.ec2.internal\" not found" node="ip-10-0-131-121.ec2.internal"
Jul 28 15:33:40.423957 ip-10-0-131-121 kubenswrapper[2568]: E0728 15:33:40.423936    2568 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-131-121.ec2.internal\" not found" node="ip-10-0-131-121.ec2.internal"
Jul 28 15:33:40.447168 ip-10-0-131-121 kubenswrapper[2568]: E0728 15:33:40.447144    2568 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-121.ec2.internal\" not found"
Jul 28 15:33:40.547637 ip-10-0-131-121 kubenswrapper[2568]: E0728 15:33:40.547571    2568 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-121.ec2.internal\" not found"
Jul 28 15:33:40.557237 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:33:40.557216    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/c3ed888f968446d5f752a5eb72daf11a-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-131-121.ec2.internal\" (UID: \"c3ed888f968446d5f752a5eb72daf11a\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-121.ec2.internal"
Jul 28 15:33:40.557293 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:33:40.557260    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c3ed888f968446d5f752a5eb72daf11a-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-131-121.ec2.internal\" (UID: \"c3ed888f968446d5f752a5eb72daf11a\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-121.ec2.internal"
Jul 28 15:33:40.557293 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:33:40.557279    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/0c5d5894bca76db08e7d990578b15cf0-config\") pod \"kube-apiserver-proxy-ip-10-0-131-121.ec2.internal\" (UID: \"0c5d5894bca76db08e7d990578b15cf0\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-131-121.ec2.internal"
Jul 28 15:33:40.647822 ip-10-0-131-121 kubenswrapper[2568]: E0728 15:33:40.647791    2568 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-121.ec2.internal\" not found"
Jul 28 15:33:40.658127 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:33:40.658103    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/c3ed888f968446d5f752a5eb72daf11a-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-131-121.ec2.internal\" (UID: \"c3ed888f968446d5f752a5eb72daf11a\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-121.ec2.internal"
Jul 28 15:33:40.658182 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:33:40.658140    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c3ed888f968446d5f752a5eb72daf11a-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-131-121.ec2.internal\" (UID: \"c3ed888f968446d5f752a5eb72daf11a\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-121.ec2.internal"
Jul 28 15:33:40.658182 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:33:40.658163    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/0c5d5894bca76db08e7d990578b15cf0-config\") pod \"kube-apiserver-proxy-ip-10-0-131-121.ec2.internal\" (UID: \"0c5d5894bca76db08e7d990578b15cf0\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-131-121.ec2.internal"
Jul 28 15:33:40.658239 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:33:40.658181    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/c3ed888f968446d5f752a5eb72daf11a-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-131-121.ec2.internal\" (UID: \"c3ed888f968446d5f752a5eb72daf11a\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-121.ec2.internal"
Jul 28 15:33:40.658239 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:33:40.658201    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/0c5d5894bca76db08e7d990578b15cf0-config\") pod \"kube-apiserver-proxy-ip-10-0-131-121.ec2.internal\" (UID: \"0c5d5894bca76db08e7d990578b15cf0\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-131-121.ec2.internal"
Jul 28 15:33:40.658239 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:33:40.658213    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c3ed888f968446d5f752a5eb72daf11a-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-131-121.ec2.internal\" (UID: \"c3ed888f968446d5f752a5eb72daf11a\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-121.ec2.internal"
Jul 28 15:33:40.722318 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:33:40.722283    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-121.ec2.internal"
Jul 28 15:33:40.727934 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:33:40.727524    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-121.ec2.internal"
Jul 28 15:33:40.748708 ip-10-0-131-121 kubenswrapper[2568]: E0728 15:33:40.748674    2568 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-121.ec2.internal\" not found"
Jul 28 15:33:40.849265 ip-10-0-131-121 kubenswrapper[2568]: E0728 15:33:40.849183    2568 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-121.ec2.internal\" not found"
Jul 28 15:33:40.949703 ip-10-0-131-121 kubenswrapper[2568]: E0728 15:33:40.949675    2568 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-121.ec2.internal\" not found"
Jul 28 15:33:41.050317 ip-10-0-131-121 kubenswrapper[2568]: E0728 15:33:41.050279    2568 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-121.ec2.internal\" not found"
Jul 28 15:33:41.100848 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:33:41.100784    2568 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Jul 28 15:33:41.101250 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:33:41.100936    2568 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Jul 28 15:33:41.150358 ip-10-0-131-121 kubenswrapper[2568]: E0728 15:33:41.150335    2568 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-121.ec2.internal\" not found"
Jul 28 15:33:41.166082 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:33:41.166059    2568 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 28 15:33:41.178008 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:33:41.177829    2568 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 28 15:33:41.216122 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:33:41.216084    2568 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-27 15:28:40 +0000 UTC" deadline="2028-05-04 22:56:26.395848048 +0000 UTC"
Jul 28 15:33:41.216122 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:33:41.216118    2568 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="15511h22m45.179734065s"
Jul 28 15:33:41.251024 ip-10-0-131-121 kubenswrapper[2568]: W0728 15:33:41.250996    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0c5d5894bca76db08e7d990578b15cf0.slice/crio-b4d3d8aebad99243927cc5159dd94fc66331d3f70d2c7963e5444098e994d488 WatchSource:0}: Error finding container b4d3d8aebad99243927cc5159dd94fc66331d3f70d2c7963e5444098e994d488: Status 404 returned error can't find the container with id b4d3d8aebad99243927cc5159dd94fc66331d3f70d2c7963e5444098e994d488
Jul 28 15:33:41.255623 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:33:41.255601    2568 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-121.ec2.internal"
Jul 28 15:33:41.257910 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:33:41.257659    2568 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 28 15:33:41.263880 ip-10-0-131-121 kubenswrapper[2568]: W0728 15:33:41.263852    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc3ed888f968446d5f752a5eb72daf11a.slice/crio-87742ce0b9c80bedc8c7f4c6ff230895a288be844577f9251fc91309cb81e1de WatchSource:0}: Error finding container 87742ce0b9c80bedc8c7f4c6ff230895a288be844577f9251fc91309cb81e1de: Status 404 returned error can't find the container with id 87742ce0b9c80bedc8c7f4c6ff230895a288be844577f9251fc91309cb81e1de
Jul 28 15:33:41.266653 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:33:41.266638    2568 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 28 15:33:41.268482 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:33:41.268468    2568 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-121.ec2.internal"
Jul 28 15:33:41.278325 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:33:41.278309    2568 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 28 15:33:41.292388 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:33:41.292345    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-121.ec2.internal" event={"ID":"c3ed888f968446d5f752a5eb72daf11a","Type":"ContainerStarted","Data":"87742ce0b9c80bedc8c7f4c6ff230895a288be844577f9251fc91309cb81e1de"}
Jul 28 15:33:41.293273 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:33:41.293249    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-121.ec2.internal" event={"ID":"0c5d5894bca76db08e7d990578b15cf0","Type":"ContainerStarted","Data":"b4d3d8aebad99243927cc5159dd94fc66331d3f70d2c7963e5444098e994d488"}
Jul 28 15:33:41.341514 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:33:41.341487    2568 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 28 15:33:42.084008 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:33:42.083970    2568 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 28 15:33:42.137412 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:33:42.137375    2568 apiserver.go:52] "Watching apiserver"
Jul 28 15:33:42.145535 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:33:42.145511    2568 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Jul 28 15:33:42.146001 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:33:42.145966    2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kube-system/global-pull-secret-syncer-tr647","kube-system/konnectivity-agent-22nh9","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2nwbt","openshift-cluster-node-tuning-operator/tuned-n2w6d","openshift-dns/node-resolver-whdfr","openshift-image-registry/node-ca-9hznr","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-121.ec2.internal","openshift-multus/multus-additional-cni-plugins-ptrdx","kube-system/kube-apiserver-proxy-ip-10-0-131-121.ec2.internal","openshift-multus/multus-nvbhc","openshift-multus/network-metrics-daemon-ckpr5","openshift-network-diagnostics/network-check-target-85ncq","openshift-network-operator/iptables-alerter-49vz8","openshift-ovn-kubernetes/ovnkube-node-2vrz4"]
Jul 28 15:33:42.148576 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:33:42.148548    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-nvbhc"
Jul 28 15:33:42.151007 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:33:42.150731    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-49vz8"
Jul 28 15:33:42.151307 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:33:42.151131    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Jul 28 15:33:42.151307 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:33:42.151179    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Jul 28 15:33:42.151307 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:33:42.151254    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Jul 28 15:33:42.151511 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:33:42.151394    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-2smfp\""
Jul 28 15:33:42.151511 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:33:42.151477    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Jul 28 15:33:42.153042 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:33:42.153022    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2nwbt"
Jul 28 15:33:42.153514 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:33:42.153493    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Jul 28 15:33:42.153646 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:33:42.153629    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-n7tjn\""
Jul 28 15:33:42.153646 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:33:42.153642    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Jul 28 15:33:42.153815 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:33:42.153800    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Jul 28 15:33:42.155322 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:33:42.155228    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-n2w6d"
Jul 28 15:33:42.155410 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:33:42.155329    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Jul 28 15:33:42.155945 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:33:42.155929    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-6g7g2\""
Jul 28 15:33:42.156351 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:33:42.156104    2568 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Jul 28 15:33:42.156351 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:33:42.156200    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Jul 28 15:33:42.156487 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:33:42.156403    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Jul 28 15:33:42.157375 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:33:42.157352    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Jul 28 15:33:42.157597 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:33:42.157577    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-2kwz4\""
Jul 28 15:33:42.157677 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:33:42.157602    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-whdfr"
Jul 28 15:33:42.158607 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:33:42.157935    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Jul 28 15:33:42.159946 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:33:42.159917    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-9hznr"
Jul 28 15:33:42.160374 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:33:42.160348    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Jul 28 15:33:42.160479 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:33:42.160462    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Jul 28 15:33:42.160586 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:33:42.160570    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-fqw5w\""
Jul 28 15:33:42.162234 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:33:42.162217    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Jul 28 15:33:42.162234 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:33:42.162231    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Jul 28 15:33:42.162466 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:33:42.162448    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-fm7f2\""
Jul 28 15:33:42.162538 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:33:42.162479    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Jul 28 15:33:42.162649 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:33:42.162634    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-2vrz4"
Jul 28 15:33:42.165046 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:33:42.165006    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Jul 28 15:33:42.165205 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:33:42.165185    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Jul 28 15:33:42.165551 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:33:42.165534    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-s2264\""
Jul 28 15:33:42.167113 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:33:42.165858    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Jul 28 15:33:42.167113 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:33:42.166032    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-ptrdx"
Jul 28 15:33:42.167113 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:33:42.166142    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Jul 28 15:33:42.167113 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:33:42.166651    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Jul 28 15:33:42.167113 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:33:42.167010    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Jul 28 15:33:42.167519 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:33:42.167135    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/ad8ebf58-b767-4341-b60e-868c5b8b637f-etc-systemd\") pod \"tuned-n2w6d\" (UID: \"ad8ebf58-b767-4341-b60e-868c5b8b637f\") " pod="openshift-cluster-node-tuning-operator/tuned-n2w6d"
Jul 28 15:33:42.167519 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:33:42.167162    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/ad8ebf58-b767-4341-b60e-868c5b8b637f-run\") pod \"tuned-n2w6d\" (UID: \"ad8ebf58-b767-4341-b60e-868c5b8b637f\") " pod="openshift-cluster-node-tuning-operator/tuned-n2w6d"
Jul 28 15:33:42.167519 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:33:42.167213    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ddv7g\" (UniqueName: \"kubernetes.io/projected/fab92b7b-ed62-4783-86be-ed4bd56a3128-kube-api-access-ddv7g\") pod \"node-resolver-whdfr\" (UID: \"fab92b7b-ed62-4783-86be-ed4bd56a3128\") " pod="openshift-dns/node-resolver-whdfr"
Jul 28 15:33:42.167519 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:33:42.167243    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/de4142bb-fcb0-460c-9fbf-1025c61e2015-iptables-alerter-script\") pod \"iptables-alerter-49vz8\" (UID: \"de4142bb-fcb0-460c-9fbf-1025c61e2015\") " pod="openshift-network-operator/iptables-alerter-49vz8"
Jul 28 15:33:42.167519 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:33:42.167268    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/ad8ebf58-b767-4341-b60e-868c5b8b637f-etc-sysconfig\") pod \"tuned-n2w6d\" (UID: \"ad8ebf58-b767-4341-b60e-868c5b8b637f\") " pod="openshift-cluster-node-tuning-operator/tuned-n2w6d"
Jul 28 15:33:42.167519 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:33:42.167293    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ad8ebf58-b767-4341-b60e-868c5b8b637f-etc-kubernetes\") pod \"tuned-n2w6d\" (UID: \"ad8ebf58-b767-4341-b60e-868c5b8b637f\") " pod="openshift-cluster-node-tuning-operator/tuned-n2w6d"
Jul 28 15:33:42.167519 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:33:42.167315    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ad8ebf58-b767-4341-b60e-868c5b8b637f-tmp\") pod \"tuned-n2w6d\" (UID: \"ad8ebf58-b767-4341-b60e-868c5b8b637f\") " pod="openshift-cluster-node-tuning-operator/tuned-n2w6d"
Jul 28 15:33:42.167519 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:33:42.167338    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/ad8ebf58-b767-4341-b60e-868c5b8b637f-host\") pod \"tuned-n2w6d\" (UID: \"ad8ebf58-b767-4341-b60e-868c5b8b637f\") " pod="openshift-cluster-node-tuning-operator/tuned-n2w6d"
Jul 28 15:33:42.167519 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:33:42.167405    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7e2caef2-3492-48d3-adbd-89fd3d4494d9-etc-kubernetes\") pod \"multus-nvbhc\" (UID: \"7e2caef2-3492-48d3-adbd-89fd3d4494d9\") " pod="openshift-multus/multus-nvbhc"
Jul 28 15:33:42.167519 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:33:42.167428    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fab92b7b-ed62-4783-86be-ed4bd56a3128-tmp-dir\") pod \"node-resolver-whdfr\" (UID: \"fab92b7b-ed62-4783-86be-ed4bd56a3128\") " pod="openshift-dns/node-resolver-whdfr"
Jul 28 15:33:42.167519 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:33:42.167450    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/7e2caef2-3492-48d3-adbd-89fd3d4494d9-cni-binary-copy\") pod \"multus-nvbhc\" (UID: \"7e2caef2-3492-48d3-adbd-89fd3d4494d9\") " pod="openshift-multus/multus-nvbhc"
Jul 28 15:33:42.167519 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:33:42.167473    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/7e2caef2-3492-48d3-adbd-89fd3d4494d9-host-run-k8s-cni-cncf-io\") pod \"multus-nvbhc\" (UID: \"7e2caef2-3492-48d3-adbd-89fd3d4494d9\") " pod="openshift-multus/multus-nvbhc"
Jul 28 15:33:42.167519 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:33:42.167498    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/7e2caef2-3492-48d3-adbd-89fd3d4494d9-host-run-netns\") pod \"multus-nvbhc\" (UID: \"7e2caef2-3492-48d3-adbd-89fd3d4494d9\") " pod="openshift-multus/multus-nvbhc"
Jul 28 15:33:42.167519 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:33:42.167523    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/7e2caef2-3492-48d3-adbd-89fd3d4494d9-host-var-lib-kubelet\") pod \"multus-nvbhc\" (UID: \"7e2caef2-3492-48d3-adbd-89fd3d4494d9\") " pod="openshift-multus/multus-nvbhc"
Jul 28 15:33:42.168251 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:33:42.167553    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/7e2caef2-3492-48d3-adbd-89fd3d4494d9-multus-conf-dir\") pod \"multus-nvbhc\" (UID: \"7e2caef2-3492-48d3-adbd-89fd3d4494d9\") " pod="openshift-multus/multus-nvbhc"
Jul 28 15:33:42.168251 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:33:42.167579    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/7e2caef2-3492-48d3-adbd-89fd3d4494d9-multus-daemon-config\") pod \"multus-nvbhc\" (UID: \"7e2caef2-3492-48d3-adbd-89fd3d4494d9\") " pod="openshift-multus/multus-nvbhc"
Jul 28 15:33:42.168251 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:33:42.167603    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/7e2caef2-3492-48d3-adbd-89fd3d4494d9-multus-cni-dir\") pod \"multus-nvbhc\" (UID: \"7e2caef2-3492-48d3-adbd-89fd3d4494d9\") " pod="openshift-multus/multus-nvbhc"
Jul 28 15:33:42.168251 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:33:42.167626    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/7e2caef2-3492-48d3-adbd-89fd3d4494d9-os-release\") pod \"multus-nvbhc\" (UID: \"7e2caef2-3492-48d3-adbd-89fd3d4494d9\") " pod="openshift-multus/multus-nvbhc"
Jul 28 15:33:42.168251 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:33:42.167661    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/7e2caef2-3492-48d3-adbd-89fd3d4494d9-host-var-lib-cni-bin\") pod \"multus-nvbhc\" (UID: \"7e2caef2-3492-48d3-adbd-89fd3d4494d9\") " pod="openshift-multus/multus-nvbhc"
Jul 28 15:33:42.168251 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:33:42.167710    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/17b1a488-5fd2-4cd5-871a-0f5d127493e2-socket-dir\") pod \"aws-ebs-csi-driver-node-2nwbt\" (UID: \"17b1a488-5fd2-4cd5-871a-0f5d127493e2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2nwbt"
Jul 28 15:33:42.168251 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:33:42.167734    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/7e2caef2-3492-48d3-adbd-89fd3d4494d9-host-run-multus-certs\") pod \"multus-nvbhc\" (UID: \"7e2caef2-3492-48d3-adbd-89fd3d4494d9\") " pod="openshift-multus/multus-nvbhc"
Jul 28 15:33:42.168251 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:33:42.167768    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/17b1a488-5fd2-4cd5-871a-0f5d127493e2-sys-fs\") pod \"aws-ebs-csi-driver-node-2nwbt\" (UID: \"17b1a488-5fd2-4cd5-871a-0f5d127493e2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2nwbt"
Jul 28 15:33:42.168251 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:33:42.167816    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ad8ebf58-b767-4341-b60e-868c5b8b637f-var-lib-kubelet\") pod \"tuned-n2w6d\" (UID: \"ad8ebf58-b767-4341-b60e-868c5b8b637f\") " pod="openshift-cluster-node-tuning-operator/tuned-n2w6d"
Jul 28 15:33:42.168251 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:33:42.167855    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/de4142bb-fcb0-460c-9fbf-1025c61e2015-host-slash\") pod \"iptables-alerter-49vz8\" (UID: \"de4142bb-fcb0-460c-9fbf-1025c61e2015\") " pod="openshift-network-operator/iptables-alerter-49vz8"
Jul 28 15:33:42.168251 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:33:42.167878    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/17b1a488-5fd2-4cd5-871a-0f5d127493e2-kubelet-dir\") pod \"aws-ebs-csi-driver-node-2nwbt\" (UID: \"17b1a488-5fd2-4cd5-871a-0f5d127493e2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2nwbt"
Jul 28 15:33:42.168251 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:33:42.167899    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/ad8ebf58-b767-4341-b60e-868c5b8b637f-etc-sysctl-conf\") pod \"tuned-n2w6d\" (UID: \"ad8ebf58-b767-4341-b60e-868c5b8b637f\") " pod="openshift-cluster-node-tuning-operator/tuned-n2w6d"
Jul 28 15:33:42.168251 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:33:42.167939    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/ad8ebf58-b767-4341-b60e-868c5b8b637f-etc-tuned\") pod \"tuned-n2w6d\" (UID: \"ad8ebf58-b767-4341-b60e-868c5b8b637f\") " pod="openshift-cluster-node-tuning-operator/tuned-n2w6d"
Jul 28 15:33:42.168251 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:33:42.167976    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/7e2caef2-3492-48d3-adbd-89fd3d4494d9-system-cni-dir\") pod \"multus-nvbhc\" (UID: \"7e2caef2-3492-48d3-adbd-89fd3d4494d9\") " pod="openshift-multus/multus-nvbhc"
Jul 28 15:33:42.168251 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:33:42.168010    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/7e2caef2-3492-48d3-adbd-89fd3d4494d9-cnibin\") pod \"multus-nvbhc\" (UID: \"7e2caef2-3492-48d3-adbd-89fd3d4494d9\") " pod="openshift-multus/multus-nvbhc"
Jul 28 15:33:42.168251 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:33:42.168035    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/7e2caef2-3492-48d3-adbd-89fd3d4494d9-multus-socket-dir-parent\") pod \"multus-nvbhc\" (UID: \"7e2caef2-3492-48d3-adbd-89fd3d4494d9\") " pod="openshift-multus/multus-nvbhc"
Jul 28 15:33:42.168251 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:33:42.168062    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/7e2caef2-3492-48d3-adbd-89fd3d4494d9-host-var-lib-cni-multus\") pod \"multus-nvbhc\" (UID: \"7e2caef2-3492-48d3-adbd-89fd3d4494d9\") " pod="openshift-multus/multus-nvbhc"
Jul 28 15:33:42.169017 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:33:42.168101    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/7e2caef2-3492-48d3-adbd-89fd3d4494d9-hostroot\") pod \"multus-nvbhc\" (UID: \"7e2caef2-3492-48d3-adbd-89fd3d4494d9\") " pod="openshift-multus/multus-nvbhc"
Jul 28 15:33:42.169017 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:33:42.168125    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/17b1a488-5fd2-4cd5-871a-0f5d127493e2-registration-dir\") pod \"aws-ebs-csi-driver-node-2nwbt\" (UID: \"17b1a488-5fd2-4cd5-871a-0f5d127493e2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2nwbt"
Jul 28 15:33:42.169017 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:33:42.168149    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/17b1a488-5fd2-4cd5-871a-0f5d127493e2-etc-selinux\") pod \"aws-ebs-csi-driver-node-2nwbt\" (UID: \"17b1a488-5fd2-4cd5-871a-0f5d127493e2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2nwbt"
Jul 28 15:33:42.169017 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:33:42.168177    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Jul 28 15:33:42.169017 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:33:42.168173    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/ad8ebf58-b767-4341-b60e-868c5b8b637f-etc-modprobe-d\") pod \"tuned-n2w6d\" (UID: \"ad8ebf58-b767-4341-b60e-868c5b8b637f\") " pod="openshift-cluster-node-tuning-operator/tuned-n2w6d"
Jul 28 15:33:42.169017 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:33:42.168236    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b297t\" (UniqueName: \"kubernetes.io/projected/de4142bb-fcb0-460c-9fbf-1025c61e2015-kube-api-access-b297t\") pod \"iptables-alerter-49vz8\" (UID: \"de4142bb-fcb0-460c-9fbf-1025c61e2015\") " pod="openshift-network-operator/iptables-alerter-49vz8"
Jul 28 15:33:42.169017 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:33:42.168281    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/17b1a488-5fd2-4cd5-871a-0f5d127493e2-device-dir\") pod \"aws-ebs-csi-driver-node-2nwbt\" (UID: \"17b1a488-5fd2-4cd5-871a-0f5d127493e2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2nwbt"
Jul 28 15:33:42.169017 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:33:42.168310    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zbhxp\" (UniqueName: \"kubernetes.io/projected/17b1a488-5fd2-4cd5-871a-0f5d127493e2-kube-api-access-zbhxp\") pod \"aws-ebs-csi-driver-node-2nwbt\" (UID: \"17b1a488-5fd2-4cd5-871a-0f5d127493e2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2nwbt"
Jul 28 15:33:42.169017 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:33:42.168353    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/ad8ebf58-b767-4341-b60e-868c5b8b637f-etc-sysctl-d\") pod \"tuned-n2w6d\" (UID: \"ad8ebf58-b767-4341-b60e-868c5b8b637f\") " pod="openshift-cluster-node-tuning-operator/tuned-n2w6d"
Jul 28 15:33:42.169430 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:33:42.169412    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-8qtkq\""
Jul 28 15:33:42.170458 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:33:42.170436    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ad8ebf58-b767-4341-b60e-868c5b8b637f-sys\") pod \"tuned-n2w6d\" (UID: \"ad8ebf58-b767-4341-b60e-868c5b8b637f\") " pod="openshift-cluster-node-tuning-operator/tuned-n2w6d"
Jul 28 15:33:42.170585 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:33:42.170555    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/ad8ebf58-b767-4341-b60e-868c5b8b637f-lib-modules\") pod \"tuned-n2w6d\" (UID: \"ad8ebf58-b767-4341-b60e-868c5b8b637f\") " pod="openshift-cluster-node-tuning-operator/tuned-n2w6d"
Jul 28 15:33:42.170661 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:33:42.170608    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m446d\" (UniqueName: \"kubernetes.io/projected/ad8ebf58-b767-4341-b60e-868c5b8b637f-kube-api-access-m446d\") pod \"tuned-n2w6d\" (UID: \"ad8ebf58-b767-4341-b60e-868c5b8b637f\") " pod="openshift-cluster-node-tuning-operator/tuned-n2w6d"
Jul 28 15:33:42.170712 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:33:42.170692    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-22nh9"
Jul 28 15:33:42.170763 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:33:42.170686    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/fab92b7b-ed62-4783-86be-ed4bd56a3128-hosts-file\") pod \"node-resolver-whdfr\" (UID: \"fab92b7b-ed62-4783-86be-ed4bd56a3128\") " pod="openshift-dns/node-resolver-whdfr"
Jul 28 15:33:42.170870 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:33:42.170823    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lqthn\" (UniqueName: \"kubernetes.io/projected/7e2caef2-3492-48d3-adbd-89fd3d4494d9-kube-api-access-lqthn\") pod \"multus-nvbhc\" (UID: \"7e2caef2-3492-48d3-adbd-89fd3d4494d9\") " pod="openshift-multus/multus-nvbhc"
Jul 28 15:33:42.171314 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:33:42.171285    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Jul 28 15:33:42.173709 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:33:42.173688    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-csbcc\""
Jul 28 15:33:42.174661 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:33:42.174373    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tr647"
Jul 28 15:33:42.174661 ip-10-0-131-121 kubenswrapper[2568]: E0728 15:33:42.174461    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tr647" podUID="3ba72ab7-7ab4-4e7f-ace4-8a6b7bbff25c"
Jul 28 15:33:42.175467 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:33:42.175085    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Jul 28 15:33:42.175467 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:33:42.175124    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Jul 28 15:33:42.177950 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:33:42.177364    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ckpr5"
Jul 28 15:33:42.177950 ip-10-0-131-121 kubenswrapper[2568]: E0728 15:33:42.177440    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ckpr5" podUID="6100b515-cfe9-47ef-b672-d698ae677bd0"
Jul 28 15:33:42.177950 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:33:42.177450    2568 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 28 15:33:42.179805 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:33:42.179788    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-85ncq"
Jul 28 15:33:42.179897 ip-10-0-131-121 kubenswrapper[2568]: E0728 15:33:42.179856    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-85ncq" podUID="644d231e-7d3c-4aef-a201-87266e83d526"
Jul 28 15:33:42.198761 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:33:42.198741    2568 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-lxr7n"
Jul 28 15:33:42.210164 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:33:42.210139    2568 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-lxr7n"
Jul 28 15:33:42.257904 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:33:42.257877    2568 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Jul 28 15:33:42.271126 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:33:42.271097    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ad8ebf58-b767-4341-b60e-868c5b8b637f-sys\") pod \"tuned-n2w6d\" (UID: \"ad8ebf58-b767-4341-b60e-868c5b8b637f\") " pod="openshift-cluster-node-tuning-operator/tuned-n2w6d"
Jul 28 15:33:42.271282 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:33:42.271140    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3ba72ab7-7ab4-4e7f-ace4-8a6b7bbff25c-original-pull-secret\") pod \"global-pull-secret-syncer-tr647\" (UID: \"3ba72ab7-7ab4-4e7f-ace4-8a6b7bbff25c\") " pod="kube-system/global-pull-secret-syncer-tr647"
Jul 28 15:33:42.271282 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:33:42.271171    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b297t\" (UniqueName: \"kubernetes.io/projected/de4142bb-fcb0-460c-9fbf-1025c61e2015-kube-api-access-b297t\") pod \"iptables-alerter-49vz8\" (UID: \"de4142bb-fcb0-460c-9fbf-1025c61e2015\") " pod="openshift-network-operator/iptables-alerter-49vz8"
Jul 28 15:33:42.271282 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:33:42.271195    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/17b1a488-5fd2-4cd5-871a-0f5d127493e2-device-dir\") pod \"aws-ebs-csi-driver-node-2nwbt\" (UID: \"17b1a488-5fd2-4cd5-871a-0f5d127493e2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2nwbt"
Jul 28 15:33:42.271282 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:33:42.271216    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ad8ebf58-b767-4341-b60e-868c5b8b637f-sys\") pod \"tuned-n2w6d\" (UID: \"ad8ebf58-b767-4341-b60e-868c5b8b637f\") " pod="openshift-cluster-node-tuning-operator/tuned-n2w6d"
Jul 28 15:33:42.271282 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:33:42.271220    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zbhxp\" (UniqueName: \"kubernetes.io/projected/17b1a488-5fd2-4cd5-871a-0f5d127493e2-kube-api-access-zbhxp\") pod \"aws-ebs-csi-driver-node-2nwbt\" (UID: \"17b1a488-5fd2-4cd5-871a-0f5d127493e2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2nwbt"
Jul 28 15:33:42.271282 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:33:42.271271    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/ad8ebf58-b767-4341-b60e-868c5b8b637f-lib-modules\") pod \"tuned-n2w6d\" (UID: \"ad8ebf58-b767-4341-b60e-868c5b8b637f\") " pod="openshift-cluster-node-tuning-operator/tuned-n2w6d"
Jul 28 15:33:42.271578 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:33:42.271311    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/dcab1041-8d3b-42e1-af05-55e847b831ad-cnibin\") pod \"multus-additional-cni-plugins-ptrdx\" (UID: \"dcab1041-8d3b-42e1-af05-55e847b831ad\") " pod="openshift-multus/multus-additional-cni-plugins-ptrdx"
Jul 28 15:33:42.271578 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:33:42.271341    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/dcab1041-8d3b-42e1-af05-55e847b831ad-tuning-conf-dir\") pod \"multus-additional-cni-plugins-ptrdx\" (UID: \"dcab1041-8d3b-42e1-af05-55e847b831ad\") " pod="openshift-multus/multus-additional-cni-plugins-ptrdx"
Jul 28 15:33:42.271578 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:33:42.271368    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/dcab1041-8d3b-42e1-af05-55e847b831ad-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-ptrdx\" (UID: \"dcab1041-8d3b-42e1-af05-55e847b831ad\") " pod="openshift-multus/multus-additional-cni-plugins-ptrdx"
Jul 28 15:33:42.271578 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:33:42.271394    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/ebf75c0d-1597-4a1c-b668-dc2a041eb35e-agent-certs\") pod \"konnectivity-agent-22nh9\" (UID: \"ebf75c0d-1597-4a1c-b668-dc2a041eb35e\") " pod="kube-system/konnectivity-agent-22nh9"
Jul 28 15:33:42.271578 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:33:42.271419    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/fab92b7b-ed62-4783-86be-ed4bd56a3128-hosts-file\") pod \"node-resolver-whdfr\" (UID: \"fab92b7b-ed62-4783-86be-ed4bd56a3128\") " pod="openshift-dns/node-resolver-whdfr"
Jul 28 15:33:42.271578 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:33:42.271443    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/ad8ebf58-b767-4341-b60e-868c5b8b637f-etc-systemd\") pod \"tuned-n2w6d\" (UID: \"ad8ebf58-b767-4341-b60e-868c5b8b637f\") " pod="openshift-cluster-node-tuning-operator/tuned-n2w6d"
Jul 28 15:33:42.271578 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:33:42.271467    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/3ba72ab7-7ab4-4e7f-ace4-8a6b7bbff25c-kubelet-config\") pod \"global-pull-secret-syncer-tr647\" (UID: \"3ba72ab7-7ab4-4e7f-ace4-8a6b7bbff25c\") " pod="kube-system/global-pull-secret-syncer-tr647"
Jul 28 15:33:42.271578 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:33:42.271491    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/dcab1041-8d3b-42e1-af05-55e847b831ad-os-release\") pod \"multus-additional-cni-plugins-ptrdx\" (UID: \"dcab1041-8d3b-42e1-af05-55e847b831ad\") " pod="openshift-multus/multus-additional-cni-plugins-ptrdx"
Jul 28 15:33:42.271578 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:33:42.271516    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/3622f792-27df-4604-a1e3-47192c7302cb-ovnkube-script-lib\") pod \"ovnkube-node-2vrz4\" (UID: \"3622f792-27df-4604-a1e3-47192c7302cb\") " pod="openshift-ovn-kubernetes/ovnkube-node-2vrz4"
Jul 28 15:33:42.271578 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:33:42.271542    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/de4142bb-fcb0-460c-9fbf-1025c61e2015-iptables-alerter-script\") pod \"iptables-alerter-49vz8\" (UID: \"de4142bb-fcb0-460c-9fbf-1025c61e2015\") " pod="openshift-network-operator/iptables-alerter-49vz8"
Jul 28 15:33:42.272083 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:33:42.271565    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ad8ebf58-b767-4341-b60e-868c5b8b637f-tmp\") pod \"tuned-n2w6d\" (UID: \"ad8ebf58-b767-4341-b60e-868c5b8b637f\") " pod="openshift-cluster-node-tuning-operator/tuned-n2w6d"
Jul 28 15:33:42.272083 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:33:42.271614    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/3622f792-27df-4604-a1e3-47192c7302cb-host-slash\") pod \"ovnkube-node-2vrz4\" (UID: \"3622f792-27df-4604-a1e3-47192c7302cb\") " pod="openshift-ovn-kubernetes/ovnkube-node-2vrz4"
Jul 28 15:33:42.272083 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:33:42.271629    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/17b1a488-5fd2-4cd5-871a-0f5d127493e2-device-dir\") pod \"aws-ebs-csi-driver-node-2nwbt\" (UID: \"17b1a488-5fd2-4cd5-871a-0f5d127493e2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2nwbt"
Jul 28 15:33:42.272083 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:33:42.271655    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/3622f792-27df-4604-a1e3-47192c7302cb-host-run-netns\") pod \"ovnkube-node-2vrz4\" (UID: \"3622f792-27df-4604-a1e3-47192c7302cb\") " pod="openshift-ovn-kubernetes/ovnkube-node-2vrz4"
Jul 28 15:33:42.272083 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:33:42.271670    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/fab92b7b-ed62-4783-86be-ed4bd56a3128-hosts-file\") pod \"node-resolver-whdfr\" (UID: \"fab92b7b-ed62-4783-86be-ed4bd56a3128\") " pod="openshift-dns/node-resolver-whdfr"
Jul 28 15:33:42.272083 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:33:42.271682    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/3622f792-27df-4604-a1e3-47192c7302cb-run-openvswitch\") pod \"ovnkube-node-2vrz4\" (UID: \"3622f792-27df-4604-a1e3-47192c7302cb\") " pod="openshift-ovn-kubernetes/ovnkube-node-2vrz4"
Jul 28 15:33:42.272083 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:33:42.271690    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/ad8ebf58-b767-4341-b60e-868c5b8b637f-etc-systemd\") pod \"tuned-n2w6d\" (UID: \"ad8ebf58-b767-4341-b60e-868c5b8b637f\") " pod="openshift-cluster-node-tuning-operator/tuned-n2w6d"
Jul 28 15:33:42.272083 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:33:42.271728    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/3622f792-27df-4604-a1e3-47192c7302cb-ovnkube-config\") pod \"ovnkube-node-2vrz4\" (UID: \"3622f792-27df-4604-a1e3-47192c7302cb\") " pod="openshift-ovn-kubernetes/ovnkube-node-2vrz4"
Jul 28 15:33:42.272083 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:33:42.271761    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/3622f792-27df-4604-a1e3-47192c7302cb-ovn-node-metrics-cert\") pod \"ovnkube-node-2vrz4\" (UID: \"3622f792-27df-4604-a1e3-47192c7302cb\") " pod="openshift-ovn-kubernetes/ovnkube-node-2vrz4"
Jul 28 15:33:42.272083 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:33:42.271816    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/ad8ebf58-b767-4341-b60e-868c5b8b637f-lib-modules\") pod \"tuned-n2w6d\" (UID: \"ad8ebf58-b767-4341-b60e-868c5b8b637f\") " pod="openshift-cluster-node-tuning-operator/tuned-n2w6d"
Jul 28 15:33:42.272083 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:33:42.271822    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/ad8ebf58-b767-4341-b60e-868c5b8b637f-host\") pod \"tuned-n2w6d\" (UID: \"ad8ebf58-b767-4341-b60e-868c5b8b637f\") " pod="openshift-cluster-node-tuning-operator/tuned-n2w6d"
Jul 28 15:33:42.272083 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:33:42.271858    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/dcab1041-8d3b-42e1-af05-55e847b831ad-system-cni-dir\") pod \"multus-additional-cni-plugins-ptrdx\" (UID: \"dcab1041-8d3b-42e1-af05-55e847b831ad\") " pod="openshift-multus/multus-additional-cni-plugins-ptrdx"
Jul 28 15:33:42.272083 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:33:42.271888    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6100b515-cfe9-47ef-b672-d698ae677bd0-metrics-certs\") pod \"network-metrics-daemon-ckpr5\" (UID: \"6100b515-cfe9-47ef-b672-d698ae677bd0\") " pod="openshift-multus/network-metrics-daemon-ckpr5"
Jul 28 15:33:42.272083 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:33:42.271957    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/ad8ebf58-b767-4341-b60e-868c5b8b637f-host\") pod \"tuned-n2w6d\" (UID: \"ad8ebf58-b767-4341-b60e-868c5b8b637f\") " pod="openshift-cluster-node-tuning-operator/tuned-n2w6d"
Jul 28 15:33:42.272083 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:33:42.271969    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/17b1a488-5fd2-4cd5-871a-0f5d127493e2-sys-fs\") pod \"aws-ebs-csi-driver-node-2nwbt\" (UID: \"17b1a488-5fd2-4cd5-871a-0f5d127493e2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2nwbt"
Jul 28 15:33:42.272083 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:33:42.272001    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/dcab1041-8d3b-42e1-af05-55e847b831ad-cni-binary-copy\") pod \"multus-additional-cni-plugins-ptrdx\" (UID: \"dcab1041-8d3b-42e1-af05-55e847b831ad\") " pod="openshift-multus/multus-additional-cni-plugins-ptrdx"
Jul 28 15:33:42.272083 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:33:42.271988    2568 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Jul 28 15:33:42.272857 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:33:42.272028    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/17b1a488-5fd2-4cd5-871a-0f5d127493e2-sys-fs\") pod \"aws-ebs-csi-driver-node-2nwbt\" (UID: \"17b1a488-5fd2-4cd5-871a-0f5d127493e2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2nwbt"
Jul 28 15:33:42.272857 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:33:42.272031    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7e2caef2-3492-48d3-adbd-89fd3d4494d9-etc-kubernetes\") pod \"multus-nvbhc\" (UID: \"7e2caef2-3492-48d3-adbd-89fd3d4494d9\") " pod="openshift-multus/multus-nvbhc"
Jul 28 15:33:42.272857 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:33:42.272073    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7e2caef2-3492-48d3-adbd-89fd3d4494d9-etc-kubernetes\") pod \"multus-nvbhc\" (UID: \"7e2caef2-3492-48d3-adbd-89fd3d4494d9\") " pod="openshift-multus/multus-nvbhc"
Jul 28 15:33:42.272857 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:33:42.272106    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/7e2caef2-3492-48d3-adbd-89fd3d4494d9-multus-conf-dir\") pod \"multus-nvbhc\" (UID: \"7e2caef2-3492-48d3-adbd-89fd3d4494d9\") " pod="openshift-multus/multus-nvbhc"
Jul 28 15:33:42.272857 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:33:42.272135    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/7e2caef2-3492-48d3-adbd-89fd3d4494d9-multus-cni-dir\") pod \"multus-nvbhc\" (UID: \"7e2caef2-3492-48d3-adbd-89fd3d4494d9\") " pod="openshift-multus/multus-nvbhc"
Jul 28 15:33:42.272857 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:33:42.272159    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/7e2caef2-3492-48d3-adbd-89fd3d4494d9-os-release\") pod \"multus-nvbhc\" (UID: \"7e2caef2-3492-48d3-adbd-89fd3d4494d9\") " pod="openshift-multus/multus-nvbhc"
Jul 28 15:33:42.272857 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:33:42.272163    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/de4142bb-fcb0-460c-9fbf-1025c61e2015-iptables-alerter-script\") pod \"iptables-alerter-49vz8\" (UID: \"de4142bb-fcb0-460c-9fbf-1025c61e2015\") " pod="openshift-network-operator/iptables-alerter-49vz8"
Jul 28 15:33:42.272857 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:33:42.272183    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/7e2caef2-3492-48d3-adbd-89fd3d4494d9-host-var-lib-cni-bin\") pod \"multus-nvbhc\" (UID: \"7e2caef2-3492-48d3-adbd-89fd3d4494d9\") " pod="openshift-multus/multus-nvbhc"
Jul 28 15:33:42.272857 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:33:42.272185    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/7e2caef2-3492-48d3-adbd-89fd3d4494d9-multus-conf-dir\") pod \"multus-nvbhc\" (UID: \"7e2caef2-3492-48d3-adbd-89fd3d4494d9\") " pod="openshift-multus/multus-nvbhc"
Jul 28 15:33:42.272857 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:33:42.272218    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/7e2caef2-3492-48d3-adbd-89fd3d4494d9-host-var-lib-cni-bin\") pod \"multus-nvbhc\" (UID: \"7e2caef2-3492-48d3-adbd-89fd3d4494d9\") " pod="openshift-multus/multus-nvbhc"
Jul 28 15:33:42.272857 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:33:42.272219    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ad8ebf58-b767-4341-b60e-868c5b8b637f-var-lib-kubelet\") pod \"tuned-n2w6d\" (UID: \"ad8ebf58-b767-4341-b60e-868c5b8b637f\") " pod="openshift-cluster-node-tuning-operator/tuned-n2w6d"
Jul 28 15:33:42.272857 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:33:42.272267    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ad8ebf58-b767-4341-b60e-868c5b8b637f-var-lib-kubelet\") pod \"tuned-n2w6d\" (UID: \"ad8ebf58-b767-4341-b60e-868c5b8b637f\") " pod="openshift-cluster-node-tuning-operator/tuned-n2w6d"
Jul 28 15:33:42.272857 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:33:42.272296    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ktx7j\" (UniqueName: \"kubernetes.io/projected/dcab1041-8d3b-42e1-af05-55e847b831ad-kube-api-access-ktx7j\") pod \"multus-additional-cni-plugins-ptrdx\" (UID: \"dcab1041-8d3b-42e1-af05-55e847b831ad\") " pod="openshift-multus/multus-additional-cni-plugins-ptrdx"
Jul 28 15:33:42.272857 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:33:42.272322    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/7e2caef2-3492-48d3-adbd-89fd3d4494d9-os-release\") pod \"multus-nvbhc\" (UID: \"7e2caef2-3492-48d3-adbd-89fd3d4494d9\") " pod="openshift-multus/multus-nvbhc"
Jul 28 15:33:42.272857 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:33:42.272327    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/3622f792-27df-4604-a1e3-47192c7302cb-host-cni-netd\") pod \"ovnkube-node-2vrz4\" (UID: \"3622f792-27df-4604-a1e3-47192c7302cb\") " pod="openshift-ovn-kubernetes/ovnkube-node-2vrz4"
Jul 28 15:33:42.272857 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:33:42.272339    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/7e2caef2-3492-48d3-adbd-89fd3d4494d9-multus-cni-dir\") pod \"multus-nvbhc\" (UID: \"7e2caef2-3492-48d3-adbd-89fd3d4494d9\") " pod="openshift-multus/multus-nvbhc"
Jul 28 15:33:42.272857 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:33:42.272355    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/ad8ebf58-b767-4341-b60e-868c5b8b637f-etc-sysctl-conf\") pod \"tuned-n2w6d\" (UID: \"ad8ebf58-b767-4341-b60e-868c5b8b637f\") " pod="openshift-cluster-node-tuning-operator/tuned-n2w6d"
Jul 28 15:33:42.272857 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:33:42.272385    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/ad8ebf58-b767-4341-b60e-868c5b8b637f-etc-tuned\") pod \"tuned-n2w6d\" (UID: \"ad8ebf58-b767-4341-b60e-868c5b8b637f\") " pod="openshift-cluster-node-tuning-operator/tuned-n2w6d"
Jul 28 15:33:42.273662 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:33:42.272443    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/3622f792-27df-4604-a1e3-47192c7302cb-var-lib-openvswitch\") pod \"ovnkube-node-2vrz4\" (UID: \"3622f792-27df-4604-a1e3-47192c7302cb\") " pod="openshift-ovn-kubernetes/ovnkube-node-2vrz4"
Jul 28 15:33:42.273662 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:33:42.272509    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/3622f792-27df-4604-a1e3-47192c7302cb-run-ovn\") pod \"ovnkube-node-2vrz4\" (UID: \"3622f792-27df-4604-a1e3-47192c7302cb\") " pod="openshift-ovn-kubernetes/ovnkube-node-2vrz4"
Jul 28 15:33:42.273662 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:33:42.272527    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/ad8ebf58-b767-4341-b60e-868c5b8b637f-etc-sysctl-conf\") pod \"tuned-n2w6d\" (UID: \"ad8ebf58-b767-4341-b60e-868c5b8b637f\") " pod="openshift-cluster-node-tuning-operator/tuned-n2w6d"
Jul 28 15:33:42.273662 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:33:42.272541    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/7e2caef2-3492-48d3-adbd-89fd3d4494d9-system-cni-dir\") pod \"multus-nvbhc\" (UID: \"7e2caef2-3492-48d3-adbd-89fd3d4494d9\") " pod="openshift-multus/multus-nvbhc"
Jul 28 15:33:42.273662 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:33:42.272579    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/17b1a488-5fd2-4cd5-871a-0f5d127493e2-etc-selinux\") pod \"aws-ebs-csi-driver-node-2nwbt\" (UID: \"17b1a488-5fd2-4cd5-871a-0f5d127493e2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2nwbt"
Jul 28 15:33:42.273662 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:33:42.272616    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/7e2caef2-3492-48d3-adbd-89fd3d4494d9-system-cni-dir\") pod \"multus-nvbhc\" (UID: \"7e2caef2-3492-48d3-adbd-89fd3d4494d9\") " pod="openshift-multus/multus-nvbhc"
Jul 28 15:33:42.273662 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:33:42.272663    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/dcab1041-8d3b-42e1-af05-55e847b831ad-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-ptrdx\" (UID: \"dcab1041-8d3b-42e1-af05-55e847b831ad\") " pod="openshift-multus/multus-additional-cni-plugins-ptrdx"
Jul 28 15:33:42.273662 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:33:42.272689    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/53a1cbf6-fc45-47f6-a8b1-5f95ec295587-host\") pod \"node-ca-9hznr\" (UID: \"53a1cbf6-fc45-47f6-a8b1-5f95ec295587\") " pod="openshift-image-registry/node-ca-9hznr"
Jul 28 15:33:42.273662 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:33:42.272765    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/ad8ebf58-b767-4341-b60e-868c5b8b637f-etc-sysctl-d\") pod \"tuned-n2w6d\" (UID: \"ad8ebf58-b767-4341-b60e-868c5b8b637f\") " pod="openshift-cluster-node-tuning-operator/tuned-n2w6d"
Jul 28 15:33:42.273662 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:33:42.272767    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/17b1a488-5fd2-4cd5-871a-0f5d127493e2-etc-selinux\") pod \"aws-ebs-csi-driver-node-2nwbt\" (UID: \"17b1a488-5fd2-4cd5-871a-0f5d127493e2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2nwbt"
Jul 28 15:33:42.273662 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:33:42.272811    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m446d\" (UniqueName: \"kubernetes.io/projected/ad8ebf58-b767-4341-b60e-868c5b8b637f-kube-api-access-m446d\") pod \"tuned-n2w6d\" (UID: \"ad8ebf58-b767-4341-b60e-868c5b8b637f\") " pod="openshift-cluster-node-tuning-operator/tuned-n2w6d"
Jul 28 15:33:42.273662 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:33:42.272838    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qks42\" (UniqueName: \"kubernetes.io/projected/644d231e-7d3c-4aef-a201-87266e83d526-kube-api-access-qks42\") pod \"network-check-target-85ncq\" (UID: \"644d231e-7d3c-4aef-a201-87266e83d526\") " pod="openshift-network-diagnostics/network-check-target-85ncq"
Jul 28 15:33:42.273662 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:33:42.272862    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/3622f792-27df-4604-a1e3-47192c7302cb-host-kubelet\") pod \"ovnkube-node-2vrz4\" (UID: \"3622f792-27df-4604-a1e3-47192c7302cb\") " pod="openshift-ovn-kubernetes/ovnkube-node-2vrz4"
Jul 28 15:33:42.273662 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:33:42.272893    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/ad8ebf58-b767-4341-b60e-868c5b8b637f-etc-sysctl-d\") pod \"tuned-n2w6d\" (UID: \"ad8ebf58-b767-4341-b60e-868c5b8b637f\") " pod="openshift-cluster-node-tuning-operator/tuned-n2w6d"
Jul 28 15:33:42.273662 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:33:42.272888    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lqthn\" (UniqueName: \"kubernetes.io/projected/7e2caef2-3492-48d3-adbd-89fd3d4494d9-kube-api-access-lqthn\") pod \"multus-nvbhc\" (UID: \"7e2caef2-3492-48d3-adbd-89fd3d4494d9\") " pod="openshift-multus/multus-nvbhc"
Jul 28 15:33:42.273662 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:33:42.272963    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/ad8ebf58-b767-4341-b60e-868c5b8b637f-run\") pod \"tuned-n2w6d\" (UID: \"ad8ebf58-b767-4341-b60e-868c5b8b637f\") " pod="openshift-cluster-node-tuning-operator/tuned-n2w6d"
Jul 28 15:33:42.273662 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:33:42.273014    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mkvrb\" (UniqueName: \"kubernetes.io/projected/53a1cbf6-fc45-47f6-a8b1-5f95ec295587-kube-api-access-mkvrb\") pod \"node-ca-9hznr\" (UID: \"53a1cbf6-fc45-47f6-a8b1-5f95ec295587\") " pod="openshift-image-registry/node-ca-9hznr"
Jul 28 15:33:42.274415 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:33:42.273039    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/3622f792-27df-4604-a1e3-47192c7302cb-node-log\") pod \"ovnkube-node-2vrz4\" (UID: \"3622f792-27df-4604-a1e3-47192c7302cb\") " pod="openshift-ovn-kubernetes/ovnkube-node-2vrz4"
Jul 28 15:33:42.274415 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:33:42.273061    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/3622f792-27df-4604-a1e3-47192c7302cb-env-overrides\") pod \"ovnkube-node-2vrz4\" (UID: \"3622f792-27df-4604-a1e3-47192c7302cb\") " pod="openshift-ovn-kubernetes/ovnkube-node-2vrz4"
Jul 28 15:33:42.274415 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:33:42.273090    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cdt9c\" (UniqueName: \"kubernetes.io/projected/3622f792-27df-4604-a1e3-47192c7302cb-kube-api-access-cdt9c\") pod \"ovnkube-node-2vrz4\" (UID: \"3622f792-27df-4604-a1e3-47192c7302cb\") " pod="openshift-ovn-kubernetes/ovnkube-node-2vrz4"
Jul 28 15:33:42.274415 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:33:42.273107    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ddv7g\" (UniqueName: \"kubernetes.io/projected/fab92b7b-ed62-4783-86be-ed4bd56a3128-kube-api-access-ddv7g\") pod \"node-resolver-whdfr\" (UID: \"fab92b7b-ed62-4783-86be-ed4bd56a3128\") " pod="openshift-dns/node-resolver-whdfr"
Jul 28 15:33:42.274415 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:33:42.273122    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/ad8ebf58-b767-4341-b60e-868c5b8b637f-run\") pod \"tuned-n2w6d\" (UID: \"ad8ebf58-b767-4341-b60e-868c5b8b637f\") " pod="openshift-cluster-node-tuning-operator/tuned-n2w6d"
Jul 28 15:33:42.274415 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:33:42.273146    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/ad8ebf58-b767-4341-b60e-868c5b8b637f-etc-sysconfig\") pod \"tuned-n2w6d\" (UID: \"ad8ebf58-b767-4341-b60e-868c5b8b637f\") " pod="openshift-cluster-node-tuning-operator/tuned-n2w6d"
Jul 28 15:33:42.274415 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:33:42.273186    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ad8ebf58-b767-4341-b60e-868c5b8b637f-etc-kubernetes\") pod \"tuned-n2w6d\" (UID: \"ad8ebf58-b767-4341-b60e-868c5b8b637f\") " pod="openshift-cluster-node-tuning-operator/tuned-n2w6d"
Jul 28 15:33:42.274415 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:33:42.273211    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/ebf75c0d-1597-4a1c-b668-dc2a041eb35e-konnectivity-ca\") pod \"konnectivity-agent-22nh9\" (UID: \"ebf75c0d-1597-4a1c-b668-dc2a041eb35e\") " pod="kube-system/konnectivity-agent-22nh9"
Jul 28 15:33:42.274415 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:33:42.273237    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/7e2caef2-3492-48d3-adbd-89fd3d4494d9-host-run-multus-certs\") pod \"multus-nvbhc\" (UID: \"7e2caef2-3492-48d3-adbd-89fd3d4494d9\") " pod="openshift-multus/multus-nvbhc"
Jul 28 15:33:42.274415 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:33:42.273260    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fab92b7b-ed62-4783-86be-ed4bd56a3128-tmp-dir\") pod \"node-resolver-whdfr\" (UID: \"fab92b7b-ed62-4783-86be-ed4bd56a3128\") " pod="openshift-dns/node-resolver-whdfr"
Jul 28 15:33:42.274415 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:33:42.273269    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ad8ebf58-b767-4341-b60e-868c5b8b637f-etc-kubernetes\") pod \"tuned-n2w6d\" (UID: \"ad8ebf58-b767-4341-b60e-868c5b8b637f\") " pod="openshift-cluster-node-tuning-operator/tuned-n2w6d"
Jul 28 15:33:42.274415 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:33:42.273283    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/7e2caef2-3492-48d3-adbd-89fd3d4494d9-cni-binary-copy\") pod \"multus-nvbhc\" (UID: \"7e2caef2-3492-48d3-adbd-89fd3d4494d9\") " pod="openshift-multus/multus-nvbhc"
Jul 28 15:33:42.274415 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:33:42.273315    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/7e2caef2-3492-48d3-adbd-89fd3d4494d9-host-run-k8s-cni-cncf-io\") pod \"multus-nvbhc\" (UID: \"7e2caef2-3492-48d3-adbd-89fd3d4494d9\") " pod="openshift-multus/multus-nvbhc"
Jul 28 15:33:42.274415 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:33:42.273320    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/7e2caef2-3492-48d3-adbd-89fd3d4494d9-host-run-multus-certs\") pod \"multus-nvbhc\" (UID: \"7e2caef2-3492-48d3-adbd-89fd3d4494d9\") " pod="openshift-multus/multus-nvbhc"
Jul 28 15:33:42.274415 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:33:42.273317    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/ad8ebf58-b767-4341-b60e-868c5b8b637f-etc-sysconfig\") pod \"tuned-n2w6d\" (UID: \"ad8ebf58-b767-4341-b60e-868c5b8b637f\") " pod="openshift-cluster-node-tuning-operator/tuned-n2w6d"
Jul 28 15:33:42.274415 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:33:42.273376    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/7e2caef2-3492-48d3-adbd-89fd3d4494d9-host-run-k8s-cni-cncf-io\") pod \"multus-nvbhc\" (UID: \"7e2caef2-3492-48d3-adbd-89fd3d4494d9\") " pod="openshift-multus/multus-nvbhc"
Jul 28 15:33:42.274415 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:33:42.273405    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/7e2caef2-3492-48d3-adbd-89fd3d4494d9-host-run-netns\") pod \"multus-nvbhc\" (UID: \"7e2caef2-3492-48d3-adbd-89fd3d4494d9\") " pod="openshift-multus/multus-nvbhc"
Jul 28 15:33:42.274415 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:33:42.273427    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/7e2caef2-3492-48d3-adbd-89fd3d4494d9-host-var-lib-kubelet\") pod \"multus-nvbhc\" (UID: \"7e2caef2-3492-48d3-adbd-89fd3d4494d9\") " pod="openshift-multus/multus-nvbhc"
Jul 28 15:33:42.275188 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:33:42.273449    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/7e2caef2-3492-48d3-adbd-89fd3d4494d9-multus-daemon-config\") pod \"multus-nvbhc\" (UID: \"7e2caef2-3492-48d3-adbd-89fd3d4494d9\") " pod="openshift-multus/multus-nvbhc"
Jul 28 15:33:42.275188 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:33:42.273474    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/53a1cbf6-fc45-47f6-a8b1-5f95ec295587-serviceca\") pod \"node-ca-9hznr\" (UID: \"53a1cbf6-fc45-47f6-a8b1-5f95ec295587\") " pod="openshift-image-registry/node-ca-9hznr"
Jul 28 15:33:42.275188 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:33:42.273500    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/17b1a488-5fd2-4cd5-871a-0f5d127493e2-socket-dir\") pod \"aws-ebs-csi-driver-node-2nwbt\" (UID: \"17b1a488-5fd2-4cd5-871a-0f5d127493e2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2nwbt"
Jul 28 15:33:42.275188 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:33:42.273523    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-85sj4\" (UniqueName: \"kubernetes.io/projected/6100b515-cfe9-47ef-b672-d698ae677bd0-kube-api-access-85sj4\") pod \"network-metrics-daemon-ckpr5\" (UID: \"6100b515-cfe9-47ef-b672-d698ae677bd0\") " pod="openshift-multus/network-metrics-daemon-ckpr5"
Jul 28 15:33:42.275188 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:33:42.273546    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/3622f792-27df-4604-a1e3-47192c7302cb-systemd-units\") pod \"ovnkube-node-2vrz4\" (UID: \"3622f792-27df-4604-a1e3-47192c7302cb\") " pod="openshift-ovn-kubernetes/ovnkube-node-2vrz4"
Jul 28 15:33:42.275188 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:33:42.273570    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/3622f792-27df-4604-a1e3-47192c7302cb-run-systemd\") pod \"ovnkube-node-2vrz4\" (UID: \"3622f792-27df-4604-a1e3-47192c7302cb\") " pod="openshift-ovn-kubernetes/ovnkube-node-2vrz4"
Jul 28 15:33:42.275188 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:33:42.273572    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fab92b7b-ed62-4783-86be-ed4bd56a3128-tmp-dir\") pod \"node-resolver-whdfr\" (UID: \"fab92b7b-ed62-4783-86be-ed4bd56a3128\") " pod="openshift-dns/node-resolver-whdfr"
Jul 28 15:33:42.275188 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:33:42.273603    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/3622f792-27df-4604-a1e3-47192c7302cb-etc-openvswitch\") pod \"ovnkube-node-2vrz4\" (UID: \"3622f792-27df-4604-a1e3-47192c7302cb\") " pod="openshift-ovn-kubernetes/ovnkube-node-2vrz4"
Jul 28 15:33:42.275188 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:33:42.273629    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/3622f792-27df-4604-a1e3-47192c7302cb-log-socket\") pod \"ovnkube-node-2vrz4\" (UID: \"3622f792-27df-4604-a1e3-47192c7302cb\") " pod="openshift-ovn-kubernetes/ovnkube-node-2vrz4"
Jul 28 15:33:42.275188 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:33:42.273655    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/3622f792-27df-4604-a1e3-47192c7302cb-host-cni-bin\") pod \"ovnkube-node-2vrz4\" (UID: \"3622f792-27df-4604-a1e3-47192c7302cb\") " pod="openshift-ovn-kubernetes/ovnkube-node-2vrz4"
Jul 28 15:33:42.275188 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:33:42.273699    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/17b1a488-5fd2-4cd5-871a-0f5d127493e2-socket-dir\") pod \"aws-ebs-csi-driver-node-2nwbt\" (UID: \"17b1a488-5fd2-4cd5-871a-0f5d127493e2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2nwbt"
Jul 28 15:33:42.275188 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:33:42.273735    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3622f792-27df-4604-a1e3-47192c7302cb-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-2vrz4\" (UID: \"3622f792-27df-4604-a1e3-47192c7302cb\") " pod="openshift-ovn-kubernetes/ovnkube-node-2vrz4"
Jul 28 15:33:42.275188 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:33:42.273790    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/de4142bb-fcb0-460c-9fbf-1025c61e2015-host-slash\") pod \"iptables-alerter-49vz8\" (UID: \"de4142bb-fcb0-460c-9fbf-1025c61e2015\") " pod="openshift-network-operator/iptables-alerter-49vz8"
Jul 28 15:33:42.275188 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:33:42.273815    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/17b1a488-5fd2-4cd5-871a-0f5d127493e2-kubelet-dir\") pod \"aws-ebs-csi-driver-node-2nwbt\" (UID: \"17b1a488-5fd2-4cd5-871a-0f5d127493e2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2nwbt"
Jul 28 15:33:42.275188 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:33:42.273838    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/3ba72ab7-7ab4-4e7f-ace4-8a6b7bbff25c-dbus\") pod \"global-pull-secret-syncer-tr647\" (UID: \"3ba72ab7-7ab4-4e7f-ace4-8a6b7bbff25c\") " pod="kube-system/global-pull-secret-syncer-tr647"
Jul 28 15:33:42.275188 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:33:42.273894    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3622f792-27df-4604-a1e3-47192c7302cb-host-run-ovn-kubernetes\") pod \"ovnkube-node-2vrz4\" (UID: \"3622f792-27df-4604-a1e3-47192c7302cb\") " pod="openshift-ovn-kubernetes/ovnkube-node-2vrz4"
Jul 28 15:33:42.275188 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:33:42.273886    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/7e2caef2-3492-48d3-adbd-89fd3d4494d9-cni-binary-copy\") pod \"multus-nvbhc\" (UID: \"7e2caef2-3492-48d3-adbd-89fd3d4494d9\") " pod="openshift-multus/multus-nvbhc"
Jul 28 15:33:42.276273 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:33:42.273923    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/7e2caef2-3492-48d3-adbd-89fd3d4494d9-cnibin\") pod \"multus-nvbhc\" (UID: \"7e2caef2-3492-48d3-adbd-89fd3d4494d9\") " pod="openshift-multus/multus-nvbhc"
Jul 28 15:33:42.276273 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:33:42.273949    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/7e2caef2-3492-48d3-adbd-89fd3d4494d9-host-var-lib-kubelet\") pod \"multus-nvbhc\" (UID: \"7e2caef2-3492-48d3-adbd-89fd3d4494d9\") " pod="openshift-multus/multus-nvbhc"
Jul 28 15:33:42.276273 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:33:42.273950    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/7e2caef2-3492-48d3-adbd-89fd3d4494d9-multus-socket-dir-parent\") pod \"multus-nvbhc\" (UID: \"7e2caef2-3492-48d3-adbd-89fd3d4494d9\") " pod="openshift-multus/multus-nvbhc"
Jul 28 15:33:42.276273 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:33:42.273985    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/7e2caef2-3492-48d3-adbd-89fd3d4494d9-host-var-lib-cni-multus\") pod \"multus-nvbhc\" (UID: \"7e2caef2-3492-48d3-adbd-89fd3d4494d9\") " pod="openshift-multus/multus-nvbhc"
Jul 28 15:33:42.276273 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:33:42.273997    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/7e2caef2-3492-48d3-adbd-89fd3d4494d9-multus-socket-dir-parent\") pod \"multus-nvbhc\" (UID: \"7e2caef2-3492-48d3-adbd-89fd3d4494d9\") " pod="openshift-multus/multus-nvbhc"
Jul 28 15:33:42.276273 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:33:42.274011    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/7e2caef2-3492-48d3-adbd-89fd3d4494d9-hostroot\") pod \"multus-nvbhc\" (UID: \"7e2caef2-3492-48d3-adbd-89fd3d4494d9\") " pod="openshift-multus/multus-nvbhc"
Jul 28 15:33:42.276273 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:33:42.274038    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/17b1a488-5fd2-4cd5-871a-0f5d127493e2-registration-dir\") pod \"aws-ebs-csi-driver-node-2nwbt\" (UID: \"17b1a488-5fd2-4cd5-871a-0f5d127493e2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2nwbt"
Jul 28 15:33:42.276273 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:33:42.274062    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/7e2caef2-3492-48d3-adbd-89fd3d4494d9-multus-daemon-config\") pod \"multus-nvbhc\" (UID: \"7e2caef2-3492-48d3-adbd-89fd3d4494d9\") " pod="openshift-multus/multus-nvbhc"
Jul 28 15:33:42.276273 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:33:42.274067    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/de4142bb-fcb0-460c-9fbf-1025c61e2015-host-slash\") pod \"iptables-alerter-49vz8\" (UID: \"de4142bb-fcb0-460c-9fbf-1025c61e2015\") " pod="openshift-network-operator/iptables-alerter-49vz8"
Jul 28 15:33:42.276273 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:33:42.274039    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/7e2caef2-3492-48d3-adbd-89fd3d4494d9-host-run-netns\") pod \"multus-nvbhc\" (UID: \"7e2caef2-3492-48d3-adbd-89fd3d4494d9\") " pod="openshift-multus/multus-nvbhc"
Jul 28 15:33:42.276273 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:33:42.274063    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/ad8ebf58-b767-4341-b60e-868c5b8b637f-etc-modprobe-d\") pod \"tuned-n2w6d\" (UID: \"ad8ebf58-b767-4341-b60e-868c5b8b637f\") " pod="openshift-cluster-node-tuning-operator/tuned-n2w6d"
Jul 28 15:33:42.276273 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:33:42.274121    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/7e2caef2-3492-48d3-adbd-89fd3d4494d9-hostroot\") pod \"multus-nvbhc\" (UID: \"7e2caef2-3492-48d3-adbd-89fd3d4494d9\") " pod="openshift-multus/multus-nvbhc"
Jul 28 15:33:42.276273 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:33:42.274137    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/ad8ebf58-b767-4341-b60e-868c5b8b637f-etc-modprobe-d\") pod \"tuned-n2w6d\" (UID: \"ad8ebf58-b767-4341-b60e-868c5b8b637f\") " pod="openshift-cluster-node-tuning-operator/tuned-n2w6d"
Jul 28 15:33:42.276273 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:33:42.274169    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/17b1a488-5fd2-4cd5-871a-0f5d127493e2-registration-dir\") pod \"aws-ebs-csi-driver-node-2nwbt\" (UID: \"17b1a488-5fd2-4cd5-871a-0f5d127493e2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2nwbt"
Jul 28 15:33:42.276273 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:33:42.274173    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/17b1a488-5fd2-4cd5-871a-0f5d127493e2-kubelet-dir\") pod \"aws-ebs-csi-driver-node-2nwbt\" (UID: \"17b1a488-5fd2-4cd5-871a-0f5d127493e2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2nwbt"
Jul 28 15:33:42.276273 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:33:42.274198    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/7e2caef2-3492-48d3-adbd-89fd3d4494d9-cnibin\") pod \"multus-nvbhc\" (UID: \"7e2caef2-3492-48d3-adbd-89fd3d4494d9\") " pod="openshift-multus/multus-nvbhc"
Jul 28 15:33:42.276273 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:33:42.274229    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/7e2caef2-3492-48d3-adbd-89fd3d4494d9-host-var-lib-cni-multus\") pod \"multus-nvbhc\" (UID: \"7e2caef2-3492-48d3-adbd-89fd3d4494d9\") " pod="openshift-multus/multus-nvbhc"
Jul 28 15:33:42.276273 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:33:42.275710    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ad8ebf58-b767-4341-b60e-868c5b8b637f-tmp\") pod \"tuned-n2w6d\" (UID: \"ad8ebf58-b767-4341-b60e-868c5b8b637f\") " pod="openshift-cluster-node-tuning-operator/tuned-n2w6d"
Jul 28 15:33:42.277180 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:33:42.275734    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/ad8ebf58-b767-4341-b60e-868c5b8b637f-etc-tuned\") pod \"tuned-n2w6d\" (UID: \"ad8ebf58-b767-4341-b60e-868c5b8b637f\") " pod="openshift-cluster-node-tuning-operator/tuned-n2w6d"
Jul 28 15:33:42.280610 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:33:42.280525    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b297t\" (UniqueName: \"kubernetes.io/projected/de4142bb-fcb0-460c-9fbf-1025c61e2015-kube-api-access-b297t\") pod \"iptables-alerter-49vz8\" (UID: \"de4142bb-fcb0-460c-9fbf-1025c61e2015\") " pod="openshift-network-operator/iptables-alerter-49vz8"
Jul 28 15:33:42.280726 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:33:42.280690    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zbhxp\" (UniqueName: \"kubernetes.io/projected/17b1a488-5fd2-4cd5-871a-0f5d127493e2-kube-api-access-zbhxp\") pod \"aws-ebs-csi-driver-node-2nwbt\" (UID: \"17b1a488-5fd2-4cd5-871a-0f5d127493e2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2nwbt"
Jul 28 15:33:42.285661 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:33:42.285640    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lqthn\" (UniqueName: \"kubernetes.io/projected/7e2caef2-3492-48d3-adbd-89fd3d4494d9-kube-api-access-lqthn\") pod \"multus-nvbhc\" (UID: \"7e2caef2-3492-48d3-adbd-89fd3d4494d9\") " pod="openshift-multus/multus-nvbhc"
Jul 28 15:33:42.285925 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:33:42.285907    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ddv7g\" (UniqueName: \"kubernetes.io/projected/fab92b7b-ed62-4783-86be-ed4bd56a3128-kube-api-access-ddv7g\") pod \"node-resolver-whdfr\" (UID: \"fab92b7b-ed62-4783-86be-ed4bd56a3128\") " pod="openshift-dns/node-resolver-whdfr"
Jul 28 15:33:42.286927 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:33:42.286872    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m446d\" (UniqueName: \"kubernetes.io/projected/ad8ebf58-b767-4341-b60e-868c5b8b637f-kube-api-access-m446d\") pod \"tuned-n2w6d\" (UID: \"ad8ebf58-b767-4341-b60e-868c5b8b637f\") " pod="openshift-cluster-node-tuning-operator/tuned-n2w6d"
Jul 28 15:33:42.374572 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:33:42.374488    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/3622f792-27df-4604-a1e3-47192c7302cb-run-openvswitch\") pod \"ovnkube-node-2vrz4\" (UID: \"3622f792-27df-4604-a1e3-47192c7302cb\") " pod="openshift-ovn-kubernetes/ovnkube-node-2vrz4"
Jul 28 15:33:42.374572 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:33:42.374523    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/3622f792-27df-4604-a1e3-47192c7302cb-ovnkube-config\") pod \"ovnkube-node-2vrz4\" (UID: \"3622f792-27df-4604-a1e3-47192c7302cb\") " pod="openshift-ovn-kubernetes/ovnkube-node-2vrz4"
Jul 28 15:33:42.374572 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:33:42.374544    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/3622f792-27df-4604-a1e3-47192c7302cb-ovn-node-metrics-cert\") pod \"ovnkube-node-2vrz4\" (UID: \"3622f792-27df-4604-a1e3-47192c7302cb\") " pod="openshift-ovn-kubernetes/ovnkube-node-2vrz4"
Jul 28 15:33:42.374572 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:33:42.374572    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/dcab1041-8d3b-42e1-af05-55e847b831ad-system-cni-dir\") pod \"multus-additional-cni-plugins-ptrdx\" (UID: \"dcab1041-8d3b-42e1-af05-55e847b831ad\") " pod="openshift-multus/multus-additional-cni-plugins-ptrdx"
Jul 28 15:33:42.374869 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:33:42.374597    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6100b515-cfe9-47ef-b672-d698ae677bd0-metrics-certs\") pod \"network-metrics-daemon-ckpr5\" (UID: \"6100b515-cfe9-47ef-b672-d698ae677bd0\") " pod="openshift-multus/network-metrics-daemon-ckpr5"
Jul 28 15:33:42.374869 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:33:42.374616    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/3622f792-27df-4604-a1e3-47192c7302cb-run-openvswitch\") pod \"ovnkube-node-2vrz4\" (UID: \"3622f792-27df-4604-a1e3-47192c7302cb\") " pod="openshift-ovn-kubernetes/ovnkube-node-2vrz4"
Jul 28 15:33:42.374869 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:33:42.374667    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/dcab1041-8d3b-42e1-af05-55e847b831ad-cni-binary-copy\") pod \"multus-additional-cni-plugins-ptrdx\" (UID: \"dcab1041-8d3b-42e1-af05-55e847b831ad\") " pod="openshift-multus/multus-additional-cni-plugins-ptrdx"
Jul 28 15:33:42.374869 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:33:42.374683    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/dcab1041-8d3b-42e1-af05-55e847b831ad-system-cni-dir\") pod \"multus-additional-cni-plugins-ptrdx\" (UID: \"dcab1041-8d3b-42e1-af05-55e847b831ad\") " pod="openshift-multus/multus-additional-cni-plugins-ptrdx"
Jul 28 15:33:42.374869 ip-10-0-131-121 kubenswrapper[2568]: E0728 15:33:42.374700    2568 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 15:33:42.374869 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:33:42.374704    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ktx7j\" (UniqueName: \"kubernetes.io/projected/dcab1041-8d3b-42e1-af05-55e847b831ad-kube-api-access-ktx7j\") pod \"multus-additional-cni-plugins-ptrdx\" (UID: \"dcab1041-8d3b-42e1-af05-55e847b831ad\") " pod="openshift-multus/multus-additional-cni-plugins-ptrdx"
Jul 28 15:33:42.374869 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:33:42.374729    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/3622f792-27df-4604-a1e3-47192c7302cb-host-cni-netd\") pod \"ovnkube-node-2vrz4\" (UID: \"3622f792-27df-4604-a1e3-47192c7302cb\") " pod="openshift-ovn-kubernetes/ovnkube-node-2vrz4"
Jul 28 15:33:42.374869 ip-10-0-131-121 kubenswrapper[2568]: E0728 15:33:42.374786    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6100b515-cfe9-47ef-b672-d698ae677bd0-metrics-certs podName:6100b515-cfe9-47ef-b672-d698ae677bd0 nodeName:}" failed. No retries permitted until 2026-07-28 15:33:42.874743233 +0000 UTC m=+3.059938397 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6100b515-cfe9-47ef-b672-d698ae677bd0-metrics-certs") pod "network-metrics-daemon-ckpr5" (UID: "6100b515-cfe9-47ef-b672-d698ae677bd0") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 15:33:42.374869 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:33:42.374792    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/3622f792-27df-4604-a1e3-47192c7302cb-host-cni-netd\") pod \"ovnkube-node-2vrz4\" (UID: \"3622f792-27df-4604-a1e3-47192c7302cb\") " pod="openshift-ovn-kubernetes/ovnkube-node-2vrz4"
Jul 28 15:33:42.374869 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:33:42.374801    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/3622f792-27df-4604-a1e3-47192c7302cb-var-lib-openvswitch\") pod \"ovnkube-node-2vrz4\" (UID: \"3622f792-27df-4604-a1e3-47192c7302cb\") " pod="openshift-ovn-kubernetes/ovnkube-node-2vrz4"
Jul 28 15:33:42.374869 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:33:42.374818    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/3622f792-27df-4604-a1e3-47192c7302cb-run-ovn\") pod \"ovnkube-node-2vrz4\" (UID: \"3622f792-27df-4604-a1e3-47192c7302cb\") " pod="openshift-ovn-kubernetes/ovnkube-node-2vrz4"
Jul 28 15:33:42.374869 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:33:42.374844    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/dcab1041-8d3b-42e1-af05-55e847b831ad-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-ptrdx\" (UID: \"dcab1041-8d3b-42e1-af05-55e847b831ad\") " pod="openshift-multus/multus-additional-cni-plugins-ptrdx"
Jul 28 15:33:42.374869 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:33:42.374860    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/53a1cbf6-fc45-47f6-a8b1-5f95ec295587-host\") pod \"node-ca-9hznr\" (UID: \"53a1cbf6-fc45-47f6-a8b1-5f95ec295587\") " pod="openshift-image-registry/node-ca-9hznr"
Jul 28 15:33:42.374869 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:33:42.374876    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qks42\" (UniqueName: \"kubernetes.io/projected/644d231e-7d3c-4aef-a201-87266e83d526-kube-api-access-qks42\") pod \"network-check-target-85ncq\" (UID: \"644d231e-7d3c-4aef-a201-87266e83d526\") " pod="openshift-network-diagnostics/network-check-target-85ncq"
Jul 28 15:33:42.375436 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:33:42.374891    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/3622f792-27df-4604-a1e3-47192c7302cb-host-kubelet\") pod \"ovnkube-node-2vrz4\" (UID: \"3622f792-27df-4604-a1e3-47192c7302cb\") " pod="openshift-ovn-kubernetes/ovnkube-node-2vrz4"
Jul 28 15:33:42.375436 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:33:42.374910    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mkvrb\" (UniqueName: \"kubernetes.io/projected/53a1cbf6-fc45-47f6-a8b1-5f95ec295587-kube-api-access-mkvrb\") pod \"node-ca-9hznr\" (UID: \"53a1cbf6-fc45-47f6-a8b1-5f95ec295587\") " pod="openshift-image-registry/node-ca-9hznr"
Jul 28 15:33:42.375436 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:33:42.374924    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/3622f792-27df-4604-a1e3-47192c7302cb-node-log\") pod \"ovnkube-node-2vrz4\" (UID: \"3622f792-27df-4604-a1e3-47192c7302cb\") " pod="openshift-ovn-kubernetes/ovnkube-node-2vrz4"
Jul 28 15:33:42.375436 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:33:42.374939    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/3622f792-27df-4604-a1e3-47192c7302cb-env-overrides\") pod \"ovnkube-node-2vrz4\" (UID: \"3622f792-27df-4604-a1e3-47192c7302cb\") " pod="openshift-ovn-kubernetes/ovnkube-node-2vrz4"
Jul 28 15:33:42.375436 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:33:42.374961    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cdt9c\" (UniqueName: \"kubernetes.io/projected/3622f792-27df-4604-a1e3-47192c7302cb-kube-api-access-cdt9c\") pod \"ovnkube-node-2vrz4\" (UID: \"3622f792-27df-4604-a1e3-47192c7302cb\") " pod="openshift-ovn-kubernetes/ovnkube-node-2vrz4"
Jul 28 15:33:42.375436 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:33:42.374980    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/ebf75c0d-1597-4a1c-b668-dc2a041eb35e-konnectivity-ca\") pod \"konnectivity-agent-22nh9\" (UID: \"ebf75c0d-1597-4a1c-b668-dc2a041eb35e\") " pod="kube-system/konnectivity-agent-22nh9"
Jul 28 15:33:42.375436 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:33:42.374999    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/53a1cbf6-fc45-47f6-a8b1-5f95ec295587-serviceca\") pod \"node-ca-9hznr\" (UID: \"53a1cbf6-fc45-47f6-a8b1-5f95ec295587\") " pod="openshift-image-registry/node-ca-9hznr"
Jul 28 15:33:42.375436 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:33:42.375015    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-85sj4\" (UniqueName: \"kubernetes.io/projected/6100b515-cfe9-47ef-b672-d698ae677bd0-kube-api-access-85sj4\") pod \"network-metrics-daemon-ckpr5\" (UID: \"6100b515-cfe9-47ef-b672-d698ae677bd0\") " pod="openshift-multus/network-metrics-daemon-ckpr5"
Jul 28 15:33:42.375436 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:33:42.375031    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/3622f792-27df-4604-a1e3-47192c7302cb-systemd-units\") pod \"ovnkube-node-2vrz4\" (UID: \"3622f792-27df-4604-a1e3-47192c7302cb\") " pod="openshift-ovn-kubernetes/ovnkube-node-2vrz4"
Jul 28 15:33:42.375436 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:33:42.375059    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/3622f792-27df-4604-a1e3-47192c7302cb-run-systemd\") pod \"ovnkube-node-2vrz4\" (UID: \"3622f792-27df-4604-a1e3-47192c7302cb\") " pod="openshift-ovn-kubernetes/ovnkube-node-2vrz4"
Jul 28 15:33:42.375436 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:33:42.375074    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/3622f792-27df-4604-a1e3-47192c7302cb-etc-openvswitch\") pod \"ovnkube-node-2vrz4\" (UID: \"3622f792-27df-4604-a1e3-47192c7302cb\") " pod="openshift-ovn-kubernetes/ovnkube-node-2vrz4"
Jul 28 15:33:42.375436 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:33:42.375088    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/3622f792-27df-4604-a1e3-47192c7302cb-log-socket\") pod \"ovnkube-node-2vrz4\" (UID: \"3622f792-27df-4604-a1e3-47192c7302cb\") " pod="openshift-ovn-kubernetes/ovnkube-node-2vrz4"
Jul 28 15:33:42.375436 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:33:42.375086    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/3622f792-27df-4604-a1e3-47192c7302cb-host-kubelet\") pod \"ovnkube-node-2vrz4\" (UID: \"3622f792-27df-4604-a1e3-47192c7302cb\") " pod="openshift-ovn-kubernetes/ovnkube-node-2vrz4"
Jul 28 15:33:42.375436 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:33:42.375136    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/3622f792-27df-4604-a1e3-47192c7302cb-ovnkube-config\") pod \"ovnkube-node-2vrz4\" (UID: \"3622f792-27df-4604-a1e3-47192c7302cb\") " pod="openshift-ovn-kubernetes/ovnkube-node-2vrz4"
Jul 28 15:33:42.375436 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:33:42.375149    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/3622f792-27df-4604-a1e3-47192c7302cb-log-socket\") pod \"ovnkube-node-2vrz4\" (UID: \"3622f792-27df-4604-a1e3-47192c7302cb\") " pod="openshift-ovn-kubernetes/ovnkube-node-2vrz4"
Jul 28 15:33:42.375436 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:33:42.375184    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/3622f792-27df-4604-a1e3-47192c7302cb-host-cni-bin\") pod \"ovnkube-node-2vrz4\" (UID: \"3622f792-27df-4604-a1e3-47192c7302cb\") " pod="openshift-ovn-kubernetes/ovnkube-node-2vrz4"
Jul 28 15:33:42.375436 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:33:42.375248    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/dcab1041-8d3b-42e1-af05-55e847b831ad-cni-binary-copy\") pod \"multus-additional-cni-plugins-ptrdx\" (UID: \"dcab1041-8d3b-42e1-af05-55e847b831ad\") " pod="openshift-multus/multus-additional-cni-plugins-ptrdx"
Jul 28 15:33:42.376294 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:33:42.375314    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/3622f792-27df-4604-a1e3-47192c7302cb-run-systemd\") pod \"ovnkube-node-2vrz4\" (UID: \"3622f792-27df-4604-a1e3-47192c7302cb\") " pod="openshift-ovn-kubernetes/ovnkube-node-2vrz4"
Jul 28 15:33:42.376294 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:33:42.375352    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/3622f792-27df-4604-a1e3-47192c7302cb-etc-openvswitch\") pod \"ovnkube-node-2vrz4\" (UID: \"3622f792-27df-4604-a1e3-47192c7302cb\") " pod="openshift-ovn-kubernetes/ovnkube-node-2vrz4"
Jul 28 15:33:42.376294 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:33:42.375131    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/3622f792-27df-4604-a1e3-47192c7302cb-host-cni-bin\") pod \"ovnkube-node-2vrz4\" (UID: \"3622f792-27df-4604-a1e3-47192c7302cb\") " pod="openshift-ovn-kubernetes/ovnkube-node-2vrz4"
Jul 28 15:33:42.376294 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:33:42.375403    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3622f792-27df-4604-a1e3-47192c7302cb-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-2vrz4\" (UID: \"3622f792-27df-4604-a1e3-47192c7302cb\") " pod="openshift-ovn-kubernetes/ovnkube-node-2vrz4"
Jul 28 15:33:42.376294 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:33:42.375429    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/3ba72ab7-7ab4-4e7f-ace4-8a6b7bbff25c-dbus\") pod \"global-pull-secret-syncer-tr647\" (UID: \"3ba72ab7-7ab4-4e7f-ace4-8a6b7bbff25c\") " pod="kube-system/global-pull-secret-syncer-tr647"
Jul 28 15:33:42.376294 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:33:42.375436    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/3622f792-27df-4604-a1e3-47192c7302cb-var-lib-openvswitch\") pod \"ovnkube-node-2vrz4\" (UID: \"3622f792-27df-4604-a1e3-47192c7302cb\") " pod="openshift-ovn-kubernetes/ovnkube-node-2vrz4"
Jul 28 15:33:42.376294 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:33:42.375455    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3622f792-27df-4604-a1e3-47192c7302cb-host-run-ovn-kubernetes\") pod \"ovnkube-node-2vrz4\" (UID: \"3622f792-27df-4604-a1e3-47192c7302cb\") " pod="openshift-ovn-kubernetes/ovnkube-node-2vrz4"
Jul 28 15:33:42.376294 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:33:42.375487    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/3622f792-27df-4604-a1e3-47192c7302cb-run-ovn\") pod \"ovnkube-node-2vrz4\" (UID: \"3622f792-27df-4604-a1e3-47192c7302cb\") " pod="openshift-ovn-kubernetes/ovnkube-node-2vrz4"
Jul 28 15:33:42.376294 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:33:42.375492    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3ba72ab7-7ab4-4e7f-ace4-8a6b7bbff25c-original-pull-secret\") pod \"global-pull-secret-syncer-tr647\" (UID: \"3ba72ab7-7ab4-4e7f-ace4-8a6b7bbff25c\") " pod="kube-system/global-pull-secret-syncer-tr647"
Jul 28 15:33:42.376294 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:33:42.375544    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/3622f792-27df-4604-a1e3-47192c7302cb-systemd-units\") pod \"ovnkube-node-2vrz4\" (UID: \"3622f792-27df-4604-a1e3-47192c7302cb\") " pod="openshift-ovn-kubernetes/ovnkube-node-2vrz4"
Jul 28 15:33:42.376294 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:33:42.375565    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/53a1cbf6-fc45-47f6-a8b1-5f95ec295587-host\") pod \"node-ca-9hznr\" (UID: \"53a1cbf6-fc45-47f6-a8b1-5f95ec295587\") " pod="openshift-image-registry/node-ca-9hznr"
Jul 28 15:33:42.376294 ip-10-0-131-121 kubenswrapper[2568]: E0728 15:33:42.375577    2568 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 28 15:33:42.376294 ip-10-0-131-121 kubenswrapper[2568]: E0728 15:33:42.375619    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3ba72ab7-7ab4-4e7f-ace4-8a6b7bbff25c-original-pull-secret podName:3ba72ab7-7ab4-4e7f-ace4-8a6b7bbff25c nodeName:}" failed. No retries permitted until 2026-07-28 15:33:42.875604643 +0000 UTC m=+3.060799790 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/3ba72ab7-7ab4-4e7f-ace4-8a6b7bbff25c-original-pull-secret") pod "global-pull-secret-syncer-tr647" (UID: "3ba72ab7-7ab4-4e7f-ace4-8a6b7bbff25c") : object "kube-system"/"original-pull-secret" not registered
Jul 28 15:33:42.376294 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:33:42.375628    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/53a1cbf6-fc45-47f6-a8b1-5f95ec295587-serviceca\") pod \"node-ca-9hznr\" (UID: \"53a1cbf6-fc45-47f6-a8b1-5f95ec295587\") " pod="openshift-image-registry/node-ca-9hznr"
Jul 28 15:33:42.376294 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:33:42.375658    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3622f792-27df-4604-a1e3-47192c7302cb-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-2vrz4\" (UID: \"3622f792-27df-4604-a1e3-47192c7302cb\") " pod="openshift-ovn-kubernetes/ovnkube-node-2vrz4"
Jul 28 15:33:42.376294 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:33:42.375721    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/3ba72ab7-7ab4-4e7f-ace4-8a6b7bbff25c-dbus\") pod \"global-pull-secret-syncer-tr647\" (UID: \"3ba72ab7-7ab4-4e7f-ace4-8a6b7bbff25c\") " pod="kube-system/global-pull-secret-syncer-tr647"
Jul 28 15:33:42.376294 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:33:42.375772    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3622f792-27df-4604-a1e3-47192c7302cb-host-run-ovn-kubernetes\") pod \"ovnkube-node-2vrz4\" (UID: \"3622f792-27df-4604-a1e3-47192c7302cb\") " pod="openshift-ovn-kubernetes/ovnkube-node-2vrz4"
Jul 28 15:33:42.377069 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:33:42.375827    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/3622f792-27df-4604-a1e3-47192c7302cb-env-overrides\") pod \"ovnkube-node-2vrz4\" (UID: \"3622f792-27df-4604-a1e3-47192c7302cb\") " pod="openshift-ovn-kubernetes/ovnkube-node-2vrz4"
Jul 28 15:33:42.377069 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:33:42.375854    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/3622f792-27df-4604-a1e3-47192c7302cb-node-log\") pod \"ovnkube-node-2vrz4\" (UID: \"3622f792-27df-4604-a1e3-47192c7302cb\") " pod="openshift-ovn-kubernetes/ovnkube-node-2vrz4"
Jul 28 15:33:42.377069 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:33:42.375882    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/dcab1041-8d3b-42e1-af05-55e847b831ad-cnibin\") pod \"multus-additional-cni-plugins-ptrdx\" (UID: \"dcab1041-8d3b-42e1-af05-55e847b831ad\") " pod="openshift-multus/multus-additional-cni-plugins-ptrdx"
Jul 28 15:33:42.377069 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:33:42.375917    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/dcab1041-8d3b-42e1-af05-55e847b831ad-tuning-conf-dir\") pod \"multus-additional-cni-plugins-ptrdx\" (UID: \"dcab1041-8d3b-42e1-af05-55e847b831ad\") " pod="openshift-multus/multus-additional-cni-plugins-ptrdx"
Jul 28 15:33:42.377069 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:33:42.375940    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/dcab1041-8d3b-42e1-af05-55e847b831ad-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-ptrdx\" (UID: \"dcab1041-8d3b-42e1-af05-55e847b831ad\") " pod="openshift-multus/multus-additional-cni-plugins-ptrdx"
Jul 28 15:33:42.377069 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:33:42.375972    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/dcab1041-8d3b-42e1-af05-55e847b831ad-cnibin\") pod \"multus-additional-cni-plugins-ptrdx\" (UID: \"dcab1041-8d3b-42e1-af05-55e847b831ad\") " pod="openshift-multus/multus-additional-cni-plugins-ptrdx"
Jul 28 15:33:42.377069 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:33:42.375977    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/ebf75c0d-1597-4a1c-b668-dc2a041eb35e-agent-certs\") pod \"konnectivity-agent-22nh9\" (UID: \"ebf75c0d-1597-4a1c-b668-dc2a041eb35e\") " pod="kube-system/konnectivity-agent-22nh9"
Jul 28 15:33:42.377069 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:33:42.376008    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/3ba72ab7-7ab4-4e7f-ace4-8a6b7bbff25c-kubelet-config\") pod \"global-pull-secret-syncer-tr647\" (UID: \"3ba72ab7-7ab4-4e7f-ace4-8a6b7bbff25c\") " pod="kube-system/global-pull-secret-syncer-tr647"
Jul 28 15:33:42.377069 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:33:42.376032    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/dcab1041-8d3b-42e1-af05-55e847b831ad-os-release\") pod \"multus-additional-cni-plugins-ptrdx\" (UID: \"dcab1041-8d3b-42e1-af05-55e847b831ad\") " pod="openshift-multus/multus-additional-cni-plugins-ptrdx"
Jul 28 15:33:42.377069 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:33:42.376034    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/dcab1041-8d3b-42e1-af05-55e847b831ad-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-ptrdx\" (UID: \"dcab1041-8d3b-42e1-af05-55e847b831ad\") " pod="openshift-multus/multus-additional-cni-plugins-ptrdx"
Jul 28 15:33:42.377069 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:33:42.376055    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/3622f792-27df-4604-a1e3-47192c7302cb-ovnkube-script-lib\") pod \"ovnkube-node-2vrz4\" (UID: \"3622f792-27df-4604-a1e3-47192c7302cb\") " pod="openshift-ovn-kubernetes/ovnkube-node-2vrz4"
Jul 28 15:33:42.377069 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:33:42.376081    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/3622f792-27df-4604-a1e3-47192c7302cb-host-slash\") pod \"ovnkube-node-2vrz4\" (UID: \"3622f792-27df-4604-a1e3-47192c7302cb\") " pod="openshift-ovn-kubernetes/ovnkube-node-2vrz4"
Jul 28 15:33:42.377069 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:33:42.376120    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/3622f792-27df-4604-a1e3-47192c7302cb-host-run-netns\") pod \"ovnkube-node-2vrz4\" (UID: \"3622f792-27df-4604-a1e3-47192c7302cb\") " pod="openshift-ovn-kubernetes/ovnkube-node-2vrz4"
Jul 28 15:33:42.377069 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:33:42.376186    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/3622f792-27df-4604-a1e3-47192c7302cb-host-run-netns\") pod \"ovnkube-node-2vrz4\" (UID: \"3622f792-27df-4604-a1e3-47192c7302cb\") " pod="openshift-ovn-kubernetes/ovnkube-node-2vrz4"
Jul 28 15:33:42.377069 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:33:42.376207    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/dcab1041-8d3b-42e1-af05-55e847b831ad-tuning-conf-dir\") pod \"multus-additional-cni-plugins-ptrdx\" (UID: \"dcab1041-8d3b-42e1-af05-55e847b831ad\") " pod="openshift-multus/multus-additional-cni-plugins-ptrdx"
Jul 28 15:33:42.377069 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:33:42.376236    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/3ba72ab7-7ab4-4e7f-ace4-8a6b7bbff25c-kubelet-config\") pod \"global-pull-secret-syncer-tr647\" (UID: \"3ba72ab7-7ab4-4e7f-ace4-8a6b7bbff25c\") " pod="kube-system/global-pull-secret-syncer-tr647"
Jul 28 15:33:42.377069 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:33:42.376276    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/3622f792-27df-4604-a1e3-47192c7302cb-host-slash\") pod \"ovnkube-node-2vrz4\" (UID: \"3622f792-27df-4604-a1e3-47192c7302cb\") " pod="openshift-ovn-kubernetes/ovnkube-node-2vrz4"
Jul 28 15:33:42.377742 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:33:42.376290    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/dcab1041-8d3b-42e1-af05-55e847b831ad-os-release\") pod \"multus-additional-cni-plugins-ptrdx\" (UID: \"dcab1041-8d3b-42e1-af05-55e847b831ad\") " pod="openshift-multus/multus-additional-cni-plugins-ptrdx"
Jul 28 15:33:42.377742 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:33:42.376301    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/ebf75c0d-1597-4a1c-b668-dc2a041eb35e-konnectivity-ca\") pod \"konnectivity-agent-22nh9\" (UID: \"ebf75c0d-1597-4a1c-b668-dc2a041eb35e\") " pod="kube-system/konnectivity-agent-22nh9"
Jul 28 15:33:42.377742 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:33:42.376736    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/dcab1041-8d3b-42e1-af05-55e847b831ad-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-ptrdx\" (UID: \"dcab1041-8d3b-42e1-af05-55e847b831ad\") " pod="openshift-multus/multus-additional-cni-plugins-ptrdx"
Jul 28 15:33:42.377742 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:33:42.376860    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/3622f792-27df-4604-a1e3-47192c7302cb-ovnkube-script-lib\") pod \"ovnkube-node-2vrz4\" (UID: \"3622f792-27df-4604-a1e3-47192c7302cb\") " pod="openshift-ovn-kubernetes/ovnkube-node-2vrz4"
Jul 28 15:33:42.378588 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:33:42.378565    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/3622f792-27df-4604-a1e3-47192c7302cb-ovn-node-metrics-cert\") pod \"ovnkube-node-2vrz4\" (UID: \"3622f792-27df-4604-a1e3-47192c7302cb\") " pod="openshift-ovn-kubernetes/ovnkube-node-2vrz4"
Jul 28 15:33:42.378710 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:33:42.378692    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/ebf75c0d-1597-4a1c-b668-dc2a041eb35e-agent-certs\") pod \"konnectivity-agent-22nh9\" (UID: \"ebf75c0d-1597-4a1c-b668-dc2a041eb35e\") " pod="kube-system/konnectivity-agent-22nh9"
Jul 28 15:33:42.386759 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:33:42.386734    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cdt9c\" (UniqueName: \"kubernetes.io/projected/3622f792-27df-4604-a1e3-47192c7302cb-kube-api-access-cdt9c\") pod \"ovnkube-node-2vrz4\" (UID: \"3622f792-27df-4604-a1e3-47192c7302cb\") " pod="openshift-ovn-kubernetes/ovnkube-node-2vrz4"
Jul 28 15:33:42.386889 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:33:42.386789    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ktx7j\" (UniqueName: \"kubernetes.io/projected/dcab1041-8d3b-42e1-af05-55e847b831ad-kube-api-access-ktx7j\") pod \"multus-additional-cni-plugins-ptrdx\" (UID: \"dcab1041-8d3b-42e1-af05-55e847b831ad\") " pod="openshift-multus/multus-additional-cni-plugins-ptrdx"
Jul 28 15:33:42.389934 ip-10-0-131-121 kubenswrapper[2568]: E0728 15:33:42.389913    2568 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 28 15:33:42.390042 ip-10-0-131-121 kubenswrapper[2568]: E0728 15:33:42.389944    2568 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 28 15:33:42.390042 ip-10-0-131-121 kubenswrapper[2568]: E0728 15:33:42.389959    2568 projected.go:196] Error preparing data for projected volume kube-api-access-qks42 for pod openshift-network-diagnostics/network-check-target-85ncq: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 15:33:42.390042 ip-10-0-131-121 kubenswrapper[2568]: E0728 15:33:42.390036    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/644d231e-7d3c-4aef-a201-87266e83d526-kube-api-access-qks42 podName:644d231e-7d3c-4aef-a201-87266e83d526 nodeName:}" failed. No retries permitted until 2026-07-28 15:33:42.890017617 +0000 UTC m=+3.075212769 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-qks42" (UniqueName: "kubernetes.io/projected/644d231e-7d3c-4aef-a201-87266e83d526-kube-api-access-qks42") pod "network-check-target-85ncq" (UID: "644d231e-7d3c-4aef-a201-87266e83d526") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 15:33:42.390688 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:33:42.390661    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-85sj4\" (UniqueName: \"kubernetes.io/projected/6100b515-cfe9-47ef-b672-d698ae677bd0-kube-api-access-85sj4\") pod \"network-metrics-daemon-ckpr5\" (UID: \"6100b515-cfe9-47ef-b672-d698ae677bd0\") " pod="openshift-multus/network-metrics-daemon-ckpr5"
Jul 28 15:33:42.391749 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:33:42.391729    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mkvrb\" (UniqueName: \"kubernetes.io/projected/53a1cbf6-fc45-47f6-a8b1-5f95ec295587-kube-api-access-mkvrb\") pod \"node-ca-9hznr\" (UID: \"53a1cbf6-fc45-47f6-a8b1-5f95ec295587\") " pod="openshift-image-registry/node-ca-9hznr"
Jul 28 15:33:42.461595 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:33:42.461558    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-nvbhc"
Jul 28 15:33:42.472162 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:33:42.472137    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-49vz8"
Jul 28 15:33:42.481955 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:33:42.481931    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2nwbt"
Jul 28 15:33:42.488797 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:33:42.488763    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-n2w6d"
Jul 28 15:33:42.496444 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:33:42.496422    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-whdfr"
Jul 28 15:33:42.504214 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:33:42.504192    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-9hznr"
Jul 28 15:33:42.513000 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:33:42.512978    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-2vrz4"
Jul 28 15:33:42.519801 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:33:42.519764    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-ptrdx"
Jul 28 15:33:42.527418 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:33:42.527397    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-22nh9"
Jul 28 15:33:42.880143 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:33:42.880115    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3ba72ab7-7ab4-4e7f-ace4-8a6b7bbff25c-original-pull-secret\") pod \"global-pull-secret-syncer-tr647\" (UID: \"3ba72ab7-7ab4-4e7f-ace4-8a6b7bbff25c\") " pod="kube-system/global-pull-secret-syncer-tr647"
Jul 28 15:33:42.880323 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:33:42.880154    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6100b515-cfe9-47ef-b672-d698ae677bd0-metrics-certs\") pod \"network-metrics-daemon-ckpr5\" (UID: \"6100b515-cfe9-47ef-b672-d698ae677bd0\") " pod="openshift-multus/network-metrics-daemon-ckpr5"
Jul 28 15:33:42.880323 ip-10-0-131-121 kubenswrapper[2568]: E0728 15:33:42.880256    2568 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 15:33:42.880323 ip-10-0-131-121 kubenswrapper[2568]: E0728 15:33:42.880279    2568 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 28 15:33:42.880479 ip-10-0-131-121 kubenswrapper[2568]: E0728 15:33:42.880330    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6100b515-cfe9-47ef-b672-d698ae677bd0-metrics-certs podName:6100b515-cfe9-47ef-b672-d698ae677bd0 nodeName:}" failed. No retries permitted until 2026-07-28 15:33:43.880292867 +0000 UTC m=+4.065488022 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6100b515-cfe9-47ef-b672-d698ae677bd0-metrics-certs") pod "network-metrics-daemon-ckpr5" (UID: "6100b515-cfe9-47ef-b672-d698ae677bd0") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 15:33:42.880479 ip-10-0-131-121 kubenswrapper[2568]: E0728 15:33:42.880357    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3ba72ab7-7ab4-4e7f-ace4-8a6b7bbff25c-original-pull-secret podName:3ba72ab7-7ab4-4e7f-ace4-8a6b7bbff25c nodeName:}" failed. No retries permitted until 2026-07-28 15:33:43.880344688 +0000 UTC m=+4.065539838 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/3ba72ab7-7ab4-4e7f-ace4-8a6b7bbff25c-original-pull-secret") pod "global-pull-secret-syncer-tr647" (UID: "3ba72ab7-7ab4-4e7f-ace4-8a6b7bbff25c") : object "kube-system"/"original-pull-secret" not registered
Jul 28 15:33:42.980481 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:33:42.980455    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qks42\" (UniqueName: \"kubernetes.io/projected/644d231e-7d3c-4aef-a201-87266e83d526-kube-api-access-qks42\") pod \"network-check-target-85ncq\" (UID: \"644d231e-7d3c-4aef-a201-87266e83d526\") " pod="openshift-network-diagnostics/network-check-target-85ncq"
Jul 28 15:33:42.980658 ip-10-0-131-121 kubenswrapper[2568]: E0728 15:33:42.980624    2568 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 28 15:33:42.980658 ip-10-0-131-121 kubenswrapper[2568]: E0728 15:33:42.980648    2568 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 28 15:33:42.980750 ip-10-0-131-121 kubenswrapper[2568]: E0728 15:33:42.980664    2568 projected.go:196] Error preparing data for projected volume kube-api-access-qks42 for pod openshift-network-diagnostics/network-check-target-85ncq: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 15:33:42.980750 ip-10-0-131-121 kubenswrapper[2568]: E0728 15:33:42.980722    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/644d231e-7d3c-4aef-a201-87266e83d526-kube-api-access-qks42 podName:644d231e-7d3c-4aef-a201-87266e83d526 nodeName:}" failed. No retries permitted until 2026-07-28 15:33:43.980703115 +0000 UTC m=+4.165898285 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-qks42" (UniqueName: "kubernetes.io/projected/644d231e-7d3c-4aef-a201-87266e83d526-kube-api-access-qks42") pod "network-check-target-85ncq" (UID: "644d231e-7d3c-4aef-a201-87266e83d526") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 15:33:43.082470 ip-10-0-131-121 kubenswrapper[2568]: W0728 15:33:43.082435    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podde4142bb_fcb0_460c_9fbf_1025c61e2015.slice/crio-b84866d44d20c59802307f45fd69ff04e981f797496fb1d2b024211a84c60560 WatchSource:0}: Error finding container b84866d44d20c59802307f45fd69ff04e981f797496fb1d2b024211a84c60560: Status 404 returned error can't find the container with id b84866d44d20c59802307f45fd69ff04e981f797496fb1d2b024211a84c60560
Jul 28 15:33:43.103056 ip-10-0-131-121 kubenswrapper[2568]: W0728 15:33:43.102970    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3622f792_27df_4604_a1e3_47192c7302cb.slice/crio-71033157ac633c0ab137f2f03b9589d5e37dc356c44f94c3f1a1826419dd5c28 WatchSource:0}: Error finding container 71033157ac633c0ab137f2f03b9589d5e37dc356c44f94c3f1a1826419dd5c28: Status 404 returned error can't find the container with id 71033157ac633c0ab137f2f03b9589d5e37dc356c44f94c3f1a1826419dd5c28
Jul 28 15:33:43.104628 ip-10-0-131-121 kubenswrapper[2568]: W0728 15:33:43.104606    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddcab1041_8d3b_42e1_af05_55e847b831ad.slice/crio-f15a6d11193fbd60cf5152ac606a80784ac205d9dc2d2e199f37e6d264bd8afa WatchSource:0}: Error finding container f15a6d11193fbd60cf5152ac606a80784ac205d9dc2d2e199f37e6d264bd8afa: Status 404 returned error can't find the container with id f15a6d11193fbd60cf5152ac606a80784ac205d9dc2d2e199f37e6d264bd8afa
Jul 28 15:33:43.105117 ip-10-0-131-121 kubenswrapper[2568]: W0728 15:33:43.105090    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podebf75c0d_1597_4a1c_b668_dc2a041eb35e.slice/crio-063ba995679b41e19eff475eee6014f8f7528840ea59bc7c51a4c89169812937 WatchSource:0}: Error finding container 063ba995679b41e19eff475eee6014f8f7528840ea59bc7c51a4c89169812937: Status 404 returned error can't find the container with id 063ba995679b41e19eff475eee6014f8f7528840ea59bc7c51a4c89169812937
Jul 28 15:33:43.110281 ip-10-0-131-121 kubenswrapper[2568]: W0728 15:33:43.110263    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod17b1a488_5fd2_4cd5_871a_0f5d127493e2.slice/crio-d1d6fc7c3e24b0d56397c54fad10de1594b197c60d5c982bf95788c034c5223a WatchSource:0}: Error finding container d1d6fc7c3e24b0d56397c54fad10de1594b197c60d5c982bf95788c034c5223a: Status 404 returned error can't find the container with id d1d6fc7c3e24b0d56397c54fad10de1594b197c60d5c982bf95788c034c5223a
Jul 28 15:33:43.111414 ip-10-0-131-121 kubenswrapper[2568]: W0728 15:33:43.111390    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podad8ebf58_b767_4341_b60e_868c5b8b637f.slice/crio-6599567c97478ad4e137730dae7e521cefa213c34c04182c27b0230fbc0b4510 WatchSource:0}: Error finding container 6599567c97478ad4e137730dae7e521cefa213c34c04182c27b0230fbc0b4510: Status 404 returned error can't find the container with id 6599567c97478ad4e137730dae7e521cefa213c34c04182c27b0230fbc0b4510
Jul 28 15:33:43.113006 ip-10-0-131-121 kubenswrapper[2568]: W0728 15:33:43.112909    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod53a1cbf6_fc45_47f6_a8b1_5f95ec295587.slice/crio-077a7360496b52b5f454d48ee282139407b187ece467a8b0da6210dc30051466 WatchSource:0}: Error finding container 077a7360496b52b5f454d48ee282139407b187ece467a8b0da6210dc30051466: Status 404 returned error can't find the container with id 077a7360496b52b5f454d48ee282139407b187ece467a8b0da6210dc30051466
Jul 28 15:33:43.114200 ip-10-0-131-121 kubenswrapper[2568]: W0728 15:33:43.114161    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7e2caef2_3492_48d3_adbd_89fd3d4494d9.slice/crio-15ad5d24246781739e5fe0889c426fcdbbc3d6221203b2744607ef55d03b5b02 WatchSource:0}: Error finding container 15ad5d24246781739e5fe0889c426fcdbbc3d6221203b2744607ef55d03b5b02: Status 404 returned error can't find the container with id 15ad5d24246781739e5fe0889c426fcdbbc3d6221203b2744607ef55d03b5b02
Jul 28 15:33:43.115985 ip-10-0-131-121 kubenswrapper[2568]: W0728 15:33:43.115953    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfab92b7b_ed62_4783_86be_ed4bd56a3128.slice/crio-2e62292612b98558115aa0fdd87d9dba1846005c34f2630b249c8f5f331fb890 WatchSource:0}: Error finding container 2e62292612b98558115aa0fdd87d9dba1846005c34f2630b249c8f5f331fb890: Status 404 returned error can't find the container with id 2e62292612b98558115aa0fdd87d9dba1846005c34f2630b249c8f5f331fb890
Jul 28 15:33:43.213379 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:33:43.213216    2568 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-27 15:28:42 +0000 UTC" deadline="2028-03-27 02:25:45.710047266 +0000 UTC"
Jul 28 15:33:43.213379 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:33:43.213376    2568 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14578h52m2.496674585s"
Jul 28 15:33:43.300037 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:33:43.300002    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-121.ec2.internal" event={"ID":"0c5d5894bca76db08e7d990578b15cf0","Type":"ContainerStarted","Data":"5db65e65d4cb1395a6b1dd134f4d69f15e862f7ef7caa730480e5702c2831b0f"}
Jul 28 15:33:43.301149 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:33:43.301121    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-n2w6d" event={"ID":"ad8ebf58-b767-4341-b60e-868c5b8b637f","Type":"ContainerStarted","Data":"6599567c97478ad4e137730dae7e521cefa213c34c04182c27b0230fbc0b4510"}
Jul 28 15:33:43.302109 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:33:43.302090    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-ptrdx" event={"ID":"dcab1041-8d3b-42e1-af05-55e847b831ad","Type":"ContainerStarted","Data":"f15a6d11193fbd60cf5152ac606a80784ac205d9dc2d2e199f37e6d264bd8afa"}
Jul 28 15:33:43.303104 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:33:43.303071    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-49vz8" event={"ID":"de4142bb-fcb0-460c-9fbf-1025c61e2015","Type":"ContainerStarted","Data":"b84866d44d20c59802307f45fd69ff04e981f797496fb1d2b024211a84c60560"}
Jul 28 15:33:43.304408 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:33:43.304389    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-whdfr" event={"ID":"fab92b7b-ed62-4783-86be-ed4bd56a3128","Type":"ContainerStarted","Data":"2e62292612b98558115aa0fdd87d9dba1846005c34f2630b249c8f5f331fb890"}
Jul 28 15:33:43.305359 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:33:43.305343    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-nvbhc" event={"ID":"7e2caef2-3492-48d3-adbd-89fd3d4494d9","Type":"ContainerStarted","Data":"15ad5d24246781739e5fe0889c426fcdbbc3d6221203b2744607ef55d03b5b02"}
Jul 28 15:33:43.306356 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:33:43.306334    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-9hznr" event={"ID":"53a1cbf6-fc45-47f6-a8b1-5f95ec295587","Type":"ContainerStarted","Data":"077a7360496b52b5f454d48ee282139407b187ece467a8b0da6210dc30051466"}
Jul 28 15:33:43.310441 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:33:43.310392    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2nwbt" event={"ID":"17b1a488-5fd2-4cd5-871a-0f5d127493e2","Type":"ContainerStarted","Data":"d1d6fc7c3e24b0d56397c54fad10de1594b197c60d5c982bf95788c034c5223a"}
Jul 28 15:33:43.311477 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:33:43.311452    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-22nh9" event={"ID":"ebf75c0d-1597-4a1c-b668-dc2a041eb35e","Type":"ContainerStarted","Data":"063ba995679b41e19eff475eee6014f8f7528840ea59bc7c51a4c89169812937"}
Jul 28 15:33:43.312425 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:33:43.312407    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-2vrz4" event={"ID":"3622f792-27df-4604-a1e3-47192c7302cb","Type":"ContainerStarted","Data":"71033157ac633c0ab137f2f03b9589d5e37dc356c44f94c3f1a1826419dd5c28"}
Jul 28 15:33:43.314405 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:33:43.314368    2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-121.ec2.internal" podStartSLOduration=2.314357877 podStartE2EDuration="2.314357877s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 15:33:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 15:33:43.314298242 +0000 UTC m=+3.499493410" watchObservedRunningTime="2026-07-28 15:33:43.314357877 +0000 UTC m=+3.499553046"
Jul 28 15:33:43.886059 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:33:43.886018    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3ba72ab7-7ab4-4e7f-ace4-8a6b7bbff25c-original-pull-secret\") pod \"global-pull-secret-syncer-tr647\" (UID: \"3ba72ab7-7ab4-4e7f-ace4-8a6b7bbff25c\") " pod="kube-system/global-pull-secret-syncer-tr647"
Jul 28 15:33:43.886234 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:33:43.886088    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6100b515-cfe9-47ef-b672-d698ae677bd0-metrics-certs\") pod \"network-metrics-daemon-ckpr5\" (UID: \"6100b515-cfe9-47ef-b672-d698ae677bd0\") " pod="openshift-multus/network-metrics-daemon-ckpr5"
Jul 28 15:33:43.886234 ip-10-0-131-121 kubenswrapper[2568]: E0728 15:33:43.886210    2568 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 15:33:43.886364 ip-10-0-131-121 kubenswrapper[2568]: E0728 15:33:43.886275    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6100b515-cfe9-47ef-b672-d698ae677bd0-metrics-certs podName:6100b515-cfe9-47ef-b672-d698ae677bd0 nodeName:}" failed. No retries permitted until 2026-07-28 15:33:45.886255538 +0000 UTC m=+6.071450716 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6100b515-cfe9-47ef-b672-d698ae677bd0-metrics-certs") pod "network-metrics-daemon-ckpr5" (UID: "6100b515-cfe9-47ef-b672-d698ae677bd0") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 15:33:43.886753 ip-10-0-131-121 kubenswrapper[2568]: E0728 15:33:43.886733    2568 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 28 15:33:43.886875 ip-10-0-131-121 kubenswrapper[2568]: E0728 15:33:43.886808    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3ba72ab7-7ab4-4e7f-ace4-8a6b7bbff25c-original-pull-secret podName:3ba72ab7-7ab4-4e7f-ace4-8a6b7bbff25c nodeName:}" failed. No retries permitted until 2026-07-28 15:33:45.886792606 +0000 UTC m=+6.071987754 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/3ba72ab7-7ab4-4e7f-ace4-8a6b7bbff25c-original-pull-secret") pod "global-pull-secret-syncer-tr647" (UID: "3ba72ab7-7ab4-4e7f-ace4-8a6b7bbff25c") : object "kube-system"/"original-pull-secret" not registered
Jul 28 15:33:43.986866 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:33:43.986806    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qks42\" (UniqueName: \"kubernetes.io/projected/644d231e-7d3c-4aef-a201-87266e83d526-kube-api-access-qks42\") pod \"network-check-target-85ncq\" (UID: \"644d231e-7d3c-4aef-a201-87266e83d526\") " pod="openshift-network-diagnostics/network-check-target-85ncq"
Jul 28 15:33:43.987049 ip-10-0-131-121 kubenswrapper[2568]: E0728 15:33:43.986984    2568 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 28 15:33:43.987049 ip-10-0-131-121 kubenswrapper[2568]: E0728 15:33:43.987007    2568 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 28 15:33:43.987049 ip-10-0-131-121 kubenswrapper[2568]: E0728 15:33:43.987020    2568 projected.go:196] Error preparing data for projected volume kube-api-access-qks42 for pod openshift-network-diagnostics/network-check-target-85ncq: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 15:33:43.987205 ip-10-0-131-121 kubenswrapper[2568]: E0728 15:33:43.987075    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/644d231e-7d3c-4aef-a201-87266e83d526-kube-api-access-qks42 podName:644d231e-7d3c-4aef-a201-87266e83d526 nodeName:}" failed. No retries permitted until 2026-07-28 15:33:45.987057776 +0000 UTC m=+6.172252926 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-qks42" (UniqueName: "kubernetes.io/projected/644d231e-7d3c-4aef-a201-87266e83d526-kube-api-access-qks42") pod "network-check-target-85ncq" (UID: "644d231e-7d3c-4aef-a201-87266e83d526") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 15:33:44.213828 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:33:44.213690    2568 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-27 15:28:42 +0000 UTC" deadline="2028-02-05 16:55:49.297756118 +0000 UTC"
Jul 28 15:33:44.213828 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:33:44.213745    2568 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13369h22m5.084015281s"
Jul 28 15:33:44.293105 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:33:44.292278    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tr647"
Jul 28 15:33:44.293105 ip-10-0-131-121 kubenswrapper[2568]: E0728 15:33:44.292404    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tr647" podUID="3ba72ab7-7ab4-4e7f-ace4-8a6b7bbff25c"
Jul 28 15:33:44.293105 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:33:44.292868    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-85ncq"
Jul 28 15:33:44.293105 ip-10-0-131-121 kubenswrapper[2568]: E0728 15:33:44.293035    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-85ncq" podUID="644d231e-7d3c-4aef-a201-87266e83d526"
Jul 28 15:33:44.293682 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:33:44.293499    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ckpr5"
Jul 28 15:33:44.293682 ip-10-0-131-121 kubenswrapper[2568]: E0728 15:33:44.293603    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ckpr5" podUID="6100b515-cfe9-47ef-b672-d698ae677bd0"
Jul 28 15:33:44.323527 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:33:44.323487    2568 generic.go:356] "Generic (PLEG): container finished" podID="c3ed888f968446d5f752a5eb72daf11a" containerID="9bc47ed2d587e1a9aa5f2c78b6a326ed028f279bb213ef9fffbb35264740b585" exitCode=0
Jul 28 15:33:44.324585 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:33:44.324556    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-121.ec2.internal" event={"ID":"c3ed888f968446d5f752a5eb72daf11a","Type":"ContainerDied","Data":"9bc47ed2d587e1a9aa5f2c78b6a326ed028f279bb213ef9fffbb35264740b585"}
Jul 28 15:33:45.349244 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:33:45.349157    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-121.ec2.internal" event={"ID":"c3ed888f968446d5f752a5eb72daf11a","Type":"ContainerStarted","Data":"7742baa4adfc1993e614a86fa10ed0ff288a0bd02213908bfb39905deb5f2530"}
Jul 28 15:33:45.376413 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:33:45.375892    2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-121.ec2.internal" podStartSLOduration=4.375869756 podStartE2EDuration="4.375869756s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 15:33:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 15:33:45.37515255 +0000 UTC m=+5.560347720" watchObservedRunningTime="2026-07-28 15:33:45.375869756 +0000 UTC m=+5.561064925"
Jul 28 15:33:45.908264 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:33:45.907432    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3ba72ab7-7ab4-4e7f-ace4-8a6b7bbff25c-original-pull-secret\") pod \"global-pull-secret-syncer-tr647\" (UID: \"3ba72ab7-7ab4-4e7f-ace4-8a6b7bbff25c\") " pod="kube-system/global-pull-secret-syncer-tr647"
Jul 28 15:33:45.908264 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:33:45.907493    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6100b515-cfe9-47ef-b672-d698ae677bd0-metrics-certs\") pod \"network-metrics-daemon-ckpr5\" (UID: \"6100b515-cfe9-47ef-b672-d698ae677bd0\") " pod="openshift-multus/network-metrics-daemon-ckpr5"
Jul 28 15:33:45.908264 ip-10-0-131-121 kubenswrapper[2568]: E0728 15:33:45.907669    2568 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 15:33:45.908264 ip-10-0-131-121 kubenswrapper[2568]: E0728 15:33:45.907734    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6100b515-cfe9-47ef-b672-d698ae677bd0-metrics-certs podName:6100b515-cfe9-47ef-b672-d698ae677bd0 nodeName:}" failed. No retries permitted until 2026-07-28 15:33:49.907713673 +0000 UTC m=+10.092908841 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6100b515-cfe9-47ef-b672-d698ae677bd0-metrics-certs") pod "network-metrics-daemon-ckpr5" (UID: "6100b515-cfe9-47ef-b672-d698ae677bd0") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 15:33:45.908264 ip-10-0-131-121 kubenswrapper[2568]: E0728 15:33:45.908142    2568 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 28 15:33:45.908264 ip-10-0-131-121 kubenswrapper[2568]: E0728 15:33:45.908193    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3ba72ab7-7ab4-4e7f-ace4-8a6b7bbff25c-original-pull-secret podName:3ba72ab7-7ab4-4e7f-ace4-8a6b7bbff25c nodeName:}" failed. No retries permitted until 2026-07-28 15:33:49.908177095 +0000 UTC m=+10.093372260 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/3ba72ab7-7ab4-4e7f-ace4-8a6b7bbff25c-original-pull-secret") pod "global-pull-secret-syncer-tr647" (UID: "3ba72ab7-7ab4-4e7f-ace4-8a6b7bbff25c") : object "kube-system"/"original-pull-secret" not registered
Jul 28 15:33:46.008859 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:33:46.008256    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qks42\" (UniqueName: \"kubernetes.io/projected/644d231e-7d3c-4aef-a201-87266e83d526-kube-api-access-qks42\") pod \"network-check-target-85ncq\" (UID: \"644d231e-7d3c-4aef-a201-87266e83d526\") " pod="openshift-network-diagnostics/network-check-target-85ncq"
Jul 28 15:33:46.008859 ip-10-0-131-121 kubenswrapper[2568]: E0728 15:33:46.008413    2568 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 28 15:33:46.008859 ip-10-0-131-121 kubenswrapper[2568]: E0728 15:33:46.008431    2568 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 28 15:33:46.008859 ip-10-0-131-121 kubenswrapper[2568]: E0728 15:33:46.008443    2568 projected.go:196] Error preparing data for projected volume kube-api-access-qks42 for pod openshift-network-diagnostics/network-check-target-85ncq: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 15:33:46.008859 ip-10-0-131-121 kubenswrapper[2568]: E0728 15:33:46.008499    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/644d231e-7d3c-4aef-a201-87266e83d526-kube-api-access-qks42 podName:644d231e-7d3c-4aef-a201-87266e83d526 nodeName:}" failed. No retries permitted until 2026-07-28 15:33:50.008481451 +0000 UTC m=+10.193676605 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-qks42" (UniqueName: "kubernetes.io/projected/644d231e-7d3c-4aef-a201-87266e83d526-kube-api-access-qks42") pod "network-check-target-85ncq" (UID: "644d231e-7d3c-4aef-a201-87266e83d526") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 15:33:46.290102 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:33:46.289790    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tr647"
Jul 28 15:33:46.290102 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:33:46.290006    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ckpr5"
Jul 28 15:33:46.290102 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:33:46.290016    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-85ncq"
Jul 28 15:33:46.290358 ip-10-0-131-121 kubenswrapper[2568]: E0728 15:33:46.290187    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-85ncq" podUID="644d231e-7d3c-4aef-a201-87266e83d526"
Jul 28 15:33:46.290358 ip-10-0-131-121 kubenswrapper[2568]: E0728 15:33:46.290101    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ckpr5" podUID="6100b515-cfe9-47ef-b672-d698ae677bd0"
Jul 28 15:33:46.290358 ip-10-0-131-121 kubenswrapper[2568]: E0728 15:33:46.290281    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tr647" podUID="3ba72ab7-7ab4-4e7f-ace4-8a6b7bbff25c"
Jul 28 15:33:48.290160 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:33:48.290128    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-85ncq"
Jul 28 15:33:48.290615 ip-10-0-131-121 kubenswrapper[2568]: E0728 15:33:48.290259    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-85ncq" podUID="644d231e-7d3c-4aef-a201-87266e83d526"
Jul 28 15:33:48.290696 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:33:48.290678    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tr647"
Jul 28 15:33:48.290828 ip-10-0-131-121 kubenswrapper[2568]: E0728 15:33:48.290798    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tr647" podUID="3ba72ab7-7ab4-4e7f-ace4-8a6b7bbff25c"
Jul 28 15:33:48.290925 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:33:48.290907    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ckpr5"
Jul 28 15:33:48.291068 ip-10-0-131-121 kubenswrapper[2568]: E0728 15:33:48.290991    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ckpr5" podUID="6100b515-cfe9-47ef-b672-d698ae677bd0"
Jul 28 15:33:49.942241 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:33:49.942205    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3ba72ab7-7ab4-4e7f-ace4-8a6b7bbff25c-original-pull-secret\") pod \"global-pull-secret-syncer-tr647\" (UID: \"3ba72ab7-7ab4-4e7f-ace4-8a6b7bbff25c\") " pod="kube-system/global-pull-secret-syncer-tr647"
Jul 28 15:33:49.942697 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:33:49.942266    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6100b515-cfe9-47ef-b672-d698ae677bd0-metrics-certs\") pod \"network-metrics-daemon-ckpr5\" (UID: \"6100b515-cfe9-47ef-b672-d698ae677bd0\") " pod="openshift-multus/network-metrics-daemon-ckpr5"
Jul 28 15:33:49.942697 ip-10-0-131-121 kubenswrapper[2568]: E0728 15:33:49.942458    2568 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 15:33:49.942697 ip-10-0-131-121 kubenswrapper[2568]: E0728 15:33:49.942522    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6100b515-cfe9-47ef-b672-d698ae677bd0-metrics-certs podName:6100b515-cfe9-47ef-b672-d698ae677bd0 nodeName:}" failed. No retries permitted until 2026-07-28 15:33:57.942502784 +0000 UTC m=+18.127697936 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6100b515-cfe9-47ef-b672-d698ae677bd0-metrics-certs") pod "network-metrics-daemon-ckpr5" (UID: "6100b515-cfe9-47ef-b672-d698ae677bd0") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 15:33:49.942956 ip-10-0-131-121 kubenswrapper[2568]: E0728 15:33:49.942939    2568 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 28 15:33:49.943018 ip-10-0-131-121 kubenswrapper[2568]: E0728 15:33:49.942989    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3ba72ab7-7ab4-4e7f-ace4-8a6b7bbff25c-original-pull-secret podName:3ba72ab7-7ab4-4e7f-ace4-8a6b7bbff25c nodeName:}" failed. No retries permitted until 2026-07-28 15:33:57.942974918 +0000 UTC m=+18.128170068 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/3ba72ab7-7ab4-4e7f-ace4-8a6b7bbff25c-original-pull-secret") pod "global-pull-secret-syncer-tr647" (UID: "3ba72ab7-7ab4-4e7f-ace4-8a6b7bbff25c") : object "kube-system"/"original-pull-secret" not registered
Jul 28 15:33:50.043502 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:33:50.043456    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qks42\" (UniqueName: \"kubernetes.io/projected/644d231e-7d3c-4aef-a201-87266e83d526-kube-api-access-qks42\") pod \"network-check-target-85ncq\" (UID: \"644d231e-7d3c-4aef-a201-87266e83d526\") " pod="openshift-network-diagnostics/network-check-target-85ncq"
Jul 28 15:33:50.043723 ip-10-0-131-121 kubenswrapper[2568]: E0728 15:33:50.043706    2568 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 28 15:33:50.043802 ip-10-0-131-121 kubenswrapper[2568]: E0728 15:33:50.043728    2568 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 28 15:33:50.043802 ip-10-0-131-121 kubenswrapper[2568]: E0728 15:33:50.043741    2568 projected.go:196] Error preparing data for projected volume kube-api-access-qks42 for pod openshift-network-diagnostics/network-check-target-85ncq: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 15:33:50.043919 ip-10-0-131-121 kubenswrapper[2568]: E0728 15:33:50.043815    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/644d231e-7d3c-4aef-a201-87266e83d526-kube-api-access-qks42 podName:644d231e-7d3c-4aef-a201-87266e83d526 nodeName:}" failed. No retries permitted until 2026-07-28 15:33:58.043796542 +0000 UTC m=+18.228991744 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-qks42" (UniqueName: "kubernetes.io/projected/644d231e-7d3c-4aef-a201-87266e83d526-kube-api-access-qks42") pod "network-check-target-85ncq" (UID: "644d231e-7d3c-4aef-a201-87266e83d526") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 15:33:50.290735 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:33:50.290650    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ckpr5"
Jul 28 15:33:50.290915 ip-10-0-131-121 kubenswrapper[2568]: E0728 15:33:50.290768    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ckpr5" podUID="6100b515-cfe9-47ef-b672-d698ae677bd0"
Jul 28 15:33:50.291155 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:33:50.291138    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tr647"
Jul 28 15:33:50.291245 ip-10-0-131-121 kubenswrapper[2568]: E0728 15:33:50.291222    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tr647" podUID="3ba72ab7-7ab4-4e7f-ace4-8a6b7bbff25c"
Jul 28 15:33:50.291369 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:33:50.291284    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-85ncq"
Jul 28 15:33:50.291430 ip-10-0-131-121 kubenswrapper[2568]: E0728 15:33:50.291382    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-85ncq" podUID="644d231e-7d3c-4aef-a201-87266e83d526"
Jul 28 15:33:52.292821 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:33:52.292683    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tr647"
Jul 28 15:33:52.293343 ip-10-0-131-121 kubenswrapper[2568]: E0728 15:33:52.292835    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tr647" podUID="3ba72ab7-7ab4-4e7f-ace4-8a6b7bbff25c"
Jul 28 15:33:52.293343 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:33:52.292857    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-85ncq"
Jul 28 15:33:52.293343 ip-10-0-131-121 kubenswrapper[2568]: E0728 15:33:52.293000    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-85ncq" podUID="644d231e-7d3c-4aef-a201-87266e83d526"
Jul 28 15:33:52.293343 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:33:52.292840    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ckpr5"
Jul 28 15:33:52.293343 ip-10-0-131-121 kubenswrapper[2568]: E0728 15:33:52.293124    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ckpr5" podUID="6100b515-cfe9-47ef-b672-d698ae677bd0"
Jul 28 15:33:54.289205 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:33:54.289148    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tr647"
Jul 28 15:33:54.289205 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:33:54.289172    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ckpr5"
Jul 28 15:33:54.289728 ip-10-0-131-121 kubenswrapper[2568]: E0728 15:33:54.289293    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tr647" podUID="3ba72ab7-7ab4-4e7f-ace4-8a6b7bbff25c"
Jul 28 15:33:54.289728 ip-10-0-131-121 kubenswrapper[2568]: E0728 15:33:54.289375    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ckpr5" podUID="6100b515-cfe9-47ef-b672-d698ae677bd0"
Jul 28 15:33:54.289728 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:33:54.289434    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-85ncq"
Jul 28 15:33:54.289728 ip-10-0-131-121 kubenswrapper[2568]: E0728 15:33:54.289558    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-85ncq" podUID="644d231e-7d3c-4aef-a201-87266e83d526"
Jul 28 15:33:56.292502 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:33:56.292466    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ckpr5"
Jul 28 15:33:56.292502 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:33:56.292479    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tr647"
Jul 28 15:33:56.293032 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:33:56.292509    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-85ncq"
Jul 28 15:33:56.293032 ip-10-0-131-121 kubenswrapper[2568]: E0728 15:33:56.292608    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ckpr5" podUID="6100b515-cfe9-47ef-b672-d698ae677bd0"
Jul 28 15:33:56.293032 ip-10-0-131-121 kubenswrapper[2568]: E0728 15:33:56.292667    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tr647" podUID="3ba72ab7-7ab4-4e7f-ace4-8a6b7bbff25c"
Jul 28 15:33:56.293032 ip-10-0-131-121 kubenswrapper[2568]: E0728 15:33:56.292733    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-85ncq" podUID="644d231e-7d3c-4aef-a201-87266e83d526"
Jul 28 15:33:58.005628 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:33:58.005588    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6100b515-cfe9-47ef-b672-d698ae677bd0-metrics-certs\") pod \"network-metrics-daemon-ckpr5\" (UID: \"6100b515-cfe9-47ef-b672-d698ae677bd0\") " pod="openshift-multus/network-metrics-daemon-ckpr5"
Jul 28 15:33:58.006183 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:33:58.005679    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3ba72ab7-7ab4-4e7f-ace4-8a6b7bbff25c-original-pull-secret\") pod \"global-pull-secret-syncer-tr647\" (UID: \"3ba72ab7-7ab4-4e7f-ace4-8a6b7bbff25c\") " pod="kube-system/global-pull-secret-syncer-tr647"
Jul 28 15:33:58.006183 ip-10-0-131-121 kubenswrapper[2568]: E0728 15:33:58.005793    2568 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 15:33:58.006183 ip-10-0-131-121 kubenswrapper[2568]: E0728 15:33:58.005864    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6100b515-cfe9-47ef-b672-d698ae677bd0-metrics-certs podName:6100b515-cfe9-47ef-b672-d698ae677bd0 nodeName:}" failed. No retries permitted until 2026-07-28 15:34:14.005846529 +0000 UTC m=+34.191041677 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6100b515-cfe9-47ef-b672-d698ae677bd0-metrics-certs") pod "network-metrics-daemon-ckpr5" (UID: "6100b515-cfe9-47ef-b672-d698ae677bd0") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 15:33:58.006183 ip-10-0-131-121 kubenswrapper[2568]: E0728 15:33:58.005800    2568 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 28 15:33:58.006183 ip-10-0-131-121 kubenswrapper[2568]: E0728 15:33:58.005941    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3ba72ab7-7ab4-4e7f-ace4-8a6b7bbff25c-original-pull-secret podName:3ba72ab7-7ab4-4e7f-ace4-8a6b7bbff25c nodeName:}" failed. No retries permitted until 2026-07-28 15:34:14.005923273 +0000 UTC m=+34.191118425 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/3ba72ab7-7ab4-4e7f-ace4-8a6b7bbff25c-original-pull-secret") pod "global-pull-secret-syncer-tr647" (UID: "3ba72ab7-7ab4-4e7f-ace4-8a6b7bbff25c") : object "kube-system"/"original-pull-secret" not registered
Jul 28 15:33:58.106186 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:33:58.106150    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qks42\" (UniqueName: \"kubernetes.io/projected/644d231e-7d3c-4aef-a201-87266e83d526-kube-api-access-qks42\") pod \"network-check-target-85ncq\" (UID: \"644d231e-7d3c-4aef-a201-87266e83d526\") " pod="openshift-network-diagnostics/network-check-target-85ncq"
Jul 28 15:33:58.106376 ip-10-0-131-121 kubenswrapper[2568]: E0728 15:33:58.106298    2568 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 28 15:33:58.106376 ip-10-0-131-121 kubenswrapper[2568]: E0728 15:33:58.106324    2568 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 28 15:33:58.106376 ip-10-0-131-121 kubenswrapper[2568]: E0728 15:33:58.106336    2568 projected.go:196] Error preparing data for projected volume kube-api-access-qks42 for pod openshift-network-diagnostics/network-check-target-85ncq: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 15:33:58.106511 ip-10-0-131-121 kubenswrapper[2568]: E0728 15:33:58.106402    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/644d231e-7d3c-4aef-a201-87266e83d526-kube-api-access-qks42 podName:644d231e-7d3c-4aef-a201-87266e83d526 nodeName:}" failed. No retries permitted until 2026-07-28 15:34:14.106382963 +0000 UTC m=+34.291578132 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-qks42" (UniqueName: "kubernetes.io/projected/644d231e-7d3c-4aef-a201-87266e83d526-kube-api-access-qks42") pod "network-check-target-85ncq" (UID: "644d231e-7d3c-4aef-a201-87266e83d526") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 15:33:58.292635 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:33:58.292560    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tr647"
Jul 28 15:33:58.292819 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:33:58.292656    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ckpr5"
Jul 28 15:33:58.292819 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:33:58.292674    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-85ncq"
Jul 28 15:33:58.292819 ip-10-0-131-121 kubenswrapper[2568]: E0728 15:33:58.292670    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tr647" podUID="3ba72ab7-7ab4-4e7f-ace4-8a6b7bbff25c"
Jul 28 15:33:58.292819 ip-10-0-131-121 kubenswrapper[2568]: E0728 15:33:58.292767    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ckpr5" podUID="6100b515-cfe9-47ef-b672-d698ae677bd0"
Jul 28 15:33:58.293027 ip-10-0-131-121 kubenswrapper[2568]: E0728 15:33:58.292861    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-85ncq" podUID="644d231e-7d3c-4aef-a201-87266e83d526"
Jul 28 15:34:00.290619 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:34:00.290584    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ckpr5"
Jul 28 15:34:00.291124 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:34:00.290733    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tr647"
Jul 28 15:34:00.291124 ip-10-0-131-121 kubenswrapper[2568]: E0728 15:34:00.290739    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ckpr5" podUID="6100b515-cfe9-47ef-b672-d698ae677bd0"
Jul 28 15:34:00.291124 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:34:00.290745    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-85ncq"
Jul 28 15:34:00.291124 ip-10-0-131-121 kubenswrapper[2568]: E0728 15:34:00.290870    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tr647" podUID="3ba72ab7-7ab4-4e7f-ace4-8a6b7bbff25c"
Jul 28 15:34:00.291124 ip-10-0-131-121 kubenswrapper[2568]: E0728 15:34:00.291027    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-85ncq" podUID="644d231e-7d3c-4aef-a201-87266e83d526"
Jul 28 15:34:02.290075 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:34:02.289724    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tr647"
Jul 28 15:34:02.290668 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:34:02.289736    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ckpr5"
Jul 28 15:34:02.290668 ip-10-0-131-121 kubenswrapper[2568]: E0728 15:34:02.290165    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tr647" podUID="3ba72ab7-7ab4-4e7f-ace4-8a6b7bbff25c"
Jul 28 15:34:02.290668 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:34:02.289812    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-85ncq"
Jul 28 15:34:02.290668 ip-10-0-131-121 kubenswrapper[2568]: E0728 15:34:02.290270    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ckpr5" podUID="6100b515-cfe9-47ef-b672-d698ae677bd0"
Jul 28 15:34:02.290668 ip-10-0-131-121 kubenswrapper[2568]: E0728 15:34:02.290369    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-85ncq" podUID="644d231e-7d3c-4aef-a201-87266e83d526"
Jul 28 15:34:02.396501 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:34:02.396408    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-n2w6d" event={"ID":"ad8ebf58-b767-4341-b60e-868c5b8b637f","Type":"ContainerStarted","Data":"1e0740f10897b3589f099ebce200f28da2eaede53d14619f8a9a5f132a157422"}
Jul 28 15:34:02.397961 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:34:02.397935    2568 generic.go:356] "Generic (PLEG): container finished" podID="dcab1041-8d3b-42e1-af05-55e847b831ad" containerID="66cca4c686a76c24d1eb2ee654238a0a9d4158c1895b0b31e31792d105b50b75" exitCode=0
Jul 28 15:34:02.398094 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:34:02.398007    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-ptrdx" event={"ID":"dcab1041-8d3b-42e1-af05-55e847b831ad","Type":"ContainerDied","Data":"66cca4c686a76c24d1eb2ee654238a0a9d4158c1895b0b31e31792d105b50b75"}
Jul 28 15:34:02.399596 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:34:02.399528    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-whdfr" event={"ID":"fab92b7b-ed62-4783-86be-ed4bd56a3128","Type":"ContainerStarted","Data":"d114e6c84c4f1475b42e48b1e61f8317f3a8a675b49ff730be3b3e0cc6e69619"}
Jul 28 15:34:02.401398 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:34:02.401347    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-nvbhc" event={"ID":"7e2caef2-3492-48d3-adbd-89fd3d4494d9","Type":"ContainerStarted","Data":"b16b314f4519d372ada796dbb32eb3cd1623d955d9ca6ac0d2977fd570a154f7"}
Jul 28 15:34:02.402959 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:34:02.402927    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-9hznr" event={"ID":"53a1cbf6-fc45-47f6-a8b1-5f95ec295587","Type":"ContainerStarted","Data":"cef65d5d6a1061d2f95374fd04d0033f6afec09d84409cbd27748cc366d03af9"}
Jul 28 15:34:02.404435 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:34:02.404414    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2nwbt" event={"ID":"17b1a488-5fd2-4cd5-871a-0f5d127493e2","Type":"ContainerStarted","Data":"feb2f74ac2afa423aefc65f37721c405635ad579c2031a46b76ba87870b10a08"}
Jul 28 15:34:02.405941 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:34:02.405917    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-22nh9" event={"ID":"ebf75c0d-1597-4a1c-b668-dc2a041eb35e","Type":"ContainerStarted","Data":"808e2cb21f907710f826e348a60a7db9472c6b4b321adad3de39e956a633cbf4"}
Jul 28 15:34:02.409321 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:34:02.409287    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-2vrz4" event={"ID":"3622f792-27df-4604-a1e3-47192c7302cb","Type":"ContainerStarted","Data":"b5d010ad65bd95a0ac55c90d109ffb35f2b9fe27c29e88cad88292fe0115900f"}
Jul 28 15:34:02.409420 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:34:02.409374    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-2vrz4" event={"ID":"3622f792-27df-4604-a1e3-47192c7302cb","Type":"ContainerStarted","Data":"e9dddaf9ab8f61e11f0b8c96b6179eaf8786f66b80f093619f6fe3e773d84174"}
Jul 28 15:34:02.409459 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:34:02.409426    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-2vrz4" event={"ID":"3622f792-27df-4604-a1e3-47192c7302cb","Type":"ContainerStarted","Data":"e1e19d7c9abe45f985f870bf111bc174775006d60c51b2f1985406092b93e104"}
Jul 28 15:34:02.409496 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:34:02.409476    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-2vrz4" event={"ID":"3622f792-27df-4604-a1e3-47192c7302cb","Type":"ContainerStarted","Data":"382b3cdd3018be6741193de18f8529ae08270bafaae451b205a9392cdec00661"}
Jul 28 15:34:02.409530 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:34:02.409522    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-2vrz4" event={"ID":"3622f792-27df-4604-a1e3-47192c7302cb","Type":"ContainerStarted","Data":"549643cc1fdd2537d825fd8d9f0316fe1ed8c0fc2024068bc18360909fad7efa"}
Jul 28 15:34:02.409577 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:34:02.409564    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-2vrz4" event={"ID":"3622f792-27df-4604-a1e3-47192c7302cb","Type":"ContainerStarted","Data":"03ebc96effc01610708d459c975053a74a2f2c9aaef31e1c00ca1d9256af5323"}
Jul 28 15:34:02.418106 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:34:02.418059    2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-n2w6d" podStartSLOduration=3.82702277 podStartE2EDuration="22.418042333s" totalImagesPullingTime="18.591019563s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 15:33:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 15:34:02.417484839 +0000 UTC m=+22.602680004" watchObservedRunningTime="2026-07-28 15:34:02.418042333 +0000 UTC m=+22.603237571"
Jul 28 15:34:02.434603 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:34:02.434545    2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-whdfr" podStartSLOduration=3.876873791 podStartE2EDuration="22.4345275s" totalImagesPullingTime="18.557653709s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 15:33:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 15:34:02.434511963 +0000 UTC m=+22.619707135" watchObservedRunningTime="2026-07-28 15:34:02.4345275 +0000 UTC m=+22.619722670"
Jul 28 15:34:02.451240 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:34:02.451181    2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-22nh9" podStartSLOduration=8.260570491 podStartE2EDuration="22.451159745s" totalImagesPullingTime="14.190589254s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 15:33:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 15:34:02.451057291 +0000 UTC m=+22.636252465" watchObservedRunningTime="2026-07-28 15:34:02.451159745 +0000 UTC m=+22.636354914"
Jul 28 15:34:02.471087 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:34:02.471023    2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-nvbhc" podStartSLOduration=3.861468442 podStartE2EDuration="22.470999811s" totalImagesPullingTime="18.609531369s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 15:33:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 15:34:02.47056957 +0000 UTC m=+22.655764740" watchObservedRunningTime="2026-07-28 15:34:02.470999811 +0000 UTC m=+22.656194982"
Jul 28 15:34:02.487207 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:34:02.487145    2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-9hznr" podStartSLOduration=8.304003674 podStartE2EDuration="22.487122931s" totalImagesPullingTime="14.183119257s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 15:33:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 15:34:02.486758122 +0000 UTC m=+22.671953305" watchObservedRunningTime="2026-07-28 15:34:02.487122931 +0000 UTC m=+22.672318104"
Jul 28 15:34:02.964656 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:34:02.964623    2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-22nh9"
Jul 28 15:34:02.965179 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:34:02.965163    2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-22nh9"
Jul 28 15:34:03.180119 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:34:03.180085    2568 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Jul 28 15:34:03.228484 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:34:03.228297    2568 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-28T15:34:03.180111692Z","UUID":"b732de88-58e4-4679-83b0-2be97f0d7d0d","Handler":null,"Name":"","Endpoint":""}
Jul 28 15:34:03.230107 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:34:03.230088    2568 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Jul 28 15:34:03.230249 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:34:03.230129    2568 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Jul 28 15:34:03.413612 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:34:03.413583    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2nwbt" event={"ID":"17b1a488-5fd2-4cd5-871a-0f5d127493e2","Type":"ContainerStarted","Data":"aa8a508ef58ec74991b2967cb2eef1a7d099e6273ff854bea23b2c09661d9a26"}
Jul 28 15:34:03.415352 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:34:03.415286    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-49vz8" event={"ID":"de4142bb-fcb0-460c-9fbf-1025c61e2015","Type":"ContainerStarted","Data":"c71d4904c6a0104fd270d6463cb87bccf44351693fbd9ca7443930173352c6b3"}
Jul 28 15:34:03.431727 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:34:03.431661    2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-49vz8" podStartSLOduration=4.860068764 podStartE2EDuration="23.431639942s" totalImagesPullingTime="18.571571178s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 15:33:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 15:34:03.431493009 +0000 UTC m=+23.616688191" watchObservedRunningTime="2026-07-28 15:34:03.431639942 +0000 UTC m=+23.616835113"
Jul 28 15:34:04.289792 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:34:04.289750    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-85ncq"
Jul 28 15:34:04.289956 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:34:04.289750    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ckpr5"
Jul 28 15:34:04.289956 ip-10-0-131-121 kubenswrapper[2568]: E0728 15:34:04.289918    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-85ncq" podUID="644d231e-7d3c-4aef-a201-87266e83d526"
Jul 28 15:34:04.290119 ip-10-0-131-121 kubenswrapper[2568]: E0728 15:34:04.289974    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ckpr5" podUID="6100b515-cfe9-47ef-b672-d698ae677bd0"
Jul 28 15:34:04.290119 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:34:04.289990    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tr647"
Jul 28 15:34:04.290119 ip-10-0-131-121 kubenswrapper[2568]: E0728 15:34:04.290075    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tr647" podUID="3ba72ab7-7ab4-4e7f-ace4-8a6b7bbff25c"
Jul 28 15:34:04.420165 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:34:04.420124    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2nwbt" event={"ID":"17b1a488-5fd2-4cd5-871a-0f5d127493e2","Type":"ContainerStarted","Data":"2ad177fd889b60d9701d03f9c2f20b6bd0dd8e3c0e4ff37eaa88ecf954da1ab4"}
Jul 28 15:34:04.420586 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:34:04.420333    2568 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness"
Jul 28 15:34:05.426519 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:34:05.426307    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-2vrz4" event={"ID":"3622f792-27df-4604-a1e3-47192c7302cb","Type":"ContainerStarted","Data":"51efdcbcdcbf6c048ac0e280ded628db204c1499260ba83213e916a8f71b05e1"}
Jul 28 15:34:05.445759 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:34:05.445706    2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2nwbt" podStartSLOduration=4.570633884 podStartE2EDuration="25.445687486s" totalImagesPullingTime="20.875053602s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 15:33:40 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 15:34:05.445068024 +0000 UTC m=+25.630263209" watchObservedRunningTime="2026-07-28 15:34:05.445687486 +0000 UTC m=+25.630882655"
Jul 28 15:34:06.289400 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:34:06.289362    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-85ncq"
Jul 28 15:34:06.289631 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:34:06.289362    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ckpr5"
Jul 28 15:34:06.289631 ip-10-0-131-121 kubenswrapper[2568]: E0728 15:34:06.289496    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-85ncq" podUID="644d231e-7d3c-4aef-a201-87266e83d526"
Jul 28 15:34:06.289631 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:34:06.289365    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tr647"
Jul 28 15:34:06.289631 ip-10-0-131-121 kubenswrapper[2568]: E0728 15:34:06.289582    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ckpr5" podUID="6100b515-cfe9-47ef-b672-d698ae677bd0"
Jul 28 15:34:06.289857 ip-10-0-131-121 kubenswrapper[2568]: E0728 15:34:06.289641    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tr647" podUID="3ba72ab7-7ab4-4e7f-ace4-8a6b7bbff25c"
Jul 28 15:34:07.434906 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:34:07.434668    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-2vrz4" event={"ID":"3622f792-27df-4604-a1e3-47192c7302cb","Type":"ContainerStarted","Data":"87b101581a9e53d68fa99acbc45b281b4115125aab214e672b1373e494d9c611"}
Jul 28 15:34:07.435718 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:34:07.434997    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-2vrz4"
Jul 28 15:34:07.435718 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:34:07.435043    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-2vrz4"
Jul 28 15:34:07.436661 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:34:07.436627    2568 generic.go:356] "Generic (PLEG): container finished" podID="dcab1041-8d3b-42e1-af05-55e847b831ad" containerID="9f7e9c4698bedd34b0cd28186b77c86d173b876318e4634d69ced0ec5dcb15ce" exitCode=0
Jul 28 15:34:07.436751 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:34:07.436676    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-ptrdx" event={"ID":"dcab1041-8d3b-42e1-af05-55e847b831ad","Type":"ContainerDied","Data":"9f7e9c4698bedd34b0cd28186b77c86d173b876318e4634d69ced0ec5dcb15ce"}
Jul 28 15:34:07.450464 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:34:07.450439    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-2vrz4"
Jul 28 15:34:07.494660 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:34:07.494611    2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-2vrz4" podStartSLOduration=8.848758735 podStartE2EDuration="27.494596188s" totalImagesPullingTime="18.645837453s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 15:33:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 15:34:07.465953196 +0000 UTC m=+27.651148365" watchObservedRunningTime="2026-07-28 15:34:07.494596188 +0000 UTC m=+27.679791357"
Jul 28 15:34:08.289377 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:34:08.289341    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ckpr5"
Jul 28 15:34:08.289587 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:34:08.289344    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tr647"
Jul 28 15:34:08.289587 ip-10-0-131-121 kubenswrapper[2568]: E0728 15:34:08.289471    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ckpr5" podUID="6100b515-cfe9-47ef-b672-d698ae677bd0"
Jul 28 15:34:08.289587 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:34:08.289481    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-85ncq"
Jul 28 15:34:08.289587 ip-10-0-131-121 kubenswrapper[2568]: E0728 15:34:08.289566    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tr647" podUID="3ba72ab7-7ab4-4e7f-ace4-8a6b7bbff25c"
Jul 28 15:34:08.289792 ip-10-0-131-121 kubenswrapper[2568]: E0728 15:34:08.289660    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-85ncq" podUID="644d231e-7d3c-4aef-a201-87266e83d526"
Jul 28 15:34:08.439382 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:34:08.439305    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-2vrz4"
Jul 28 15:34:08.453906 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:34:08.453881    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-2vrz4"
Jul 28 15:34:08.905037 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:34:08.905003    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-ckpr5"]
Jul 28 15:34:08.905227 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:34:08.905163    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ckpr5"
Jul 28 15:34:08.905289 ip-10-0-131-121 kubenswrapper[2568]: E0728 15:34:08.905268    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ckpr5" podUID="6100b515-cfe9-47ef-b672-d698ae677bd0"
Jul 28 15:34:08.905760 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:34:08.905741    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-85ncq"]
Jul 28 15:34:08.905872 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:34:08.905851    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-85ncq"
Jul 28 15:34:08.905994 ip-10-0-131-121 kubenswrapper[2568]: E0728 15:34:08.905932    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-85ncq" podUID="644d231e-7d3c-4aef-a201-87266e83d526"
Jul 28 15:34:08.906331 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:34:08.906294    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-tr647"]
Jul 28 15:34:08.906420 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:34:08.906378    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tr647"
Jul 28 15:34:08.906466 ip-10-0-131-121 kubenswrapper[2568]: E0728 15:34:08.906451    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tr647" podUID="3ba72ab7-7ab4-4e7f-ace4-8a6b7bbff25c"
Jul 28 15:34:09.443758 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:34:09.443722    2568 generic.go:356] "Generic (PLEG): container finished" podID="dcab1041-8d3b-42e1-af05-55e847b831ad" containerID="c4680eda7e834ba571c363cddcfcb087d01a7b9700c480d012946b2859fdb050" exitCode=0
Jul 28 15:34:09.444249 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:34:09.443811    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-ptrdx" event={"ID":"dcab1041-8d3b-42e1-af05-55e847b831ad","Type":"ContainerDied","Data":"c4680eda7e834ba571c363cddcfcb087d01a7b9700c480d012946b2859fdb050"}
Jul 28 15:34:10.257505 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:34:10.257332    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22nh9_ebf75c0d-1597-4a1c-b668-dc2a041eb35e/konnectivity-agent/0.log" limit=1024
Jul 28 15:34:10.291168 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:34:10.291141    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-85ncq"
Jul 28 15:34:10.291359 ip-10-0-131-121 kubenswrapper[2568]: E0728 15:34:10.291259    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-85ncq" podUID="644d231e-7d3c-4aef-a201-87266e83d526"
Jul 28 15:34:10.344333 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:34:10.344280    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22nh9_ebf75c0d-1597-4a1c-b668-dc2a041eb35e/konnectivity-agent/0.log" limit=1024
Jul 28 15:34:10.448542 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:34:10.448467    2568 generic.go:356] "Generic (PLEG): container finished" podID="dcab1041-8d3b-42e1-af05-55e847b831ad" containerID="32c6e542722cd0099d5e4430cd154973cd4d5e42f742f88af7003ffc626de644" exitCode=0
Jul 28 15:34:10.448542 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:34:10.448521    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-ptrdx" event={"ID":"dcab1041-8d3b-42e1-af05-55e847b831ad","Type":"ContainerDied","Data":"32c6e542722cd0099d5e4430cd154973cd4d5e42f742f88af7003ffc626de644"}
Jul 28 15:34:11.034122 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:34:11.034072    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-22nh9"
Jul 28 15:34:11.034306 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:34:11.034264    2568 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness"
Jul 28 15:34:11.035240 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:34:11.035197    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-22nh9"
Jul 28 15:34:11.114529 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:34:11.114491    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22nh9_ebf75c0d-1597-4a1c-b668-dc2a041eb35e/konnectivity-agent/0.log" limit=1024
Jul 28 15:34:11.211573 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:34:11.211533    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22nh9_ebf75c0d-1597-4a1c-b668-dc2a041eb35e/konnectivity-agent/0.log" limit=1024
Jul 28 15:34:11.289925 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:34:11.289833    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tr647"
Jul 28 15:34:11.289925 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:34:11.289868    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ckpr5"
Jul 28 15:34:11.290143 ip-10-0-131-121 kubenswrapper[2568]: E0728 15:34:11.289959    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tr647" podUID="3ba72ab7-7ab4-4e7f-ace4-8a6b7bbff25c"
Jul 28 15:34:11.290143 ip-10-0-131-121 kubenswrapper[2568]: E0728 15:34:11.290099    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ckpr5" podUID="6100b515-cfe9-47ef-b672-d698ae677bd0"
Jul 28 15:34:11.313790 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:34:11.313734    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22nh9_ebf75c0d-1597-4a1c-b668-dc2a041eb35e/konnectivity-agent/0.log" limit=1024
Jul 28 15:34:12.289501 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:34:12.289470    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-85ncq"
Jul 28 15:34:12.290346 ip-10-0-131-121 kubenswrapper[2568]: E0728 15:34:12.289609    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-85ncq" podUID="644d231e-7d3c-4aef-a201-87266e83d526"
Jul 28 15:34:13.192438 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:34:13.192403    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22nh9_ebf75c0d-1597-4a1c-b668-dc2a041eb35e/konnectivity-agent/0.log" limit=1024
Jul 28 15:34:13.289638 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:34:13.289606    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ckpr5"
Jul 28 15:34:13.290097 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:34:13.289614    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tr647"
Jul 28 15:34:13.290097 ip-10-0-131-121 kubenswrapper[2568]: E0728 15:34:13.289747    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ckpr5" podUID="6100b515-cfe9-47ef-b672-d698ae677bd0"
Jul 28 15:34:13.290097 ip-10-0-131-121 kubenswrapper[2568]: E0728 15:34:13.289792    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tr647" podUID="3ba72ab7-7ab4-4e7f-ace4-8a6b7bbff25c"
Jul 28 15:34:13.291084 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:34:13.291051    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22nh9_ebf75c0d-1597-4a1c-b668-dc2a041eb35e/konnectivity-agent/0.log" limit=1024
Jul 28 15:34:13.396627 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:34:13.396547    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22nh9_ebf75c0d-1597-4a1c-b668-dc2a041eb35e/konnectivity-agent/0.log" limit=1024
Jul 28 15:34:13.637637 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:34:13.637594    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22nh9_ebf75c0d-1597-4a1c-b668-dc2a041eb35e/konnectivity-agent/0.log" limit=1024
Jul 28 15:34:14.030352 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:34:14.030317    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3ba72ab7-7ab4-4e7f-ace4-8a6b7bbff25c-original-pull-secret\") pod \"global-pull-secret-syncer-tr647\" (UID: \"3ba72ab7-7ab4-4e7f-ace4-8a6b7bbff25c\") " pod="kube-system/global-pull-secret-syncer-tr647"
Jul 28 15:34:14.030547 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:34:14.030378    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6100b515-cfe9-47ef-b672-d698ae677bd0-metrics-certs\") pod \"network-metrics-daemon-ckpr5\" (UID: \"6100b515-cfe9-47ef-b672-d698ae677bd0\") " pod="openshift-multus/network-metrics-daemon-ckpr5"
Jul 28 15:34:14.030547 ip-10-0-131-121 kubenswrapper[2568]: E0728 15:34:14.030473    2568 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 28 15:34:14.030547 ip-10-0-131-121 kubenswrapper[2568]: E0728 15:34:14.030512    2568 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 15:34:14.030707 ip-10-0-131-121 kubenswrapper[2568]: E0728 15:34:14.030557    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3ba72ab7-7ab4-4e7f-ace4-8a6b7bbff25c-original-pull-secret podName:3ba72ab7-7ab4-4e7f-ace4-8a6b7bbff25c nodeName:}" failed. No retries permitted until 2026-07-28 15:34:46.030531365 +0000 UTC m=+66.215726559 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/3ba72ab7-7ab4-4e7f-ace4-8a6b7bbff25c-original-pull-secret") pod "global-pull-secret-syncer-tr647" (UID: "3ba72ab7-7ab4-4e7f-ace4-8a6b7bbff25c") : object "kube-system"/"original-pull-secret" not registered
Jul 28 15:34:14.030707 ip-10-0-131-121 kubenswrapper[2568]: E0728 15:34:14.030580    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6100b515-cfe9-47ef-b672-d698ae677bd0-metrics-certs podName:6100b515-cfe9-47ef-b672-d698ae677bd0 nodeName:}" failed. No retries permitted until 2026-07-28 15:34:46.030569632 +0000 UTC m=+66.215764780 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6100b515-cfe9-47ef-b672-d698ae677bd0-metrics-certs") pod "network-metrics-daemon-ckpr5" (UID: "6100b515-cfe9-47ef-b672-d698ae677bd0") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 15:34:14.131328 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:34:14.131289    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qks42\" (UniqueName: \"kubernetes.io/projected/644d231e-7d3c-4aef-a201-87266e83d526-kube-api-access-qks42\") pod \"network-check-target-85ncq\" (UID: \"644d231e-7d3c-4aef-a201-87266e83d526\") " pod="openshift-network-diagnostics/network-check-target-85ncq"
Jul 28 15:34:14.131488 ip-10-0-131-121 kubenswrapper[2568]: E0728 15:34:14.131448    2568 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 28 15:34:14.131488 ip-10-0-131-121 kubenswrapper[2568]: E0728 15:34:14.131470    2568 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 28 15:34:14.131488 ip-10-0-131-121 kubenswrapper[2568]: E0728 15:34:14.131480    2568 projected.go:196] Error preparing data for projected volume kube-api-access-qks42 for pod openshift-network-diagnostics/network-check-target-85ncq: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 15:34:14.131585 ip-10-0-131-121 kubenswrapper[2568]: E0728 15:34:14.131538    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/644d231e-7d3c-4aef-a201-87266e83d526-kube-api-access-qks42 podName:644d231e-7d3c-4aef-a201-87266e83d526 nodeName:}" failed. No retries permitted until 2026-07-28 15:34:46.13152011 +0000 UTC m=+66.316715266 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-qks42" (UniqueName: "kubernetes.io/projected/644d231e-7d3c-4aef-a201-87266e83d526-kube-api-access-qks42") pod "network-check-target-85ncq" (UID: "644d231e-7d3c-4aef-a201-87266e83d526") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 15:34:14.289392 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:34:14.289312    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-85ncq"
Jul 28 15:34:14.289557 ip-10-0-131-121 kubenswrapper[2568]: E0728 15:34:14.289448    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-85ncq" podUID="644d231e-7d3c-4aef-a201-87266e83d526"
Jul 28 15:34:14.601182 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:34:14.601102    2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-121.ec2.internal" event="NodeReady"
Jul 28 15:34:14.601660 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:34:14.601242    2568 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Jul 28 15:34:14.647579 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:34:14.647536    2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-2m6ls"]
Jul 28 15:34:14.663545 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:34:14.663510    2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-8gpg6"]
Jul 28 15:34:14.663745 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:34:14.663630    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-2m6ls"
Jul 28 15:34:14.666379 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:34:14.666349    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Jul 28 15:34:14.666514 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:34:14.666419    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-sv4z4\""
Jul 28 15:34:14.666558 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:34:14.666353    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Jul 28 15:34:14.693678 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:34:14.693644    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-2m6ls"]
Jul 28 15:34:14.693856 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:34:14.693706    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-8gpg6"]
Jul 28 15:34:14.693856 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:34:14.693812    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-8gpg6"
Jul 28 15:34:14.696688 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:34:14.696667    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Jul 28 15:34:14.696944 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:34:14.696930    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Jul 28 15:34:14.696944 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:34:14.696940    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Jul 28 15:34:14.697095 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:34:14.696940    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-qlvzt\""
Jul 28 15:34:14.836694 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:34:14.836651    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/11c0aa38-e5f8-4e7d-9003-755ac2ba3a6c-config-volume\") pod \"dns-default-2m6ls\" (UID: \"11c0aa38-e5f8-4e7d-9003-755ac2ba3a6c\") " pod="openshift-dns/dns-default-2m6ls"
Jul 28 15:34:14.836694 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:34:14.836693    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/4ef2e055-ee39-422e-9cd0-5c7e2e29c7c1-cert\") pod \"ingress-canary-8gpg6\" (UID: \"4ef2e055-ee39-422e-9cd0-5c7e2e29c7c1\") " pod="openshift-ingress-canary/ingress-canary-8gpg6"
Jul 28 15:34:14.836913 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:34:14.836807    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/11c0aa38-e5f8-4e7d-9003-755ac2ba3a6c-metrics-tls\") pod \"dns-default-2m6ls\" (UID: \"11c0aa38-e5f8-4e7d-9003-755ac2ba3a6c\") " pod="openshift-dns/dns-default-2m6ls"
Jul 28 15:34:14.836913 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:34:14.836832    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sxf8c\" (UniqueName: \"kubernetes.io/projected/4ef2e055-ee39-422e-9cd0-5c7e2e29c7c1-kube-api-access-sxf8c\") pod \"ingress-canary-8gpg6\" (UID: \"4ef2e055-ee39-422e-9cd0-5c7e2e29c7c1\") " pod="openshift-ingress-canary/ingress-canary-8gpg6"
Jul 28 15:34:14.836913 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:34:14.836858    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/11c0aa38-e5f8-4e7d-9003-755ac2ba3a6c-tmp-dir\") pod \"dns-default-2m6ls\" (UID: \"11c0aa38-e5f8-4e7d-9003-755ac2ba3a6c\") " pod="openshift-dns/dns-default-2m6ls"
Jul 28 15:34:14.836913 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:34:14.836905    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v4nvx\" (UniqueName: \"kubernetes.io/projected/11c0aa38-e5f8-4e7d-9003-755ac2ba3a6c-kube-api-access-v4nvx\") pod \"dns-default-2m6ls\" (UID: \"11c0aa38-e5f8-4e7d-9003-755ac2ba3a6c\") " pod="openshift-dns/dns-default-2m6ls"
Jul 28 15:34:14.938263 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:34:14.938180    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/11c0aa38-e5f8-4e7d-9003-755ac2ba3a6c-tmp-dir\") pod \"dns-default-2m6ls\" (UID: \"11c0aa38-e5f8-4e7d-9003-755ac2ba3a6c\") " pod="openshift-dns/dns-default-2m6ls"
Jul 28 15:34:14.938263 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:34:14.938218    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v4nvx\" (UniqueName: \"kubernetes.io/projected/11c0aa38-e5f8-4e7d-9003-755ac2ba3a6c-kube-api-access-v4nvx\") pod \"dns-default-2m6ls\" (UID: \"11c0aa38-e5f8-4e7d-9003-755ac2ba3a6c\") " pod="openshift-dns/dns-default-2m6ls"
Jul 28 15:34:14.938263 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:34:14.938247    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/11c0aa38-e5f8-4e7d-9003-755ac2ba3a6c-config-volume\") pod \"dns-default-2m6ls\" (UID: \"11c0aa38-e5f8-4e7d-9003-755ac2ba3a6c\") " pod="openshift-dns/dns-default-2m6ls"
Jul 28 15:34:14.938263 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:34:14.938263    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/4ef2e055-ee39-422e-9cd0-5c7e2e29c7c1-cert\") pod \"ingress-canary-8gpg6\" (UID: \"4ef2e055-ee39-422e-9cd0-5c7e2e29c7c1\") " pod="openshift-ingress-canary/ingress-canary-8gpg6"
Jul 28 15:34:14.938572 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:34:14.938369    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/11c0aa38-e5f8-4e7d-9003-755ac2ba3a6c-metrics-tls\") pod \"dns-default-2m6ls\" (UID: \"11c0aa38-e5f8-4e7d-9003-755ac2ba3a6c\") " pod="openshift-dns/dns-default-2m6ls"
Jul 28 15:34:14.938572 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:34:14.938410    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sxf8c\" (UniqueName: \"kubernetes.io/projected/4ef2e055-ee39-422e-9cd0-5c7e2e29c7c1-kube-api-access-sxf8c\") pod \"ingress-canary-8gpg6\" (UID: \"4ef2e055-ee39-422e-9cd0-5c7e2e29c7c1\") " pod="openshift-ingress-canary/ingress-canary-8gpg6"
Jul 28 15:34:14.938572 ip-10-0-131-121 kubenswrapper[2568]: E0728 15:34:14.938467    2568 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 28 15:34:14.938572 ip-10-0-131-121 kubenswrapper[2568]: E0728 15:34:14.938545    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/11c0aa38-e5f8-4e7d-9003-755ac2ba3a6c-metrics-tls podName:11c0aa38-e5f8-4e7d-9003-755ac2ba3a6c nodeName:}" failed. No retries permitted until 2026-07-28 15:34:15.4385242 +0000 UTC m=+35.623719368 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/11c0aa38-e5f8-4e7d-9003-755ac2ba3a6c-metrics-tls") pod "dns-default-2m6ls" (UID: "11c0aa38-e5f8-4e7d-9003-755ac2ba3a6c") : secret "dns-default-metrics-tls" not found
Jul 28 15:34:14.938756 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:34:14.938581    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/11c0aa38-e5f8-4e7d-9003-755ac2ba3a6c-tmp-dir\") pod \"dns-default-2m6ls\" (UID: \"11c0aa38-e5f8-4e7d-9003-755ac2ba3a6c\") " pod="openshift-dns/dns-default-2m6ls"
Jul 28 15:34:14.938916 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:34:14.938893    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/11c0aa38-e5f8-4e7d-9003-755ac2ba3a6c-config-volume\") pod \"dns-default-2m6ls\" (UID: \"11c0aa38-e5f8-4e7d-9003-755ac2ba3a6c\") " pod="openshift-dns/dns-default-2m6ls"
Jul 28 15:34:14.942734 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:34:14.942706    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/4ef2e055-ee39-422e-9cd0-5c7e2e29c7c1-cert\") pod \"ingress-canary-8gpg6\" (UID: \"4ef2e055-ee39-422e-9cd0-5c7e2e29c7c1\") " pod="openshift-ingress-canary/ingress-canary-8gpg6"
Jul 28 15:34:14.947970 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:34:14.947947    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v4nvx\" (UniqueName: \"kubernetes.io/projected/11c0aa38-e5f8-4e7d-9003-755ac2ba3a6c-kube-api-access-v4nvx\") pod \"dns-default-2m6ls\" (UID: \"11c0aa38-e5f8-4e7d-9003-755ac2ba3a6c\") " pod="openshift-dns/dns-default-2m6ls"
Jul 28 15:34:14.948133 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:34:14.948013    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sxf8c\" (UniqueName: \"kubernetes.io/projected/4ef2e055-ee39-422e-9cd0-5c7e2e29c7c1-kube-api-access-sxf8c\") pod \"ingress-canary-8gpg6\" (UID: \"4ef2e055-ee39-422e-9cd0-5c7e2e29c7c1\") " pod="openshift-ingress-canary/ingress-canary-8gpg6"
Jul 28 15:34:15.002503 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:34:15.002468    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-8gpg6"
Jul 28 15:34:15.129284 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:34:15.129056    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22nh9_ebf75c0d-1597-4a1c-b668-dc2a041eb35e/konnectivity-agent/0.log" limit=1024
Jul 28 15:34:15.185045 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:34:15.185010    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22nh9_ebf75c0d-1597-4a1c-b668-dc2a041eb35e/konnectivity-agent/0.log" limit=1024
Jul 28 15:34:15.201069 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:34:15.201039    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-8gpg6"]
Jul 28 15:34:15.205029 ip-10-0-131-121 kubenswrapper[2568]: W0728 15:34:15.205000    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4ef2e055_ee39_422e_9cd0_5c7e2e29c7c1.slice/crio-52a82f0aa2a7081183c2503fcff653fe99d0341755fb4bfa6d8c6088c2db3899 WatchSource:0}: Error finding container 52a82f0aa2a7081183c2503fcff653fe99d0341755fb4bfa6d8c6088c2db3899: Status 404 returned error can't find the container with id 52a82f0aa2a7081183c2503fcff653fe99d0341755fb4bfa6d8c6088c2db3899
Jul 28 15:34:15.289282 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:34:15.289259    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ckpr5"
Jul 28 15:34:15.289387 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:34:15.289287    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tr647"
Jul 28 15:34:15.292239 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:34:15.292221    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-xmhl6\""
Jul 28 15:34:15.292336 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:34:15.292250    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 28 15:34:15.292336 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:34:15.292250    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 28 15:34:15.442618 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:34:15.442584    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/11c0aa38-e5f8-4e7d-9003-755ac2ba3a6c-metrics-tls\") pod \"dns-default-2m6ls\" (UID: \"11c0aa38-e5f8-4e7d-9003-755ac2ba3a6c\") " pod="openshift-dns/dns-default-2m6ls"
Jul 28 15:34:15.442765 ip-10-0-131-121 kubenswrapper[2568]: E0728 15:34:15.442748    2568 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 28 15:34:15.442846 ip-10-0-131-121 kubenswrapper[2568]: E0728 15:34:15.442826    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/11c0aa38-e5f8-4e7d-9003-755ac2ba3a6c-metrics-tls podName:11c0aa38-e5f8-4e7d-9003-755ac2ba3a6c nodeName:}" failed. No retries permitted until 2026-07-28 15:34:16.44280829 +0000 UTC m=+36.628003437 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/11c0aa38-e5f8-4e7d-9003-755ac2ba3a6c-metrics-tls") pod "dns-default-2m6ls" (UID: "11c0aa38-e5f8-4e7d-9003-755ac2ba3a6c") : secret "dns-default-metrics-tls" not found
Jul 28 15:34:15.463908 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:34:15.463829    2568 generic.go:356] "Generic (PLEG): container finished" podID="dcab1041-8d3b-42e1-af05-55e847b831ad" containerID="f222f4eef2be60b49766c9ce8215097408f290fa344da3182c7c9f6cb5397612" exitCode=0
Jul 28 15:34:15.464044 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:34:15.463903    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-ptrdx" event={"ID":"dcab1041-8d3b-42e1-af05-55e847b831ad","Type":"ContainerDied","Data":"f222f4eef2be60b49766c9ce8215097408f290fa344da3182c7c9f6cb5397612"}
Jul 28 15:34:15.465047 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:34:15.465022    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-8gpg6" event={"ID":"4ef2e055-ee39-422e-9cd0-5c7e2e29c7c1","Type":"ContainerStarted","Data":"52a82f0aa2a7081183c2503fcff653fe99d0341755fb4bfa6d8c6088c2db3899"}
Jul 28 15:34:16.289743 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:34:16.289707    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-85ncq"
Jul 28 15:34:16.293830 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:34:16.293680    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 28 15:34:16.293830 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:34:16.293703    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 28 15:34:16.293830 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:34:16.293682    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-ptv7b\""
Jul 28 15:34:16.451702 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:34:16.451617    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/11c0aa38-e5f8-4e7d-9003-755ac2ba3a6c-metrics-tls\") pod \"dns-default-2m6ls\" (UID: \"11c0aa38-e5f8-4e7d-9003-755ac2ba3a6c\") " pod="openshift-dns/dns-default-2m6ls"
Jul 28 15:34:16.451875 ip-10-0-131-121 kubenswrapper[2568]: E0728 15:34:16.451794    2568 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 28 15:34:16.451875 ip-10-0-131-121 kubenswrapper[2568]: E0728 15:34:16.451874    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/11c0aa38-e5f8-4e7d-9003-755ac2ba3a6c-metrics-tls podName:11c0aa38-e5f8-4e7d-9003-755ac2ba3a6c nodeName:}" failed. No retries permitted until 2026-07-28 15:34:18.451856208 +0000 UTC m=+38.637051355 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/11c0aa38-e5f8-4e7d-9003-755ac2ba3a6c-metrics-tls") pod "dns-default-2m6ls" (UID: "11c0aa38-e5f8-4e7d-9003-755ac2ba3a6c") : secret "dns-default-metrics-tls" not found
Jul 28 15:34:16.470799 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:34:16.470735    2568 generic.go:356] "Generic (PLEG): container finished" podID="dcab1041-8d3b-42e1-af05-55e847b831ad" containerID="70be1a5b01f0402355f188939e095579bb4ab760797d15a7f6ddc9c1235561b4" exitCode=0
Jul 28 15:34:16.470938 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:34:16.470812    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-ptrdx" event={"ID":"dcab1041-8d3b-42e1-af05-55e847b831ad","Type":"ContainerDied","Data":"70be1a5b01f0402355f188939e095579bb4ab760797d15a7f6ddc9c1235561b4"}
Jul 28 15:34:17.476418 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:34:17.476177    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-ptrdx" event={"ID":"dcab1041-8d3b-42e1-af05-55e847b831ad","Type":"ContainerStarted","Data":"831d503c0f3067690a5901cb1934df375289439d80a147b50991648a2241f5a9"}
Jul 28 15:34:17.477747 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:34:17.477722    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-8gpg6" event={"ID":"4ef2e055-ee39-422e-9cd0-5c7e2e29c7c1","Type":"ContainerStarted","Data":"cb9bd890298f8f41352bb2122a3da159c12c65b9cf5985735dd0a5dd3d9ea7a3"}
Jul 28 15:34:17.502604 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:34:17.502548    2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-ptrdx" podStartSLOduration=7.9731717920000005 podStartE2EDuration="37.502518581s" totalImagesPullingTime="29.529346789s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 15:33:40 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 15:34:17.500975291 +0000 UTC m=+37.686170495" watchObservedRunningTime="2026-07-28 15:34:17.502518581 +0000 UTC m=+37.687713752"
Jul 28 15:34:17.521116 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:34:17.521050    2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-8gpg6" podStartSLOduration=1.382315194 podStartE2EDuration="3.52102996s" totalImagesPullingTime="2.138714766s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 15:34:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 15:34:17.520856539 +0000 UTC m=+37.706051711" watchObservedRunningTime="2026-07-28 15:34:17.52102996 +0000 UTC m=+37.706225130"
Jul 28 15:34:17.630407 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:34:17.630322    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22nh9_ebf75c0d-1597-4a1c-b668-dc2a041eb35e/konnectivity-agent/0.log" limit=1024
Jul 28 15:34:17.740316 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:34:17.740284    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22nh9_ebf75c0d-1597-4a1c-b668-dc2a041eb35e/konnectivity-agent/0.log" limit=1024
Jul 28 15:34:17.875925 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:34:17.875895    2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-rsd4q"]
Jul 28 15:34:17.877936 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:34:17.877915    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-rsd4q"
Jul 28 15:34:17.880357 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:34:17.880317    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Jul 28 15:34:17.880357 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:34:17.880326    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Jul 28 15:34:17.880357 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:34:17.880340    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Jul 28 15:34:17.880607 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:34:17.880316    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Jul 28 15:34:17.880666 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:34:17.880653    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-kgdjb\""
Jul 28 15:34:17.891115 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:34:17.891088    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-rsd4q"]
Jul 28 15:34:17.908750 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:34:17.908718    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22nh9_ebf75c0d-1597-4a1c-b668-dc2a041eb35e/konnectivity-agent/0.log" limit=1024
Jul 28 15:34:17.972270 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:34:17.972236    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22nh9_ebf75c0d-1597-4a1c-b668-dc2a041eb35e/konnectivity-agent/0.log" limit=1024
Jul 28 15:34:18.064217 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:34:18.064170    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-465vr\" (UniqueName: \"kubernetes.io/projected/45da22e1-42e5-4dcf-b132-105b7b343538-kube-api-access-465vr\") pod \"insights-runtime-extractor-rsd4q\" (UID: \"45da22e1-42e5-4dcf-b132-105b7b343538\") " pod="openshift-insights/insights-runtime-extractor-rsd4q"
Jul 28 15:34:18.064217 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:34:18.064214    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/45da22e1-42e5-4dcf-b132-105b7b343538-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-rsd4q\" (UID: \"45da22e1-42e5-4dcf-b132-105b7b343538\") " pod="openshift-insights/insights-runtime-extractor-rsd4q"
Jul 28 15:34:18.064411 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:34:18.064315    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/45da22e1-42e5-4dcf-b132-105b7b343538-crio-socket\") pod \"insights-runtime-extractor-rsd4q\" (UID: \"45da22e1-42e5-4dcf-b132-105b7b343538\") " pod="openshift-insights/insights-runtime-extractor-rsd4q"
Jul 28 15:34:18.064411 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:34:18.064342    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/45da22e1-42e5-4dcf-b132-105b7b343538-data-volume\") pod \"insights-runtime-extractor-rsd4q\" (UID: \"45da22e1-42e5-4dcf-b132-105b7b343538\") " pod="openshift-insights/insights-runtime-extractor-rsd4q"
Jul 28 15:34:18.064411 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:34:18.064378    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/45da22e1-42e5-4dcf-b132-105b7b343538-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-rsd4q\" (UID: \"45da22e1-42e5-4dcf-b132-105b7b343538\") " pod="openshift-insights/insights-runtime-extractor-rsd4q"
Jul 28 15:34:18.120629 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:34:18.120595    2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-6865c9967-xwgqx"]
Jul 28 15:34:18.121265 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:34:18.121217    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22nh9_ebf75c0d-1597-4a1c-b668-dc2a041eb35e/konnectivity-agent/0.log" limit=1024
Jul 28 15:34:18.122558 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:34:18.122544    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-6865c9967-xwgqx"
Jul 28 15:34:18.124993 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:34:18.124962    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\""
Jul 28 15:34:18.125104 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:34:18.125000    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-6kbp5\""
Jul 28 15:34:18.125104 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:34:18.125038    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\""
Jul 28 15:34:18.135062 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:34:18.135004    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-6865c9967-xwgqx"]
Jul 28 15:34:18.165085 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:34:18.165043    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/45da22e1-42e5-4dcf-b132-105b7b343538-crio-socket\") pod \"insights-runtime-extractor-rsd4q\" (UID: \"45da22e1-42e5-4dcf-b132-105b7b343538\") " pod="openshift-insights/insights-runtime-extractor-rsd4q"
Jul 28 15:34:18.165085 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:34:18.165079    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/45da22e1-42e5-4dcf-b132-105b7b343538-data-volume\") pod \"insights-runtime-extractor-rsd4q\" (UID: \"45da22e1-42e5-4dcf-b132-105b7b343538\") " pod="openshift-insights/insights-runtime-extractor-rsd4q"
Jul 28 15:34:18.165346 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:34:18.165116    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/45da22e1-42e5-4dcf-b132-105b7b343538-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-rsd4q\" (UID: \"45da22e1-42e5-4dcf-b132-105b7b343538\") " pod="openshift-insights/insights-runtime-extractor-rsd4q"
Jul 28 15:34:18.165346 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:34:18.165154    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-465vr\" (UniqueName: \"kubernetes.io/projected/45da22e1-42e5-4dcf-b132-105b7b343538-kube-api-access-465vr\") pod \"insights-runtime-extractor-rsd4q\" (UID: \"45da22e1-42e5-4dcf-b132-105b7b343538\") " pod="openshift-insights/insights-runtime-extractor-rsd4q"
Jul 28 15:34:18.165346 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:34:18.165175    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/45da22e1-42e5-4dcf-b132-105b7b343538-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-rsd4q\" (UID: \"45da22e1-42e5-4dcf-b132-105b7b343538\") " pod="openshift-insights/insights-runtime-extractor-rsd4q"
Jul 28 15:34:18.165346 ip-10-0-131-121 kubenswrapper[2568]: E0728 15:34:18.165272    2568 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 28 15:34:18.165346 ip-10-0-131-121 kubenswrapper[2568]: E0728 15:34:18.165289    2568 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 28 15:34:18.165346 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:34:18.165318    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/45da22e1-42e5-4dcf-b132-105b7b343538-crio-socket\") pod \"insights-runtime-extractor-rsd4q\" (UID: \"45da22e1-42e5-4dcf-b132-105b7b343538\") " pod="openshift-insights/insights-runtime-extractor-rsd4q"
Jul 28 15:34:18.165346 ip-10-0-131-121 kubenswrapper[2568]: E0728 15:34:18.165340    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/45da22e1-42e5-4dcf-b132-105b7b343538-insights-runtime-extractor-tls podName:45da22e1-42e5-4dcf-b132-105b7b343538 nodeName:}" failed. No retries permitted until 2026-07-28 15:34:18.665326408 +0000 UTC m=+38.850521559 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/45da22e1-42e5-4dcf-b132-105b7b343538-insights-runtime-extractor-tls") pod "insights-runtime-extractor-rsd4q" (UID: "45da22e1-42e5-4dcf-b132-105b7b343538") : secret "insights-runtime-extractor-tls" not found
Jul 28 15:34:18.165663 ip-10-0-131-121 kubenswrapper[2568]: E0728 15:34:18.165363    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/45da22e1-42e5-4dcf-b132-105b7b343538-kube-rbac-proxy-cm podName:45da22e1-42e5-4dcf-b132-105b7b343538 nodeName:}" failed. No retries permitted until 2026-07-28 15:34:18.665355515 +0000 UTC m=+38.850550701 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/45da22e1-42e5-4dcf-b132-105b7b343538-kube-rbac-proxy-cm") pod "insights-runtime-extractor-rsd4q" (UID: "45da22e1-42e5-4dcf-b132-105b7b343538") : configmap "kube-rbac-proxy" not found
Jul 28 15:34:18.165663 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:34:18.165483    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/45da22e1-42e5-4dcf-b132-105b7b343538-data-volume\") pod \"insights-runtime-extractor-rsd4q\" (UID: \"45da22e1-42e5-4dcf-b132-105b7b343538\") " pod="openshift-insights/insights-runtime-extractor-rsd4q"
Jul 28 15:34:18.176025 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:34:18.176003    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-465vr\" (UniqueName: \"kubernetes.io/projected/45da22e1-42e5-4dcf-b132-105b7b343538-kube-api-access-465vr\") pod \"insights-runtime-extractor-rsd4q\" (UID: \"45da22e1-42e5-4dcf-b132-105b7b343538\") " pod="openshift-insights/insights-runtime-extractor-rsd4q"
Jul 28 15:34:18.191148 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:34:18.191118    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22nh9_ebf75c0d-1597-4a1c-b668-dc2a041eb35e/konnectivity-agent/0.log" limit=1024
Jul 28 15:34:18.265540 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:34:18.265505    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nvgvs\" (UniqueName: \"kubernetes.io/projected/b940bfbc-30c7-4eff-9501-484fa354d3b5-kube-api-access-nvgvs\") pod \"migrator-6865c9967-xwgqx\" (UID: \"b940bfbc-30c7-4eff-9501-484fa354d3b5\") " pod="openshift-kube-storage-version-migrator/migrator-6865c9967-xwgqx"
Jul 28 15:34:18.292767 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:34:18.292737    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22nh9_ebf75c0d-1597-4a1c-b668-dc2a041eb35e/konnectivity-agent/0.log" limit=1024
Jul 28 15:34:18.366224 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:34:18.366184    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nvgvs\" (UniqueName: \"kubernetes.io/projected/b940bfbc-30c7-4eff-9501-484fa354d3b5-kube-api-access-nvgvs\") pod \"migrator-6865c9967-xwgqx\" (UID: \"b940bfbc-30c7-4eff-9501-484fa354d3b5\") " pod="openshift-kube-storage-version-migrator/migrator-6865c9967-xwgqx"
Jul 28 15:34:18.376286 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:34:18.376255    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nvgvs\" (UniqueName: \"kubernetes.io/projected/b940bfbc-30c7-4eff-9501-484fa354d3b5-kube-api-access-nvgvs\") pod \"migrator-6865c9967-xwgqx\" (UID: \"b940bfbc-30c7-4eff-9501-484fa354d3b5\") " pod="openshift-kube-storage-version-migrator/migrator-6865c9967-xwgqx"
Jul 28 15:34:18.431495 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:34:18.431404    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-6865c9967-xwgqx"
Jul 28 15:34:18.467293 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:34:18.467256    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/11c0aa38-e5f8-4e7d-9003-755ac2ba3a6c-metrics-tls\") pod \"dns-default-2m6ls\" (UID: \"11c0aa38-e5f8-4e7d-9003-755ac2ba3a6c\") " pod="openshift-dns/dns-default-2m6ls"
Jul 28 15:34:18.467471 ip-10-0-131-121 kubenswrapper[2568]: E0728 15:34:18.467381    2568 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 28 15:34:18.467471 ip-10-0-131-121 kubenswrapper[2568]: E0728 15:34:18.467433    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/11c0aa38-e5f8-4e7d-9003-755ac2ba3a6c-metrics-tls podName:11c0aa38-e5f8-4e7d-9003-755ac2ba3a6c nodeName:}" failed. No retries permitted until 2026-07-28 15:34:22.467420136 +0000 UTC m=+42.652615284 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/11c0aa38-e5f8-4e7d-9003-755ac2ba3a6c-metrics-tls") pod "dns-default-2m6ls" (UID: "11c0aa38-e5f8-4e7d-9003-755ac2ba3a6c") : secret "dns-default-metrics-tls" not found
Jul 28 15:34:18.561013 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:34:18.560983    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-6865c9967-xwgqx"]
Jul 28 15:34:18.564887 ip-10-0-131-121 kubenswrapper[2568]: W0728 15:34:18.564710    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb940bfbc_30c7_4eff_9501_484fa354d3b5.slice/crio-4fad7b3f043b8be3094bf3fd17b3874350a107db0d7c24c08a79dfc095d79eca WatchSource:0}: Error finding container 4fad7b3f043b8be3094bf3fd17b3874350a107db0d7c24c08a79dfc095d79eca: Status 404 returned error can't find the container with id 4fad7b3f043b8be3094bf3fd17b3874350a107db0d7c24c08a79dfc095d79eca
Jul 28 15:34:18.618358 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:34:18.618326    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22nh9_ebf75c0d-1597-4a1c-b668-dc2a041eb35e/konnectivity-agent/0.log" limit=1024
Jul 28 15:34:18.669313 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:34:18.669277    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/45da22e1-42e5-4dcf-b132-105b7b343538-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-rsd4q\" (UID: \"45da22e1-42e5-4dcf-b132-105b7b343538\") " pod="openshift-insights/insights-runtime-extractor-rsd4q"
Jul 28 15:34:18.669446 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:34:18.669344    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/45da22e1-42e5-4dcf-b132-105b7b343538-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-rsd4q\" (UID: \"45da22e1-42e5-4dcf-b132-105b7b343538\") " pod="openshift-insights/insights-runtime-extractor-rsd4q"
Jul 28 15:34:18.669446 ip-10-0-131-121 kubenswrapper[2568]: E0728 15:34:18.669391    2568 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 28 15:34:18.669446 ip-10-0-131-121 kubenswrapper[2568]: E0728 15:34:18.669423    2568 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 28 15:34:18.669559 ip-10-0-131-121 kubenswrapper[2568]: E0728 15:34:18.669461    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/45da22e1-42e5-4dcf-b132-105b7b343538-kube-rbac-proxy-cm podName:45da22e1-42e5-4dcf-b132-105b7b343538 nodeName:}" failed. No retries permitted until 2026-07-28 15:34:19.669439648 +0000 UTC m=+39.854634815 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/45da22e1-42e5-4dcf-b132-105b7b343538-kube-rbac-proxy-cm") pod "insights-runtime-extractor-rsd4q" (UID: "45da22e1-42e5-4dcf-b132-105b7b343538") : configmap "kube-rbac-proxy" not found
Jul 28 15:34:18.669559 ip-10-0-131-121 kubenswrapper[2568]: E0728 15:34:18.669482    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/45da22e1-42e5-4dcf-b132-105b7b343538-insights-runtime-extractor-tls podName:45da22e1-42e5-4dcf-b132-105b7b343538 nodeName:}" failed. No retries permitted until 2026-07-28 15:34:19.669472332 +0000 UTC m=+39.854667478 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/45da22e1-42e5-4dcf-b132-105b7b343538-insights-runtime-extractor-tls") pod "insights-runtime-extractor-rsd4q" (UID: "45da22e1-42e5-4dcf-b132-105b7b343538") : secret "insights-runtime-extractor-tls" not found
Jul 28 15:34:18.886285 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:34:18.886232    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22nh9_ebf75c0d-1597-4a1c-b668-dc2a041eb35e/konnectivity-agent/0.log" limit=1024
Jul 28 15:34:18.988238 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:34:18.988204    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22nh9_ebf75c0d-1597-4a1c-b668-dc2a041eb35e/konnectivity-agent/0.log" limit=1024
Jul 28 15:34:19.487005 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:34:19.486969    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-6865c9967-xwgqx" event={"ID":"b940bfbc-30c7-4eff-9501-484fa354d3b5","Type":"ContainerStarted","Data":"4fad7b3f043b8be3094bf3fd17b3874350a107db0d7c24c08a79dfc095d79eca"}
Jul 28 15:34:19.653334 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:34:19.653306    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-whdfr_fab92b7b-ed62-4783-86be-ed4bd56a3128/dns-node-resolver/0.log"
Jul 28 15:34:19.677416 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:34:19.677393    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/45da22e1-42e5-4dcf-b132-105b7b343538-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-rsd4q\" (UID: \"45da22e1-42e5-4dcf-b132-105b7b343538\") " pod="openshift-insights/insights-runtime-extractor-rsd4q"
Jul 28 15:34:19.677514 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:34:19.677454    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/45da22e1-42e5-4dcf-b132-105b7b343538-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-rsd4q\" (UID: \"45da22e1-42e5-4dcf-b132-105b7b343538\") " pod="openshift-insights/insights-runtime-extractor-rsd4q"
Jul 28 15:34:19.677571 ip-10-0-131-121 kubenswrapper[2568]: E0728 15:34:19.677542    2568 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 28 15:34:19.677624 ip-10-0-131-121 kubenswrapper[2568]: E0728 15:34:19.677607    2568 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 28 15:34:19.677673 ip-10-0-131-121 kubenswrapper[2568]: E0728 15:34:19.677632    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/45da22e1-42e5-4dcf-b132-105b7b343538-kube-rbac-proxy-cm podName:45da22e1-42e5-4dcf-b132-105b7b343538 nodeName:}" failed. No retries permitted until 2026-07-28 15:34:21.677610437 +0000 UTC m=+41.862805587 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/45da22e1-42e5-4dcf-b132-105b7b343538-kube-rbac-proxy-cm") pod "insights-runtime-extractor-rsd4q" (UID: "45da22e1-42e5-4dcf-b132-105b7b343538") : configmap "kube-rbac-proxy" not found
Jul 28 15:34:19.677673 ip-10-0-131-121 kubenswrapper[2568]: E0728 15:34:19.677661    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/45da22e1-42e5-4dcf-b132-105b7b343538-insights-runtime-extractor-tls podName:45da22e1-42e5-4dcf-b132-105b7b343538 nodeName:}" failed. No retries permitted until 2026-07-28 15:34:21.677644806 +0000 UTC m=+41.862839968 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/45da22e1-42e5-4dcf-b132-105b7b343538-insights-runtime-extractor-tls") pod "insights-runtime-extractor-rsd4q" (UID: "45da22e1-42e5-4dcf-b132-105b7b343538") : secret "insights-runtime-extractor-tls" not found
Jul 28 15:34:19.691336 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:34:19.691296    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22nh9_ebf75c0d-1597-4a1c-b668-dc2a041eb35e/konnectivity-agent/0.log" limit=1024
Jul 28 15:34:19.795977 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:34:19.795944    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22nh9_ebf75c0d-1597-4a1c-b668-dc2a041eb35e/konnectivity-agent/0.log" limit=1024
Jul 28 15:34:20.486983 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:34:20.486946    2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-658d7b7c78-hh52q"]
Jul 28 15:34:20.488804 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:34:20.488769    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-658d7b7c78-hh52q"
Jul 28 15:34:20.490704 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:34:20.490677    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-6865c9967-xwgqx" event={"ID":"b940bfbc-30c7-4eff-9501-484fa354d3b5","Type":"ContainerStarted","Data":"0691dfca8e8071532443a64875f70625c55471195db20b978514f1c81e91ad19"}
Jul 28 15:34:20.490819 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:34:20.490720    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-6865c9967-xwgqx" event={"ID":"b940bfbc-30c7-4eff-9501-484fa354d3b5","Type":"ContainerStarted","Data":"61d9d6a48bb7a06a4465e3e4a11bc905394be762d0e457d5546385a3f93ab83b"}
Jul 28 15:34:20.491325 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:34:20.491306    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\""
Jul 28 15:34:20.491471 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:34:20.491457    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\""
Jul 28 15:34:20.492541 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:34:20.492523    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\""
Jul 28 15:34:20.492617 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:34:20.492524    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-jhxzs\""
Jul 28 15:34:20.492617 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:34:20.492562    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\""
Jul 28 15:34:20.499908 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:34:20.499886    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-658d7b7c78-hh52q"]
Jul 28 15:34:20.531430 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:34:20.531401    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22nh9_ebf75c0d-1597-4a1c-b668-dc2a041eb35e/konnectivity-agent/0.log" limit=1024
Jul 28 15:34:20.535700 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:34:20.535659    2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-6865c9967-xwgqx" podStartSLOduration=1.49451267 podStartE2EDuration="2.535615633s" totalImagesPullingTime="1.041102963s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 15:34:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 15:34:20.534682855 +0000 UTC m=+40.719878036" watchObservedRunningTime="2026-07-28 15:34:20.535615633 +0000 UTC m=+40.720810861"
Jul 28 15:34:20.583292 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:34:20.583249    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/040becb9-a46f-40e0-ad85-50cf15ddcafc-signing-key\") pod \"service-ca-658d7b7c78-hh52q\" (UID: \"040becb9-a46f-40e0-ad85-50cf15ddcafc\") " pod="openshift-service-ca/service-ca-658d7b7c78-hh52q"
Jul 28 15:34:20.583292 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:34:20.583294    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/040becb9-a46f-40e0-ad85-50cf15ddcafc-signing-cabundle\") pod \"service-ca-658d7b7c78-hh52q\" (UID: \"040becb9-a46f-40e0-ad85-50cf15ddcafc\") " pod="openshift-service-ca/service-ca-658d7b7c78-hh52q"
Jul 28 15:34:20.583439 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:34:20.583382    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7xx8s\" (UniqueName: \"kubernetes.io/projected/040becb9-a46f-40e0-ad85-50cf15ddcafc-kube-api-access-7xx8s\") pod \"service-ca-658d7b7c78-hh52q\" (UID: \"040becb9-a46f-40e0-ad85-50cf15ddcafc\") " pod="openshift-service-ca/service-ca-658d7b7c78-hh52q"
Jul 28 15:34:20.620349 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:34:20.620306    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22nh9_ebf75c0d-1597-4a1c-b668-dc2a041eb35e/konnectivity-agent/0.log" limit=1024
Jul 28 15:34:20.684809 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:34:20.684761    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/040becb9-a46f-40e0-ad85-50cf15ddcafc-signing-key\") pod \"service-ca-658d7b7c78-hh52q\" (UID: \"040becb9-a46f-40e0-ad85-50cf15ddcafc\") " pod="openshift-service-ca/service-ca-658d7b7c78-hh52q"
Jul 28 15:34:20.684809 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:34:20.684811    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/040becb9-a46f-40e0-ad85-50cf15ddcafc-signing-cabundle\") pod \"service-ca-658d7b7c78-hh52q\" (UID: \"040becb9-a46f-40e0-ad85-50cf15ddcafc\") " pod="openshift-service-ca/service-ca-658d7b7c78-hh52q"
Jul 28 15:34:20.685257 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:34:20.684857    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7xx8s\" (UniqueName: \"kubernetes.io/projected/040becb9-a46f-40e0-ad85-50cf15ddcafc-kube-api-access-7xx8s\") pod \"service-ca-658d7b7c78-hh52q\" (UID: \"040becb9-a46f-40e0-ad85-50cf15ddcafc\") " pod="openshift-service-ca/service-ca-658d7b7c78-hh52q"
Jul 28 15:34:20.685588 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:34:20.685567    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/040becb9-a46f-40e0-ad85-50cf15ddcafc-signing-cabundle\") pod \"service-ca-658d7b7c78-hh52q\" (UID: \"040becb9-a46f-40e0-ad85-50cf15ddcafc\") " pod="openshift-service-ca/service-ca-658d7b7c78-hh52q"
Jul 28 15:34:20.687974 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:34:20.687957    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/040becb9-a46f-40e0-ad85-50cf15ddcafc-signing-key\") pod \"service-ca-658d7b7c78-hh52q\" (UID: \"040becb9-a46f-40e0-ad85-50cf15ddcafc\") " pod="openshift-service-ca/service-ca-658d7b7c78-hh52q"
Jul 28 15:34:20.694679 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:34:20.694658    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7xx8s\" (UniqueName: \"kubernetes.io/projected/040becb9-a46f-40e0-ad85-50cf15ddcafc-kube-api-access-7xx8s\") pod \"service-ca-658d7b7c78-hh52q\" (UID: \"040becb9-a46f-40e0-ad85-50cf15ddcafc\") " pod="openshift-service-ca/service-ca-658d7b7c78-hh52q"
Jul 28 15:34:20.724739 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:34:20.724701    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22nh9_ebf75c0d-1597-4a1c-b668-dc2a041eb35e/konnectivity-agent/0.log" limit=1024
Jul 28 15:34:20.798366 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:34:20.798334    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-658d7b7c78-hh52q"
Jul 28 15:34:20.941725 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:34:20.941696    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-658d7b7c78-hh52q"]
Jul 28 15:34:20.944861 ip-10-0-131-121 kubenswrapper[2568]: W0728 15:34:20.944834    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod040becb9_a46f_40e0_ad85_50cf15ddcafc.slice/crio-b44278b5c235b9616c709ed44974b64ca572cd2f7816ca26003e5575073086f5 WatchSource:0}: Error finding container b44278b5c235b9616c709ed44974b64ca572cd2f7816ca26003e5575073086f5: Status 404 returned error can't find the container with id b44278b5c235b9616c709ed44974b64ca572cd2f7816ca26003e5575073086f5
Jul 28 15:34:21.053422 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:34:21.053339    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-9hznr_53a1cbf6-fc45-47f6-a8b1-5f95ec295587/node-ca/0.log"
Jul 28 15:34:21.456067 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:34:21.455991    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-8gpg6_4ef2e055-ee39-422e-9cd0-5c7e2e29c7c1/serve-healthcheck-canary/0.log"
Jul 28 15:34:21.494124 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:34:21.494090    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-658d7b7c78-hh52q" event={"ID":"040becb9-a46f-40e0-ad85-50cf15ddcafc","Type":"ContainerStarted","Data":"b44278b5c235b9616c709ed44974b64ca572cd2f7816ca26003e5575073086f5"}
Jul 28 15:34:21.556708 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:34:21.556676    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22nh9_ebf75c0d-1597-4a1c-b668-dc2a041eb35e/konnectivity-agent/0.log" limit=1024
Jul 28 15:34:21.692990 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:34:21.692951    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/45da22e1-42e5-4dcf-b132-105b7b343538-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-rsd4q\" (UID: \"45da22e1-42e5-4dcf-b132-105b7b343538\") " pod="openshift-insights/insights-runtime-extractor-rsd4q"
Jul 28 15:34:21.693552 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:34:21.693008    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/45da22e1-42e5-4dcf-b132-105b7b343538-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-rsd4q\" (UID: \"45da22e1-42e5-4dcf-b132-105b7b343538\") " pod="openshift-insights/insights-runtime-extractor-rsd4q"
Jul 28 15:34:21.693552 ip-10-0-131-121 kubenswrapper[2568]: E0728 15:34:21.693091    2568 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 28 15:34:21.693552 ip-10-0-131-121 kubenswrapper[2568]: E0728 15:34:21.693163    2568 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 28 15:34:21.693552 ip-10-0-131-121 kubenswrapper[2568]: E0728 15:34:21.693170    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/45da22e1-42e5-4dcf-b132-105b7b343538-kube-rbac-proxy-cm podName:45da22e1-42e5-4dcf-b132-105b7b343538 nodeName:}" failed. No retries permitted until 2026-07-28 15:34:25.693150616 +0000 UTC m=+45.878345781 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/45da22e1-42e5-4dcf-b132-105b7b343538-kube-rbac-proxy-cm") pod "insights-runtime-extractor-rsd4q" (UID: "45da22e1-42e5-4dcf-b132-105b7b343538") : configmap "kube-rbac-proxy" not found
Jul 28 15:34:21.693552 ip-10-0-131-121 kubenswrapper[2568]: E0728 15:34:21.693214    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/45da22e1-42e5-4dcf-b132-105b7b343538-insights-runtime-extractor-tls podName:45da22e1-42e5-4dcf-b132-105b7b343538 nodeName:}" failed. No retries permitted until 2026-07-28 15:34:25.693199144 +0000 UTC m=+45.878394291 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/45da22e1-42e5-4dcf-b132-105b7b343538-insights-runtime-extractor-tls") pod "insights-runtime-extractor-rsd4q" (UID: "45da22e1-42e5-4dcf-b132-105b7b343538") : secret "insights-runtime-extractor-tls" not found
Jul 28 15:34:22.499853 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:34:22.499818    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/11c0aa38-e5f8-4e7d-9003-755ac2ba3a6c-metrics-tls\") pod \"dns-default-2m6ls\" (UID: \"11c0aa38-e5f8-4e7d-9003-755ac2ba3a6c\") " pod="openshift-dns/dns-default-2m6ls"
Jul 28 15:34:22.500057 ip-10-0-131-121 kubenswrapper[2568]: E0728 15:34:22.499979    2568 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 28 15:34:22.500124 ip-10-0-131-121 kubenswrapper[2568]: E0728 15:34:22.500063    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/11c0aa38-e5f8-4e7d-9003-755ac2ba3a6c-metrics-tls podName:11c0aa38-e5f8-4e7d-9003-755ac2ba3a6c nodeName:}" failed. No retries permitted until 2026-07-28 15:34:30.500038534 +0000 UTC m=+50.685233684 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/11c0aa38-e5f8-4e7d-9003-755ac2ba3a6c-metrics-tls") pod "dns-default-2m6ls" (UID: "11c0aa38-e5f8-4e7d-9003-755ac2ba3a6c") : secret "dns-default-metrics-tls" not found
Jul 28 15:34:23.501876 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:34:23.501839    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-658d7b7c78-hh52q" event={"ID":"040becb9-a46f-40e0-ad85-50cf15ddcafc","Type":"ContainerStarted","Data":"2c34023052eb83f58d9593a90393da7892a9571196809e8c0ad7a7c254b4b038"}
Jul 28 15:34:23.519605 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:34:23.519549    2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca/service-ca-658d7b7c78-hh52q" podStartSLOduration=1.095650517 podStartE2EDuration="3.519530598s" totalImagesPullingTime="2.423880081s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 15:34:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 15:34:23.518813921 +0000 UTC m=+43.704009090" watchObservedRunningTime="2026-07-28 15:34:23.519530598 +0000 UTC m=+43.704725767"
Jul 28 15:34:24.140730 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:34:24.140682    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22nh9_ebf75c0d-1597-4a1c-b668-dc2a041eb35e/konnectivity-agent/0.log" limit=1024
Jul 28 15:34:24.748910 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:34:24.748862    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22nh9_ebf75c0d-1597-4a1c-b668-dc2a041eb35e/konnectivity-agent/0.log" limit=1024
Jul 28 15:34:24.990344 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:34:24.990306    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22nh9_ebf75c0d-1597-4a1c-b668-dc2a041eb35e/konnectivity-agent/0.log" limit=1024
Jul 28 15:34:25.086084 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:34:25.086029    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22nh9_ebf75c0d-1597-4a1c-b668-dc2a041eb35e/konnectivity-agent/0.log" limit=1024
Jul 28 15:34:25.185595 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:34:25.185562    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22nh9_ebf75c0d-1597-4a1c-b668-dc2a041eb35e/konnectivity-agent/0.log" limit=1024
Jul 28 15:34:25.285683 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:34:25.285644    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22nh9_ebf75c0d-1597-4a1c-b668-dc2a041eb35e/konnectivity-agent/0.log" limit=1024
Jul 28 15:34:25.386703 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:34:25.386615    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22nh9_ebf75c0d-1597-4a1c-b668-dc2a041eb35e/konnectivity-agent/0.log" limit=1024
Jul 28 15:34:25.631956 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:34:25.631923    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22nh9_ebf75c0d-1597-4a1c-b668-dc2a041eb35e/konnectivity-agent/0.log" limit=1024
Jul 28 15:34:25.723455 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:34:25.723381    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/45da22e1-42e5-4dcf-b132-105b7b343538-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-rsd4q\" (UID: \"45da22e1-42e5-4dcf-b132-105b7b343538\") " pod="openshift-insights/insights-runtime-extractor-rsd4q"
Jul 28 15:34:25.723455 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:34:25.723435    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/45da22e1-42e5-4dcf-b132-105b7b343538-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-rsd4q\" (UID: \"45da22e1-42e5-4dcf-b132-105b7b343538\") " pod="openshift-insights/insights-runtime-extractor-rsd4q"
Jul 28 15:34:25.723631 ip-10-0-131-121 kubenswrapper[2568]: E0728 15:34:25.723608    2568 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 28 15:34:25.723688 ip-10-0-131-121 kubenswrapper[2568]: E0728 15:34:25.723672    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/45da22e1-42e5-4dcf-b132-105b7b343538-insights-runtime-extractor-tls podName:45da22e1-42e5-4dcf-b132-105b7b343538 nodeName:}" failed. No retries permitted until 2026-07-28 15:34:33.723658403 +0000 UTC m=+53.908853550 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/45da22e1-42e5-4dcf-b132-105b7b343538-insights-runtime-extractor-tls") pod "insights-runtime-extractor-rsd4q" (UID: "45da22e1-42e5-4dcf-b132-105b7b343538") : secret "insights-runtime-extractor-tls" not found
Jul 28 15:34:25.724049 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:34:25.724026    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/45da22e1-42e5-4dcf-b132-105b7b343538-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-rsd4q\" (UID: \"45da22e1-42e5-4dcf-b132-105b7b343538\") " pod="openshift-insights/insights-runtime-extractor-rsd4q"
Jul 28 15:34:25.724811 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:34:25.724771    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22nh9_ebf75c0d-1597-4a1c-b668-dc2a041eb35e/konnectivity-agent/0.log" limit=1024
Jul 28 15:34:26.667589 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:34:26.667534    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22nh9_ebf75c0d-1597-4a1c-b668-dc2a041eb35e/konnectivity-agent/0.log" limit=1024
Jul 28 15:34:26.873847 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:34:26.873809    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22nh9_ebf75c0d-1597-4a1c-b668-dc2a041eb35e/konnectivity-agent/0.log" limit=1024
Jul 28 15:34:27.266542 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:34:27.266490    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22nh9_ebf75c0d-1597-4a1c-b668-dc2a041eb35e/konnectivity-agent/0.log" limit=1024
Jul 28 15:34:27.470303 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:34:27.470268    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22nh9_ebf75c0d-1597-4a1c-b668-dc2a041eb35e/konnectivity-agent/0.log" limit=1024
Jul 28 15:34:28.058772 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:34:28.058741    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22nh9_ebf75c0d-1597-4a1c-b668-dc2a041eb35e/konnectivity-agent/0.log" limit=1024
Jul 28 15:34:28.264098 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:34:28.264064    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22nh9_ebf75c0d-1597-4a1c-b668-dc2a041eb35e/konnectivity-agent/0.log" limit=1024
Jul 28 15:34:28.465454 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:34:28.465371    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22nh9_ebf75c0d-1597-4a1c-b668-dc2a041eb35e/konnectivity-agent/0.log" limit=1024
Jul 28 15:34:28.655271 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:34:28.655241    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22nh9_ebf75c0d-1597-4a1c-b668-dc2a041eb35e/konnectivity-agent/0.log" limit=1024
Jul 28 15:34:28.790325 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:34:28.790297    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22nh9_ebf75c0d-1597-4a1c-b668-dc2a041eb35e/konnectivity-agent/0.log" limit=1024
Jul 28 15:34:29.457302 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:34:29.457272    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22nh9_ebf75c0d-1597-4a1c-b668-dc2a041eb35e/konnectivity-agent/0.log" limit=1024
Jul 28 15:34:29.529303 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:34:29.529264    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22nh9_ebf75c0d-1597-4a1c-b668-dc2a041eb35e/konnectivity-agent/0.log" limit=1024
Jul 28 15:34:29.891701 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:34:29.891670    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22nh9_ebf75c0d-1597-4a1c-b668-dc2a041eb35e/konnectivity-agent/0.log" limit=1024
Jul 28 15:34:30.062576 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:34:30.062544    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22nh9_ebf75c0d-1597-4a1c-b668-dc2a041eb35e/konnectivity-agent/0.log" limit=1024
Jul 28 15:34:30.163242 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:34:30.163160    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22nh9_ebf75c0d-1597-4a1c-b668-dc2a041eb35e/konnectivity-agent/0.log" limit=1024
Jul 28 15:34:30.263326 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:34:30.263293    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22nh9_ebf75c0d-1597-4a1c-b668-dc2a041eb35e/konnectivity-agent/0.log" limit=1024
Jul 28 15:34:30.364338 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:34:30.364295    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22nh9_ebf75c0d-1597-4a1c-b668-dc2a041eb35e/konnectivity-agent/0.log" limit=1024
Jul 28 15:34:30.561023 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:34:30.560986    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/11c0aa38-e5f8-4e7d-9003-755ac2ba3a6c-metrics-tls\") pod \"dns-default-2m6ls\" (UID: \"11c0aa38-e5f8-4e7d-9003-755ac2ba3a6c\") " pod="openshift-dns/dns-default-2m6ls"
Jul 28 15:34:30.563309 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:34:30.563285    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/11c0aa38-e5f8-4e7d-9003-755ac2ba3a6c-metrics-tls\") pod \"dns-default-2m6ls\" (UID: \"11c0aa38-e5f8-4e7d-9003-755ac2ba3a6c\") " pod="openshift-dns/dns-default-2m6ls"
Jul 28 15:34:30.573738 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:34:30.573717    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-2m6ls"
Jul 28 15:34:30.664073 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:34:30.664001    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22nh9_ebf75c0d-1597-4a1c-b668-dc2a041eb35e/konnectivity-agent/0.log" limit=1024
Jul 28 15:34:30.710374 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:34:30.710342    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-2m6ls"]
Jul 28 15:34:30.713419 ip-10-0-131-121 kubenswrapper[2568]: W0728 15:34:30.713383    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod11c0aa38_e5f8_4e7d_9003_755ac2ba3a6c.slice/crio-5aacb8b0e33aaa78a5141d3784b20587c379ef061ba791aba8b4af6f78abdd2d WatchSource:0}: Error finding container 5aacb8b0e33aaa78a5141d3784b20587c379ef061ba791aba8b4af6f78abdd2d: Status 404 returned error can't find the container with id 5aacb8b0e33aaa78a5141d3784b20587c379ef061ba791aba8b4af6f78abdd2d
Jul 28 15:34:31.181611 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:34:31.181573    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22nh9_ebf75c0d-1597-4a1c-b668-dc2a041eb35e/konnectivity-agent/0.log" limit=1024
Jul 28 15:34:31.282681 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:34:31.282643    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22nh9_ebf75c0d-1597-4a1c-b668-dc2a041eb35e/konnectivity-agent/0.log" limit=1024
Jul 28 15:34:31.523965 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:34:31.523929    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-2m6ls" event={"ID":"11c0aa38-e5f8-4e7d-9003-755ac2ba3a6c","Type":"ContainerStarted","Data":"5aacb8b0e33aaa78a5141d3784b20587c379ef061ba791aba8b4af6f78abdd2d"}
Jul 28 15:34:31.669295 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:34:31.669262    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22nh9_ebf75c0d-1597-4a1c-b668-dc2a041eb35e/konnectivity-agent/0.log" limit=1024
Jul 28 15:34:32.060379 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:34:32.060253    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22nh9_ebf75c0d-1597-4a1c-b668-dc2a041eb35e/konnectivity-agent/0.log" limit=1024
Jul 28 15:34:32.255627 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:34:32.255565    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22nh9_ebf75c0d-1597-4a1c-b668-dc2a041eb35e/konnectivity-agent/0.log" limit=1024
Jul 28 15:34:32.663248 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:34:32.663212    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22nh9_ebf75c0d-1597-4a1c-b668-dc2a041eb35e/konnectivity-agent/0.log" limit=1024
Jul 28 15:34:32.726559 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:34:32.726533    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22nh9_ebf75c0d-1597-4a1c-b668-dc2a041eb35e/konnectivity-agent/0.log" limit=1024
Jul 28 15:34:33.464426 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:34:33.464392    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22nh9_ebf75c0d-1597-4a1c-b668-dc2a041eb35e/konnectivity-agent/0.log" limit=1024
Jul 28 15:34:33.531943 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:34:33.531909    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-2m6ls" event={"ID":"11c0aa38-e5f8-4e7d-9003-755ac2ba3a6c","Type":"ContainerStarted","Data":"77a96ced56192b36989ed01879c43643b7ea21585d82d0e9f672d9735fa2f707"}
Jul 28 15:34:33.532133 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:34:33.531961    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-2m6ls" event={"ID":"11c0aa38-e5f8-4e7d-9003-755ac2ba3a6c","Type":"ContainerStarted","Data":"5d927c0979a34769e29a8d00c99336008bbcc2fe4e4a1f3bf08e1e145af3d958"}
Jul 28 15:34:33.532133 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:34:33.532052    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-2m6ls"
Jul 28 15:34:33.551159 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:34:33.551109    2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-2m6ls" podStartSLOduration=17.622407933 podStartE2EDuration="19.551090892s" totalImagesPullingTime="1.928682959s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 15:34:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 15:34:33.550452334 +0000 UTC m=+53.735647530" watchObservedRunningTime="2026-07-28 15:34:33.551090892 +0000 UTC m=+53.736286063"
Jul 28 15:34:33.790665 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:34:33.790631    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/45da22e1-42e5-4dcf-b132-105b7b343538-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-rsd4q\" (UID: \"45da22e1-42e5-4dcf-b132-105b7b343538\") " pod="openshift-insights/insights-runtime-extractor-rsd4q"
Jul 28 15:34:33.793076 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:34:33.793049    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/45da22e1-42e5-4dcf-b132-105b7b343538-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-rsd4q\" (UID: \"45da22e1-42e5-4dcf-b132-105b7b343538\") " pod="openshift-insights/insights-runtime-extractor-rsd4q"
Jul 28 15:34:33.866101 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:34:33.866064    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22nh9_ebf75c0d-1597-4a1c-b668-dc2a041eb35e/konnectivity-agent/0.log" limit=1024
Jul 28 15:34:34.032537 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:34:34.032505    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22nh9_ebf75c0d-1597-4a1c-b668-dc2a041eb35e/konnectivity-agent/0.log" limit=1024
Jul 28 15:34:34.086720 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:34:34.086628    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-rsd4q"
Jul 28 15:34:34.134669 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:34:34.134637    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22nh9_ebf75c0d-1597-4a1c-b668-dc2a041eb35e/konnectivity-agent/0.log" limit=1024
Jul 28 15:34:34.195652 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:34:34.195623    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22nh9_ebf75c0d-1597-4a1c-b668-dc2a041eb35e/konnectivity-agent/0.log" limit=1024
Jul 28 15:34:34.237447 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:34:34.237412    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-rsd4q"]
Jul 28 15:34:34.240579 ip-10-0-131-121 kubenswrapper[2568]: W0728 15:34:34.240539    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod45da22e1_42e5_4dcf_b132_105b7b343538.slice/crio-4ff33a07ca0595a959c0daa624fb7a0121162091076a6bc53eeef8d73fb67eba WatchSource:0}: Error finding container 4ff33a07ca0595a959c0daa624fb7a0121162091076a6bc53eeef8d73fb67eba: Status 404 returned error can't find the container with id 4ff33a07ca0595a959c0daa624fb7a0121162091076a6bc53eeef8d73fb67eba
Jul 28 15:34:34.268250 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:34:34.268207    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22nh9_ebf75c0d-1597-4a1c-b668-dc2a041eb35e/konnectivity-agent/0.log" limit=1024
Jul 28 15:34:34.369328 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:34:34.369245    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22nh9_ebf75c0d-1597-4a1c-b668-dc2a041eb35e/konnectivity-agent/0.log" limit=1024
Jul 28 15:34:34.469570 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:34:34.469541    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22nh9_ebf75c0d-1597-4a1c-b668-dc2a041eb35e/konnectivity-agent/0.log" limit=1024
Jul 28 15:34:34.537223 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:34:34.537186    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-rsd4q" event={"ID":"45da22e1-42e5-4dcf-b132-105b7b343538","Type":"ContainerStarted","Data":"4ff33a07ca0595a959c0daa624fb7a0121162091076a6bc53eeef8d73fb67eba"}
Jul 28 15:34:34.570846 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:34:34.570798    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22nh9_ebf75c0d-1597-4a1c-b668-dc2a041eb35e/konnectivity-agent/0.log" limit=1024
Jul 28 15:34:34.702012 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:34:34.701924    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22nh9_ebf75c0d-1597-4a1c-b668-dc2a041eb35e/konnectivity-agent/0.log" limit=1024
Jul 28 15:34:34.906635 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:34:34.906549    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22nh9_ebf75c0d-1597-4a1c-b668-dc2a041eb35e/konnectivity-agent/0.log" limit=1024
Jul 28 15:34:34.971767 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:34:34.971674    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22nh9_ebf75c0d-1597-4a1c-b668-dc2a041eb35e/konnectivity-agent/0.log" limit=1024
Jul 28 15:34:35.103535 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:34:35.103497    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22nh9_ebf75c0d-1597-4a1c-b668-dc2a041eb35e/konnectivity-agent/0.log" limit=1024
Jul 28 15:34:35.163684 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:34:35.163619    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22nh9_ebf75c0d-1597-4a1c-b668-dc2a041eb35e/konnectivity-agent/0.log" limit=1024
Jul 28 15:34:35.250382 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:34:35.250298    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22nh9_ebf75c0d-1597-4a1c-b668-dc2a041eb35e/konnectivity-agent/0.log" limit=1024
Jul 28 15:34:35.467826 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:34:35.467765    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22nh9_ebf75c0d-1597-4a1c-b668-dc2a041eb35e/konnectivity-agent/0.log" limit=1024
Jul 28 15:34:35.667131 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:34:35.667101    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22nh9_ebf75c0d-1597-4a1c-b668-dc2a041eb35e/konnectivity-agent/0.log" limit=1024
Jul 28 15:34:35.921655 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:34:35.921577    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22nh9_ebf75c0d-1597-4a1c-b668-dc2a041eb35e/konnectivity-agent/0.log" limit=1024
Jul 28 15:34:36.124375 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:34:36.124331    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22nh9_ebf75c0d-1597-4a1c-b668-dc2a041eb35e/konnectivity-agent/0.log" limit=1024
Jul 28 15:34:36.338592 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:34:36.338548    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22nh9_ebf75c0d-1597-4a1c-b668-dc2a041eb35e/konnectivity-agent/0.log" limit=1024
Jul 28 15:34:36.541636 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:34:36.541599    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22nh9_ebf75c0d-1597-4a1c-b668-dc2a041eb35e/konnectivity-agent/0.log" limit=1024
Jul 28 15:34:36.607681 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:34:36.607603    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22nh9_ebf75c0d-1597-4a1c-b668-dc2a041eb35e/konnectivity-agent/0.log" limit=1024
Jul 28 15:34:39.554321 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:34:39.554285    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-rsd4q" event={"ID":"45da22e1-42e5-4dcf-b132-105b7b343538","Type":"ContainerStarted","Data":"f4a84dec2ef466c02381ae581ce8d285c54f675a2baf87899769e4fcc2b388c5"}
Jul 28 15:34:40.468446 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:34:40.468413    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-2vrz4"
Jul 28 15:34:41.366112 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:34:41.366072    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22nh9_ebf75c0d-1597-4a1c-b668-dc2a041eb35e/konnectivity-agent/0.log" limit=1024
Jul 28 15:34:41.440609 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:34:41.440579    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22nh9_ebf75c0d-1597-4a1c-b668-dc2a041eb35e/konnectivity-agent/0.log" limit=1024
Jul 28 15:34:43.540252 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:34:43.540219    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-2m6ls"
Jul 28 15:34:43.570226 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:34:43.570179    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-rsd4q" event={"ID":"45da22e1-42e5-4dcf-b132-105b7b343538","Type":"ContainerStarted","Data":"17871d166ff75c2a0260bffaf1a352520ee5effa31d2b5e2f7a7061f8918056e"}
Jul 28 15:34:43.740658 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:34:43.740624    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22nh9_ebf75c0d-1597-4a1c-b668-dc2a041eb35e/konnectivity-agent/0.log" limit=1024
Jul 28 15:34:43.841421 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:34:43.841332    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22nh9_ebf75c0d-1597-4a1c-b668-dc2a041eb35e/konnectivity-agent/0.log" limit=1024
Jul 28 15:34:44.074869 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:34:44.074830    2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-57df544f56-959nt"]
Jul 28 15:34:44.077118 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:34:44.077090    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-57df544f56-959nt"
Jul 28 15:34:44.079721 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:34:44.079690    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-dockercfg-rhrcw\""
Jul 28 15:34:44.080823 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:34:44.080768    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-hub-kubeconfig\""
Jul 28 15:34:44.080945 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:34:44.080829    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\""
Jul 28 15:34:44.080945 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:34:44.080839    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\""
Jul 28 15:34:44.081138 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:34:44.081122    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\""
Jul 28 15:34:44.090171 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:34:44.090143    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-57df544f56-959nt"]
Jul 28 15:34:44.172477 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:34:44.172401    2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-console/networking-console-plugin-74c9f884b7-cd6sf"]
Jul 28 15:34:44.173621 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:34:44.173591    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vdbcz\" (UniqueName: \"kubernetes.io/projected/94325de5-8c27-450e-bcfa-e709d05f0266-kube-api-access-vdbcz\") pod \"managed-serviceaccount-addon-agent-57df544f56-959nt\" (UID: \"94325de5-8c27-450e-bcfa-e709d05f0266\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-57df544f56-959nt"
Jul 28 15:34:44.173760 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:34:44.173641    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/94325de5-8c27-450e-bcfa-e709d05f0266-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-57df544f56-959nt\" (UID: \"94325de5-8c27-450e-bcfa-e709d05f0266\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-57df544f56-959nt"
Jul 28 15:34:44.174639 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:34:44.174619    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-74c9f884b7-cd6sf"
Jul 28 15:34:44.177219 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:34:44.177194    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\""
Jul 28 15:34:44.177341 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:34:44.177243    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-rtzf7\""
Jul 28 15:34:44.177633 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:34:44.177611    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\""
Jul 28 15:34:44.183208 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:34:44.182592    2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-74d48dfc45-9c7c5"]
Jul 28 15:34:44.186467 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:34:44.186347    2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-7c97dc4555-x5gtq"]
Jul 28 15:34:44.186602 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:34:44.186511    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-74d48dfc45-9c7c5"
Jul 28 15:34:44.188617 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:34:44.188598    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-74c9f884b7-cd6sf"]
Jul 28 15:34:44.188733 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:34:44.188694    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-7c97dc4555-x5gtq"
Jul 28 15:34:44.192258 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:34:44.192239    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Jul 28 15:34:44.192492 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:34:44.192474    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Jul 28 15:34:44.193196 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:34:44.192970    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Jul 28 15:34:44.193411 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:34:44.193330    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-4ssqh\""
Jul 28 15:34:44.194205 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:34:44.193665    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Jul 28 15:34:44.194205 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:34:44.193980    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Jul 28 15:34:44.194456 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:34:44.194213    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Jul 28 15:34:44.194554 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:34:44.194534    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"default-dockercfg-7txn9\""
Jul 28 15:34:44.194762 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:34:44.194739    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Jul 28 15:34:44.200396 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:34:44.200374    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-7c97dc4555-x5gtq"]
Jul 28 15:34:44.203595 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:34:44.203569    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-74d48dfc45-9c7c5"]
Jul 28 15:34:44.220626 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:34:44.220592    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22nh9_ebf75c0d-1597-4a1c-b668-dc2a041eb35e/konnectivity-agent/0.log" limit=1024
Jul 28 15:34:44.274556 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:34:44.274505    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/f81be785-6cfa-452b-b654-3f8c54893ab1-console-config\") pod \"console-74d48dfc45-9c7c5\" (UID: \"f81be785-6cfa-452b-b654-3f8c54893ab1\") " pod="openshift-console/console-74d48dfc45-9c7c5"
Jul 28 15:34:44.274825 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:34:44.274609    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/39b0e13f-c4ec-48bd-9c67-99b1d4506463-networking-console-plugin-cert\") pod \"networking-console-plugin-74c9f884b7-cd6sf\" (UID: \"39b0e13f-c4ec-48bd-9c67-99b1d4506463\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-cd6sf"
Jul 28 15:34:44.274825 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:34:44.274637    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/f81be785-6cfa-452b-b654-3f8c54893ab1-console-serving-cert\") pod \"console-74d48dfc45-9c7c5\" (UID: \"f81be785-6cfa-452b-b654-3f8c54893ab1\") " pod="openshift-console/console-74d48dfc45-9c7c5"
Jul 28 15:34:44.274825 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:34:44.274659    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p77d2\" (UniqueName: \"kubernetes.io/projected/1e75cd2a-7fb1-4d78-aef3-523a82c5dde6-kube-api-access-p77d2\") pod \"downloads-7c97dc4555-x5gtq\" (UID: \"1e75cd2a-7fb1-4d78-aef3-523a82c5dde6\") " pod="openshift-console/downloads-7c97dc4555-x5gtq"
Jul 28 15:34:44.274825 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:34:44.274689    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vdbcz\" (UniqueName: \"kubernetes.io/projected/94325de5-8c27-450e-bcfa-e709d05f0266-kube-api-access-vdbcz\") pod \"managed-serviceaccount-addon-agent-57df544f56-959nt\" (UID: \"94325de5-8c27-450e-bcfa-e709d05f0266\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-57df544f56-959nt"
Jul 28 15:34:44.274825 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:34:44.274659    2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-65444944d6-t5hmt"]
Jul 28 15:34:44.274825 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:34:44.274716    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/f81be785-6cfa-452b-b654-3f8c54893ab1-oauth-serving-cert\") pod \"console-74d48dfc45-9c7c5\" (UID: \"f81be785-6cfa-452b-b654-3f8c54893ab1\") " pod="openshift-console/console-74d48dfc45-9c7c5"
Jul 28 15:34:44.274825 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:34:44.274756    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-px687\" (UniqueName: \"kubernetes.io/projected/f81be785-6cfa-452b-b654-3f8c54893ab1-kube-api-access-px687\") pod \"console-74d48dfc45-9c7c5\" (UID: \"f81be785-6cfa-452b-b654-3f8c54893ab1\") " pod="openshift-console/console-74d48dfc45-9c7c5"
Jul 28 15:34:44.275251 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:34:44.274836    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/94325de5-8c27-450e-bcfa-e709d05f0266-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-57df544f56-959nt\" (UID: \"94325de5-8c27-450e-bcfa-e709d05f0266\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-57df544f56-959nt"
Jul 28 15:34:44.275251 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:34:44.275093    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/f81be785-6cfa-452b-b654-3f8c54893ab1-service-ca\") pod \"console-74d48dfc45-9c7c5\" (UID: \"f81be785-6cfa-452b-b654-3f8c54893ab1\") " pod="openshift-console/console-74d48dfc45-9c7c5"
Jul 28 15:34:44.275251 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:34:44.275157    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/39b0e13f-c4ec-48bd-9c67-99b1d4506463-nginx-conf\") pod \"networking-console-plugin-74c9f884b7-cd6sf\" (UID: \"39b0e13f-c4ec-48bd-9c67-99b1d4506463\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-cd6sf"
Jul 28 15:34:44.275251 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:34:44.275187    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/f81be785-6cfa-452b-b654-3f8c54893ab1-console-oauth-config\") pod \"console-74d48dfc45-9c7c5\" (UID: \"f81be785-6cfa-452b-b654-3f8c54893ab1\") " pod="openshift-console/console-74d48dfc45-9c7c5"
Jul 28 15:34:44.277303 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:34:44.277282    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-65444944d6-t5hmt"
Jul 28 15:34:44.277595 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:34:44.277572    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/94325de5-8c27-450e-bcfa-e709d05f0266-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-57df544f56-959nt\" (UID: \"94325de5-8c27-450e-bcfa-e709d05f0266\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-57df544f56-959nt"
Jul 28 15:34:44.280084 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:34:44.280063    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Jul 28 15:34:44.280196 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:34:44.280122    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Jul 28 15:34:44.280196 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:34:44.280160    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-fxdc4\""
Jul 28 15:34:44.280374 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:34:44.280349    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Jul 28 15:34:44.286910 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:34:44.286888    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Jul 28 15:34:44.288608 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:34:44.288583    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vdbcz\" (UniqueName: \"kubernetes.io/projected/94325de5-8c27-450e-bcfa-e709d05f0266-kube-api-access-vdbcz\") pod \"managed-serviceaccount-addon-agent-57df544f56-959nt\" (UID: \"94325de5-8c27-450e-bcfa-e709d05f0266\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-57df544f56-959nt"
Jul 28 15:34:44.294174 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:34:44.294142    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-65444944d6-t5hmt"]
Jul 28 15:34:44.320915 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:34:44.320879    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22nh9_ebf75c0d-1597-4a1c-b668-dc2a041eb35e/konnectivity-agent/0.log" limit=1024
Jul 28 15:34:44.376253 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:34:44.376219    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9p8v7\" (UniqueName: \"kubernetes.io/projected/8a2e179f-a54d-480c-84f2-d7278c611f86-kube-api-access-9p8v7\") pod \"image-registry-65444944d6-t5hmt\" (UID: \"8a2e179f-a54d-480c-84f2-d7278c611f86\") " pod="openshift-image-registry/image-registry-65444944d6-t5hmt"
Jul 28 15:34:44.376491 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:34:44.376259    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/8a2e179f-a54d-480c-84f2-d7278c611f86-registry-certificates\") pod \"image-registry-65444944d6-t5hmt\" (UID: \"8a2e179f-a54d-480c-84f2-d7278c611f86\") " pod="openshift-image-registry/image-registry-65444944d6-t5hmt"
Jul 28 15:34:44.376491 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:34:44.376291    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/8a2e179f-a54d-480c-84f2-d7278c611f86-installation-pull-secrets\") pod \"image-registry-65444944d6-t5hmt\" (UID: \"8a2e179f-a54d-480c-84f2-d7278c611f86\") " pod="openshift-image-registry/image-registry-65444944d6-t5hmt"
Jul 28 15:34:44.376491 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:34:44.376358    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-px687\" (UniqueName: \"kubernetes.io/projected/f81be785-6cfa-452b-b654-3f8c54893ab1-kube-api-access-px687\") pod \"console-74d48dfc45-9c7c5\" (UID: \"f81be785-6cfa-452b-b654-3f8c54893ab1\") " pod="openshift-console/console-74d48dfc45-9c7c5"
Jul 28 15:34:44.376491 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:34:44.376453    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/39b0e13f-c4ec-48bd-9c67-99b1d4506463-networking-console-plugin-cert\") pod \"networking-console-plugin-74c9f884b7-cd6sf\" (UID: \"39b0e13f-c4ec-48bd-9c67-99b1d4506463\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-cd6sf"
Jul 28 15:34:44.376828 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:34:44.376493    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/8a2e179f-a54d-480c-84f2-d7278c611f86-registry-tls\") pod \"image-registry-65444944d6-t5hmt\" (UID: \"8a2e179f-a54d-480c-84f2-d7278c611f86\") " pod="openshift-image-registry/image-registry-65444944d6-t5hmt"
Jul 28 15:34:44.376828 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:34:44.376525    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/8a2e179f-a54d-480c-84f2-d7278c611f86-trusted-ca\") pod \"image-registry-65444944d6-t5hmt\" (UID: \"8a2e179f-a54d-480c-84f2-d7278c611f86\") " pod="openshift-image-registry/image-registry-65444944d6-t5hmt"
Jul 28 15:34:44.376828 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:34:44.376569    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/39b0e13f-c4ec-48bd-9c67-99b1d4506463-nginx-conf\") pod \"networking-console-plugin-74c9f884b7-cd6sf\" (UID: \"39b0e13f-c4ec-48bd-9c67-99b1d4506463\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-cd6sf"
Jul 28 15:34:44.376828 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:34:44.376591    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/8a2e179f-a54d-480c-84f2-d7278c611f86-ca-trust-extracted\") pod \"image-registry-65444944d6-t5hmt\" (UID: \"8a2e179f-a54d-480c-84f2-d7278c611f86\") " pod="openshift-image-registry/image-registry-65444944d6-t5hmt"
Jul 28 15:34:44.376828 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:34:44.376612    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/8a2e179f-a54d-480c-84f2-d7278c611f86-image-registry-private-configuration\") pod \"image-registry-65444944d6-t5hmt\" (UID: \"8a2e179f-a54d-480c-84f2-d7278c611f86\") " pod="openshift-image-registry/image-registry-65444944d6-t5hmt"
Jul 28 15:34:44.376828 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:34:44.376645    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/f81be785-6cfa-452b-b654-3f8c54893ab1-console-config\") pod \"console-74d48dfc45-9c7c5\" (UID: \"f81be785-6cfa-452b-b654-3f8c54893ab1\") " pod="openshift-console/console-74d48dfc45-9c7c5"
Jul 28 15:34:44.376828 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:34:44.376677    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p77d2\" (UniqueName: \"kubernetes.io/projected/1e75cd2a-7fb1-4d78-aef3-523a82c5dde6-kube-api-access-p77d2\") pod \"downloads-7c97dc4555-x5gtq\" (UID: \"1e75cd2a-7fb1-4d78-aef3-523a82c5dde6\") " pod="openshift-console/downloads-7c97dc4555-x5gtq"
Jul 28 15:34:44.376828 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:34:44.376718    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/f81be785-6cfa-452b-b654-3f8c54893ab1-console-serving-cert\") pod \"console-74d48dfc45-9c7c5\" (UID: \"f81be785-6cfa-452b-b654-3f8c54893ab1\") " pod="openshift-console/console-74d48dfc45-9c7c5"
Jul 28 15:34:44.376828 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:34:44.376741    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/8a2e179f-a54d-480c-84f2-d7278c611f86-bound-sa-token\") pod \"image-registry-65444944d6-t5hmt\" (UID: \"8a2e179f-a54d-480c-84f2-d7278c611f86\") " pod="openshift-image-registry/image-registry-65444944d6-t5hmt"
Jul 28 15:34:44.376828 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:34:44.376767    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/f81be785-6cfa-452b-b654-3f8c54893ab1-oauth-serving-cert\") pod \"console-74d48dfc45-9c7c5\" (UID: \"f81be785-6cfa-452b-b654-3f8c54893ab1\") " pod="openshift-console/console-74d48dfc45-9c7c5"
Jul 28 15:34:44.376828 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:34:44.376812    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/f81be785-6cfa-452b-b654-3f8c54893ab1-service-ca\") pod \"console-74d48dfc45-9c7c5\" (UID: \"f81be785-6cfa-452b-b654-3f8c54893ab1\") " pod="openshift-console/console-74d48dfc45-9c7c5"
Jul 28 15:34:44.377344 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:34:44.376851    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/f81be785-6cfa-452b-b654-3f8c54893ab1-console-oauth-config\") pod \"console-74d48dfc45-9c7c5\" (UID: \"f81be785-6cfa-452b-b654-3f8c54893ab1\") " pod="openshift-console/console-74d48dfc45-9c7c5"
Jul 28 15:34:44.377398 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:34:44.377375    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/39b0e13f-c4ec-48bd-9c67-99b1d4506463-nginx-conf\") pod \"networking-console-plugin-74c9f884b7-cd6sf\" (UID: \"39b0e13f-c4ec-48bd-9c67-99b1d4506463\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-cd6sf"
Jul 28 15:34:44.377739 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:34:44.377713    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/f81be785-6cfa-452b-b654-3f8c54893ab1-oauth-serving-cert\") pod \"console-74d48dfc45-9c7c5\" (UID: \"f81be785-6cfa-452b-b654-3f8c54893ab1\") " pod="openshift-console/console-74d48dfc45-9c7c5"
Jul 28 15:34:44.377739 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:34:44.377731    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/f81be785-6cfa-452b-b654-3f8c54893ab1-console-config\") pod \"console-74d48dfc45-9c7c5\" (UID: \"f81be785-6cfa-452b-b654-3f8c54893ab1\") " pod="openshift-console/console-74d48dfc45-9c7c5"
Jul 28 15:34:44.377915 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:34:44.377750    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/f81be785-6cfa-452b-b654-3f8c54893ab1-service-ca\") pod \"console-74d48dfc45-9c7c5\" (UID: \"f81be785-6cfa-452b-b654-3f8c54893ab1\") " pod="openshift-console/console-74d48dfc45-9c7c5"
Jul 28 15:34:44.379418 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:34:44.379393    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/39b0e13f-c4ec-48bd-9c67-99b1d4506463-networking-console-plugin-cert\") pod \"networking-console-plugin-74c9f884b7-cd6sf\" (UID: \"39b0e13f-c4ec-48bd-9c67-99b1d4506463\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-cd6sf"
Jul 28 15:34:44.379530 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:34:44.379429    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/f81be785-6cfa-452b-b654-3f8c54893ab1-console-oauth-config\") pod \"console-74d48dfc45-9c7c5\" (UID: \"f81be785-6cfa-452b-b654-3f8c54893ab1\") " pod="openshift-console/console-74d48dfc45-9c7c5"
Jul 28 15:34:44.379667 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:34:44.379649    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/f81be785-6cfa-452b-b654-3f8c54893ab1-console-serving-cert\") pod \"console-74d48dfc45-9c7c5\" (UID: \"f81be785-6cfa-452b-b654-3f8c54893ab1\") " pod="openshift-console/console-74d48dfc45-9c7c5"
Jul 28 15:34:44.385578 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:34:44.385550    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-px687\" (UniqueName: \"kubernetes.io/projected/f81be785-6cfa-452b-b654-3f8c54893ab1-kube-api-access-px687\") pod \"console-74d48dfc45-9c7c5\" (UID: \"f81be785-6cfa-452b-b654-3f8c54893ab1\") " pod="openshift-console/console-74d48dfc45-9c7c5"
Jul 28 15:34:44.386205 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:34:44.386170    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p77d2\" (UniqueName: \"kubernetes.io/projected/1e75cd2a-7fb1-4d78-aef3-523a82c5dde6-kube-api-access-p77d2\") pod \"downloads-7c97dc4555-x5gtq\" (UID: \"1e75cd2a-7fb1-4d78-aef3-523a82c5dde6\") " pod="openshift-console/downloads-7c97dc4555-x5gtq"
Jul 28 15:34:44.397579 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:34:44.397547    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22nh9_ebf75c0d-1597-4a1c-b668-dc2a041eb35e/konnectivity-agent/0.log" limit=1024
Jul 28 15:34:44.402493 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:34:44.402459    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-57df544f56-959nt"
Jul 28 15:34:44.477465 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:34:44.477427    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/8a2e179f-a54d-480c-84f2-d7278c611f86-registry-certificates\") pod \"image-registry-65444944d6-t5hmt\" (UID: \"8a2e179f-a54d-480c-84f2-d7278c611f86\") " pod="openshift-image-registry/image-registry-65444944d6-t5hmt"
Jul 28 15:34:44.477643 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:34:44.477476    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/8a2e179f-a54d-480c-84f2-d7278c611f86-installation-pull-secrets\") pod \"image-registry-65444944d6-t5hmt\" (UID: \"8a2e179f-a54d-480c-84f2-d7278c611f86\") " pod="openshift-image-registry/image-registry-65444944d6-t5hmt"
Jul 28 15:34:44.477643 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:34:44.477541    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/8a2e179f-a54d-480c-84f2-d7278c611f86-registry-tls\") pod \"image-registry-65444944d6-t5hmt\" (UID: \"8a2e179f-a54d-480c-84f2-d7278c611f86\") " pod="openshift-image-registry/image-registry-65444944d6-t5hmt"
Jul 28 15:34:44.477643 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:34:44.477577    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/8a2e179f-a54d-480c-84f2-d7278c611f86-trusted-ca\") pod \"image-registry-65444944d6-t5hmt\" (UID: \"8a2e179f-a54d-480c-84f2-d7278c611f86\") " pod="openshift-image-registry/image-registry-65444944d6-t5hmt"
Jul 28 15:34:44.477643 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:34:44.477621    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/8a2e179f-a54d-480c-84f2-d7278c611f86-ca-trust-extracted\") pod \"image-registry-65444944d6-t5hmt\" (UID: \"8a2e179f-a54d-480c-84f2-d7278c611f86\") " pod="openshift-image-registry/image-registry-65444944d6-t5hmt"
Jul 28 15:34:44.477883 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:34:44.477647    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/8a2e179f-a54d-480c-84f2-d7278c611f86-image-registry-private-configuration\") pod \"image-registry-65444944d6-t5hmt\" (UID: \"8a2e179f-a54d-480c-84f2-d7278c611f86\") " pod="openshift-image-registry/image-registry-65444944d6-t5hmt"
Jul 28 15:34:44.477883 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:34:44.477716    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/8a2e179f-a54d-480c-84f2-d7278c611f86-bound-sa-token\") pod \"image-registry-65444944d6-t5hmt\" (UID: \"8a2e179f-a54d-480c-84f2-d7278c611f86\") " pod="openshift-image-registry/image-registry-65444944d6-t5hmt"
Jul 28 15:34:44.477883 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:34:44.477762    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9p8v7\" (UniqueName: \"kubernetes.io/projected/8a2e179f-a54d-480c-84f2-d7278c611f86-kube-api-access-9p8v7\") pod \"image-registry-65444944d6-t5hmt\" (UID: \"8a2e179f-a54d-480c-84f2-d7278c611f86\") " pod="openshift-image-registry/image-registry-65444944d6-t5hmt"
Jul 28 15:34:44.478426 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:34:44.478398    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/8a2e179f-a54d-480c-84f2-d7278c611f86-ca-trust-extracted\") pod \"image-registry-65444944d6-t5hmt\" (UID: \"8a2e179f-a54d-480c-84f2-d7278c611f86\") " pod="openshift-image-registry/image-registry-65444944d6-t5hmt"
Jul 28 15:34:44.478709 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:34:44.478684    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/8a2e179f-a54d-480c-84f2-d7278c611f86-registry-certificates\") pod \"image-registry-65444944d6-t5hmt\" (UID: \"8a2e179f-a54d-480c-84f2-d7278c611f86\") " pod="openshift-image-registry/image-registry-65444944d6-t5hmt"
Jul 28 15:34:44.479533 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:34:44.479507    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/8a2e179f-a54d-480c-84f2-d7278c611f86-trusted-ca\") pod \"image-registry-65444944d6-t5hmt\" (UID: \"8a2e179f-a54d-480c-84f2-d7278c611f86\") " pod="openshift-image-registry/image-registry-65444944d6-t5hmt"
Jul 28 15:34:44.481504 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:34:44.481460    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/8a2e179f-a54d-480c-84f2-d7278c611f86-installation-pull-secrets\") pod \"image-registry-65444944d6-t5hmt\" (UID: \"8a2e179f-a54d-480c-84f2-d7278c611f86\") " pod="openshift-image-registry/image-registry-65444944d6-t5hmt"
Jul 28 15:34:44.482041 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:34:44.482002    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/8a2e179f-a54d-480c-84f2-d7278c611f86-registry-tls\") pod \"image-registry-65444944d6-t5hmt\" (UID: \"8a2e179f-a54d-480c-84f2-d7278c611f86\") " pod="openshift-image-registry/image-registry-65444944d6-t5hmt"
Jul 28 15:34:44.483975 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:34:44.483940    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/8a2e179f-a54d-480c-84f2-d7278c611f86-image-registry-private-configuration\") pod \"image-registry-65444944d6-t5hmt\" (UID: \"8a2e179f-a54d-480c-84f2-d7278c611f86\") " pod="openshift-image-registry/image-registry-65444944d6-t5hmt"
Jul 28 15:34:44.486743 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:34:44.486723    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-74c9f884b7-cd6sf"
Jul 28 15:34:44.487655 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:34:44.487597    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/8a2e179f-a54d-480c-84f2-d7278c611f86-bound-sa-token\") pod \"image-registry-65444944d6-t5hmt\" (UID: \"8a2e179f-a54d-480c-84f2-d7278c611f86\") " pod="openshift-image-registry/image-registry-65444944d6-t5hmt"
Jul 28 15:34:44.487770 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:34:44.487752    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9p8v7\" (UniqueName: \"kubernetes.io/projected/8a2e179f-a54d-480c-84f2-d7278c611f86-kube-api-access-9p8v7\") pod \"image-registry-65444944d6-t5hmt\" (UID: \"8a2e179f-a54d-480c-84f2-d7278c611f86\") " pod="openshift-image-registry/image-registry-65444944d6-t5hmt"
Jul 28 15:34:44.502474 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:34:44.502356    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-74d48dfc45-9c7c5"
Jul 28 15:34:44.508867 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:34:44.508839    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-7c97dc4555-x5gtq"
Jul 28 15:34:44.547240 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:34:44.547204    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-57df544f56-959nt"]
Jul 28 15:34:44.585400 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:34:44.585368    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22nh9_ebf75c0d-1597-4a1c-b668-dc2a041eb35e/konnectivity-agent/0.log" limit=1024
Jul 28 15:34:44.600318 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:34:44.600280    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-65444944d6-t5hmt"
Jul 28 15:34:44.685398 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:34:44.685358    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22nh9_ebf75c0d-1597-4a1c-b668-dc2a041eb35e/konnectivity-agent/0.log" limit=1024
Jul 28 15:34:45.128657 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:34:45.128623    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22nh9_ebf75c0d-1597-4a1c-b668-dc2a041eb35e/konnectivity-agent/0.log" limit=1024
Jul 28 15:34:45.200618 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:34:45.200582    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22nh9_ebf75c0d-1597-4a1c-b668-dc2a041eb35e/konnectivity-agent/0.log" limit=1024
Jul 28 15:34:45.589967 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:34:45.589928    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22nh9_ebf75c0d-1597-4a1c-b668-dc2a041eb35e/konnectivity-agent/0.log" limit=1024
Jul 28 15:34:45.952694 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:34:45.952618    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22nh9_ebf75c0d-1597-4a1c-b668-dc2a041eb35e/konnectivity-agent/0.log" limit=1024
Jul 28 15:34:46.052603 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:34:46.052572    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22nh9_ebf75c0d-1597-4a1c-b668-dc2a041eb35e/konnectivity-agent/0.log" limit=1024
Jul 28 15:34:46.092887 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:34:46.092849    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3ba72ab7-7ab4-4e7f-ace4-8a6b7bbff25c-original-pull-secret\") pod \"global-pull-secret-syncer-tr647\" (UID: \"3ba72ab7-7ab4-4e7f-ace4-8a6b7bbff25c\") " pod="kube-system/global-pull-secret-syncer-tr647"
Jul 28 15:34:46.093162 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:34:46.092924    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6100b515-cfe9-47ef-b672-d698ae677bd0-metrics-certs\") pod \"network-metrics-daemon-ckpr5\" (UID: \"6100b515-cfe9-47ef-b672-d698ae677bd0\") " pod="openshift-multus/network-metrics-daemon-ckpr5"
Jul 28 15:34:46.095586 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:34:46.095566    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 28 15:34:46.096595 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:34:46.096581    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 28 15:34:46.106043 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:34:46.106021    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6100b515-cfe9-47ef-b672-d698ae677bd0-metrics-certs\") pod \"network-metrics-daemon-ckpr5\" (UID: \"6100b515-cfe9-47ef-b672-d698ae677bd0\") " pod="openshift-multus/network-metrics-daemon-ckpr5"
Jul 28 15:34:46.106361 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:34:46.106342    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3ba72ab7-7ab4-4e7f-ace4-8a6b7bbff25c-original-pull-secret\") pod \"global-pull-secret-syncer-tr647\" (UID: \"3ba72ab7-7ab4-4e7f-ace4-8a6b7bbff25c\") " pod="kube-system/global-pull-secret-syncer-tr647"
Jul 28 15:34:46.142201 ip-10-0-131-121 kubenswrapper[2568]: W0728 15:34:46.142162    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod94325de5_8c27_450e_bcfa_e709d05f0266.slice/crio-797e0a64b9439e71c95d4869b79dad5634367e432f4af59395fe53b8e6dc9226 WatchSource:0}: Error finding container 797e0a64b9439e71c95d4869b79dad5634367e432f4af59395fe53b8e6dc9226: Status 404 returned error can't find the container with id 797e0a64b9439e71c95d4869b79dad5634367e432f4af59395fe53b8e6dc9226
Jul 28 15:34:46.194251 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:34:46.194165    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qks42\" (UniqueName: \"kubernetes.io/projected/644d231e-7d3c-4aef-a201-87266e83d526-kube-api-access-qks42\") pod \"network-check-target-85ncq\" (UID: \"644d231e-7d3c-4aef-a201-87266e83d526\") " pod="openshift-network-diagnostics/network-check-target-85ncq"
Jul 28 15:34:46.197323 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:34:46.197040    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 28 15:34:46.208398 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:34:46.208119    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 28 15:34:46.230590 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:34:46.229216    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tr647"
Jul 28 15:34:46.230590 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:34:46.229817    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-xmhl6\""
Jul 28 15:34:46.230590 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:34:46.230367    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qks42\" (UniqueName: \"kubernetes.io/projected/644d231e-7d3c-4aef-a201-87266e83d526-kube-api-access-qks42\") pod \"network-check-target-85ncq\" (UID: \"644d231e-7d3c-4aef-a201-87266e83d526\") " pod="openshift-network-diagnostics/network-check-target-85ncq"
Jul 28 15:34:46.237433 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:34:46.236941    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ckpr5"
Jul 28 15:34:46.307921 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:34:46.307676    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-ptv7b\""
Jul 28 15:34:46.315457 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:34:46.315216    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-85ncq"
Jul 28 15:34:46.326044 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:34:46.325976    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-74d48dfc45-9c7c5"]
Jul 28 15:34:46.337077 ip-10-0-131-121 kubenswrapper[2568]: W0728 15:34:46.337042    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf81be785_6cfa_452b_b654_3f8c54893ab1.slice/crio-49f764c327385e640c0a6767b8e1c088a4e1d73040ff24781fedd3de57d2cef0 WatchSource:0}: Error finding container 49f764c327385e640c0a6767b8e1c088a4e1d73040ff24781fedd3de57d2cef0: Status 404 returned error can't find the container with id 49f764c327385e640c0a6767b8e1c088a4e1d73040ff24781fedd3de57d2cef0
Jul 28 15:34:46.428086 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:34:46.428024    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-ckpr5"]
Jul 28 15:34:46.430978 ip-10-0-131-121 kubenswrapper[2568]: W0728 15:34:46.430949    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6100b515_cfe9_47ef_b672_d698ae677bd0.slice/crio-466cc3098c7eb7b621b389f797c4a3353d47bbf21aff00fc21f1138986a9d42d WatchSource:0}: Error finding container 466cc3098c7eb7b621b389f797c4a3353d47bbf21aff00fc21f1138986a9d42d: Status 404 returned error can't find the container with id 466cc3098c7eb7b621b389f797c4a3353d47bbf21aff00fc21f1138986a9d42d
Jul 28 15:34:46.442471 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:34:46.442383    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-tr647"]
Jul 28 15:34:46.446075 ip-10-0-131-121 kubenswrapper[2568]: W0728 15:34:46.445984    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3ba72ab7_7ab4_4e7f_ace4_8a6b7bbff25c.slice/crio-ee04d17962859fe8485cb8f905c73372104e8d1e9688f441e1c0a32b9ba03662 WatchSource:0}: Error finding container ee04d17962859fe8485cb8f905c73372104e8d1e9688f441e1c0a32b9ba03662: Status 404 returned error can't find the container with id ee04d17962859fe8485cb8f905c73372104e8d1e9688f441e1c0a32b9ba03662
Jul 28 15:34:46.476855 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:34:46.476759    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-85ncq"]
Jul 28 15:34:46.479531 ip-10-0-131-121 kubenswrapper[2568]: W0728 15:34:46.479499    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod644d231e_7d3c_4aef_a201_87266e83d526.slice/crio-0c9f95bf4c0ddae90cb542cc040b7f6ab83ad6d39e0ca2e647b055af906b9b6c WatchSource:0}: Error finding container 0c9f95bf4c0ddae90cb542cc040b7f6ab83ad6d39e0ca2e647b055af906b9b6c: Status 404 returned error can't find the container with id 0c9f95bf4c0ddae90cb542cc040b7f6ab83ad6d39e0ca2e647b055af906b9b6c
Jul 28 15:34:46.548827 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:34:46.548796    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-65444944d6-t5hmt"]
Jul 28 15:34:46.551241 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:34:46.551206    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-7c97dc4555-x5gtq"]
Jul 28 15:34:46.552621 ip-10-0-131-121 kubenswrapper[2568]: W0728 15:34:46.552600    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8a2e179f_a54d_480c_84f2_d7278c611f86.slice/crio-3cae3ea3843e624bede19f1656b52fafce8c22168621aef47996481a8e24fcae WatchSource:0}: Error finding container 3cae3ea3843e624bede19f1656b52fafce8c22168621aef47996481a8e24fcae: Status 404 returned error can't find the container with id 3cae3ea3843e624bede19f1656b52fafce8c22168621aef47996481a8e24fcae
Jul 28 15:34:46.554267 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:34:46.554246    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-74c9f884b7-cd6sf"]
Jul 28 15:34:46.554364 ip-10-0-131-121 kubenswrapper[2568]: W0728 15:34:46.554315    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1e75cd2a_7fb1_4d78_aef3_523a82c5dde6.slice/crio-71927a215a205852f4838dcd972d1a076bfdb0ea48811d83dce2ab3d74a1951c WatchSource:0}: Error finding container 71927a215a205852f4838dcd972d1a076bfdb0ea48811d83dce2ab3d74a1951c: Status 404 returned error can't find the container with id 71927a215a205852f4838dcd972d1a076bfdb0ea48811d83dce2ab3d74a1951c
Jul 28 15:34:46.557026 ip-10-0-131-121 kubenswrapper[2568]: W0728 15:34:46.557006    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod39b0e13f_c4ec_48bd_9c67_99b1d4506463.slice/crio-084f9fb592185a4d71e2bbd34fe2ce6ec2d4108c7198e4b68a8ee5f6d146c671 WatchSource:0}: Error finding container 084f9fb592185a4d71e2bbd34fe2ce6ec2d4108c7198e4b68a8ee5f6d146c671: Status 404 returned error can't find the container with id 084f9fb592185a4d71e2bbd34fe2ce6ec2d4108c7198e4b68a8ee5f6d146c671
Jul 28 15:34:46.582487 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:34:46.582458    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-ckpr5" event={"ID":"6100b515-cfe9-47ef-b672-d698ae677bd0","Type":"ContainerStarted","Data":"466cc3098c7eb7b621b389f797c4a3353d47bbf21aff00fc21f1138986a9d42d"}
Jul 28 15:34:46.583596 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:34:46.583573    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-74c9f884b7-cd6sf" event={"ID":"39b0e13f-c4ec-48bd-9c67-99b1d4506463","Type":"ContainerStarted","Data":"084f9fb592185a4d71e2bbd34fe2ce6ec2d4108c7198e4b68a8ee5f6d146c671"}
Jul 28 15:34:46.585826 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:34:46.585800    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-rsd4q" event={"ID":"45da22e1-42e5-4dcf-b132-105b7b343538","Type":"ContainerStarted","Data":"154be9798765459eb6291b222a4ca36a22b56c4141151151dbd4aecc45f18f22"}
Jul 28 15:34:46.586959 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:34:46.586937    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-tr647" event={"ID":"3ba72ab7-7ab4-4e7f-ace4-8a6b7bbff25c","Type":"ContainerStarted","Data":"ee04d17962859fe8485cb8f905c73372104e8d1e9688f441e1c0a32b9ba03662"}
Jul 28 15:34:46.587998 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:34:46.587977    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-7c97dc4555-x5gtq" event={"ID":"1e75cd2a-7fb1-4d78-aef3-523a82c5dde6","Type":"ContainerStarted","Data":"71927a215a205852f4838dcd972d1a076bfdb0ea48811d83dce2ab3d74a1951c"}
Jul 28 15:34:46.589084 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:34:46.589064    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-57df544f56-959nt" event={"ID":"94325de5-8c27-450e-bcfa-e709d05f0266","Type":"ContainerStarted","Data":"797e0a64b9439e71c95d4869b79dad5634367e432f4af59395fe53b8e6dc9226"}
Jul 28 15:34:46.590345 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:34:46.590323    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-65444944d6-t5hmt" event={"ID":"8a2e179f-a54d-480c-84f2-d7278c611f86","Type":"ContainerStarted","Data":"3cae3ea3843e624bede19f1656b52fafce8c22168621aef47996481a8e24fcae"}
Jul 28 15:34:46.591406 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:34:46.591382    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-85ncq" event={"ID":"644d231e-7d3c-4aef-a201-87266e83d526","Type":"ContainerStarted","Data":"0c9f95bf4c0ddae90cb542cc040b7f6ab83ad6d39e0ca2e647b055af906b9b6c"}
Jul 28 15:34:46.592681 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:34:46.592660    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-74d48dfc45-9c7c5" event={"ID":"f81be785-6cfa-452b-b654-3f8c54893ab1","Type":"ContainerStarted","Data":"49f764c327385e640c0a6767b8e1c088a4e1d73040ff24781fedd3de57d2cef0"}
Jul 28 15:34:46.605120 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:34:46.605081    2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-rsd4q" podStartSLOduration=17.733116241 podStartE2EDuration="29.605065788s" totalImagesPullingTime="11.871949547s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 15:34:17 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 15:34:46.604564735 +0000 UTC m=+66.789759910" watchObservedRunningTime="2026-07-28 15:34:46.605065788 +0000 UTC m=+66.790260956"
Jul 28 15:34:47.609875 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:34:47.609835    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-65444944d6-t5hmt" event={"ID":"8a2e179f-a54d-480c-84f2-d7278c611f86","Type":"ContainerStarted","Data":"d4174f4eaf2adc9aa825b13215282ef04730b409fe66d84610b2525ca89d1e04"}
Jul 28 15:34:47.610349 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:34:47.609935    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-65444944d6-t5hmt"
Jul 28 15:34:47.635204 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:34:47.633708    2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-65444944d6-t5hmt" podStartSLOduration=3.633685604 podStartE2EDuration="3.633685604s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 15:34:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 15:34:47.633523814 +0000 UTC m=+67.818718984" watchObservedRunningTime="2026-07-28 15:34:47.633685604 +0000 UTC m=+67.818880756"
Jul 28 15:34:48.438538 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:34:48.438430    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22nh9_ebf75c0d-1597-4a1c-b668-dc2a041eb35e/konnectivity-agent/0.log" limit=1024
Jul 28 15:34:48.544013 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:34:48.543915    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22nh9_ebf75c0d-1597-4a1c-b668-dc2a041eb35e/konnectivity-agent/0.log" limit=1024
Jul 28 15:34:48.635788 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:34:48.635685    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22nh9_ebf75c0d-1597-4a1c-b668-dc2a041eb35e/konnectivity-agent/0.log" limit=1024
Jul 28 15:34:48.639566 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:34:48.638444    2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-cbpwg"]
Jul 28 15:34:48.658272 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:34:48.658241    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-cbpwg"]
Jul 28 15:34:48.658442 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:34:48.658379    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-cbpwg"
Jul 28 15:34:48.662679 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:34:48.662648    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\""
Jul 28 15:34:48.662884 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:34:48.662707    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-2pcvw\""
Jul 28 15:34:48.730903 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:34:48.729422    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/d918112c-b663-4576-bf8b-3225bc38edb4-tls-certificates\") pod \"prometheus-operator-admission-webhook-7f8d78f54b-cbpwg\" (UID: \"d918112c-b663-4576-bf8b-3225bc38edb4\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-cbpwg"
Jul 28 15:34:48.830948 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:34:48.830404    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/d918112c-b663-4576-bf8b-3225bc38edb4-tls-certificates\") pod \"prometheus-operator-admission-webhook-7f8d78f54b-cbpwg\" (UID: \"d918112c-b663-4576-bf8b-3225bc38edb4\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-cbpwg"
Jul 28 15:34:48.830948 ip-10-0-131-121 kubenswrapper[2568]: E0728 15:34:48.830583    2568 secret.go:189] Couldn't get secret openshift-monitoring/prometheus-operator-admission-webhook-tls: secret "prometheus-operator-admission-webhook-tls" not found
Jul 28 15:34:48.830948 ip-10-0-131-121 kubenswrapper[2568]: E0728 15:34:48.830651    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d918112c-b663-4576-bf8b-3225bc38edb4-tls-certificates podName:d918112c-b663-4576-bf8b-3225bc38edb4 nodeName:}" failed. No retries permitted until 2026-07-28 15:34:49.330628366 +0000 UTC m=+69.515823520 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certificates" (UniqueName: "kubernetes.io/secret/d918112c-b663-4576-bf8b-3225bc38edb4-tls-certificates") pod "prometheus-operator-admission-webhook-7f8d78f54b-cbpwg" (UID: "d918112c-b663-4576-bf8b-3225bc38edb4") : secret "prometheus-operator-admission-webhook-tls" not found
Jul 28 15:34:49.335946 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:34:49.335285    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/d918112c-b663-4576-bf8b-3225bc38edb4-tls-certificates\") pod \"prometheus-operator-admission-webhook-7f8d78f54b-cbpwg\" (UID: \"d918112c-b663-4576-bf8b-3225bc38edb4\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-cbpwg"
Jul 28 15:34:49.335946 ip-10-0-131-121 kubenswrapper[2568]: E0728 15:34:49.335498    2568 secret.go:189] Couldn't get secret openshift-monitoring/prometheus-operator-admission-webhook-tls: secret "prometheus-operator-admission-webhook-tls" not found
Jul 28 15:34:49.335946 ip-10-0-131-121 kubenswrapper[2568]: E0728 15:34:49.335565    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d918112c-b663-4576-bf8b-3225bc38edb4-tls-certificates podName:d918112c-b663-4576-bf8b-3225bc38edb4 nodeName:}" failed. No retries permitted until 2026-07-28 15:34:50.335544988 +0000 UTC m=+70.520740148 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "tls-certificates" (UniqueName: "kubernetes.io/secret/d918112c-b663-4576-bf8b-3225bc38edb4-tls-certificates") pod "prometheus-operator-admission-webhook-7f8d78f54b-cbpwg" (UID: "d918112c-b663-4576-bf8b-3225bc38edb4") : secret "prometheus-operator-admission-webhook-tls" not found
Jul 28 15:34:50.305471 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:34:50.305401    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22nh9_ebf75c0d-1597-4a1c-b668-dc2a041eb35e/konnectivity-agent/0.log" limit=1024
Jul 28 15:34:50.346412 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:34:50.345902    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/d918112c-b663-4576-bf8b-3225bc38edb4-tls-certificates\") pod \"prometheus-operator-admission-webhook-7f8d78f54b-cbpwg\" (UID: \"d918112c-b663-4576-bf8b-3225bc38edb4\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-cbpwg"
Jul 28 15:34:50.351830 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:34:50.351750    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/d918112c-b663-4576-bf8b-3225bc38edb4-tls-certificates\") pod \"prometheus-operator-admission-webhook-7f8d78f54b-cbpwg\" (UID: \"d918112c-b663-4576-bf8b-3225bc38edb4\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-cbpwg"
Jul 28 15:34:50.475823 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:34:50.475709    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-cbpwg"
Jul 28 15:34:51.633923 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:34:51.633839    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22nh9_ebf75c0d-1597-4a1c-b668-dc2a041eb35e/konnectivity-agent/0.log" limit=1024
Jul 28 15:34:51.718534 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:34:51.718440    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22nh9_ebf75c0d-1597-4a1c-b668-dc2a041eb35e/konnectivity-agent/0.log" limit=1024
Jul 28 15:34:52.146656 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:34:52.146566    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22nh9_ebf75c0d-1597-4a1c-b668-dc2a041eb35e/konnectivity-agent/0.log" limit=1024
Jul 28 15:34:52.248538 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:34:52.248500    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22nh9_ebf75c0d-1597-4a1c-b668-dc2a041eb35e/konnectivity-agent/0.log" limit=1024
Jul 28 15:34:53.195604 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:34:53.195554    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22nh9_ebf75c0d-1597-4a1c-b668-dc2a041eb35e/konnectivity-agent/0.log" limit=1024
Jul 28 15:34:53.298744 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:34:53.298697    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22nh9_ebf75c0d-1597-4a1c-b668-dc2a041eb35e/konnectivity-agent/0.log" limit=1024
Jul 28 15:34:55.570730 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:34:55.570658    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22nh9_ebf75c0d-1597-4a1c-b668-dc2a041eb35e/konnectivity-agent/0.log" limit=1024
Jul 28 15:34:55.662854 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:34:55.662729    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-cbpwg"]
Jul 28 15:34:55.664713 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:34:55.664633    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22nh9_ebf75c0d-1597-4a1c-b668-dc2a041eb35e/konnectivity-agent/0.log" limit=1024
Jul 28 15:34:55.665020 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:34:55.664991    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-ckpr5" event={"ID":"6100b515-cfe9-47ef-b672-d698ae677bd0","Type":"ContainerStarted","Data":"8826b5c70f157cd91a9e15991ecc28ab50666b87539f79ca315e26767c1ccc7e"}
Jul 28 15:34:55.670515 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:34:55.669040    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-74c9f884b7-cd6sf" event={"ID":"39b0e13f-c4ec-48bd-9c67-99b1d4506463","Type":"ContainerStarted","Data":"1ad3c74ce761022702c5ef8ba68682ec75823022f0e702ef1e542fab8a4c5f9f"}
Jul 28 15:34:55.676767 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:34:55.676268    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-57df544f56-959nt" event={"ID":"94325de5-8c27-450e-bcfa-e709d05f0266","Type":"ContainerStarted","Data":"5f4c2398815c94ac3dbd0cfe1e42844194f31ccc12846000d26cd8fb56d45286"}
Jul 28 15:34:55.691968 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:34:55.691151    2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-console/networking-console-plugin-74c9f884b7-cd6sf" podStartSLOduration=2.799662261 podStartE2EDuration="11.691128768s" totalImagesPullingTime="8.891466507s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 15:34:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 15:34:55.691009176 +0000 UTC m=+75.876204350" watchObservedRunningTime="2026-07-28 15:34:55.691128768 +0000 UTC m=+75.876323938"
Jul 28 15:34:55.806697 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:34:55.806606    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22nh9_ebf75c0d-1597-4a1c-b668-dc2a041eb35e/konnectivity-agent/0.log" limit=1024
Jul 28 15:34:55.903280 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:34:55.903239    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22nh9_ebf75c0d-1597-4a1c-b668-dc2a041eb35e/konnectivity-agent/0.log" limit=1024
Jul 28 15:34:56.002466 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:34:56.002353    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22nh9_ebf75c0d-1597-4a1c-b668-dc2a041eb35e/konnectivity-agent/0.log" limit=1024
Jul 28 15:34:56.682625 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:34:56.682586    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-tr647" event={"ID":"3ba72ab7-7ab4-4e7f-ace4-8a6b7bbff25c","Type":"ContainerStarted","Data":"707b468470a7f6adedf477df9bc548be981048bf4c741f5f4dc6a88c6874ad47"}
Jul 28 15:34:56.685136 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:34:56.685101    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-85ncq" event={"ID":"644d231e-7d3c-4aef-a201-87266e83d526","Type":"ContainerStarted","Data":"643f161e31deb712f2d141655fc2aed461e68dcdc9210db515109f888512b576"}
Jul 28 15:34:56.685595 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:34:56.685573    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-85ncq"
Jul 28 15:34:56.687646 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:34:56.687599    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-74d48dfc45-9c7c5" event={"ID":"f81be785-6cfa-452b-b654-3f8c54893ab1","Type":"ContainerStarted","Data":"a89fd78b58ce0580be23d548a877751539262ec7d553a9cb29b2f1d83ded9672"}
Jul 28 15:34:56.689124 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:34:56.689101    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-cbpwg" event={"ID":"d918112c-b663-4576-bf8b-3225bc38edb4","Type":"ContainerStarted","Data":"da4eea5e555d0e164bc804cc318900ba174bdcf894602b27ea80a3bba5af082d"}
Jul 28 15:34:56.691266 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:34:56.691243    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-ckpr5" event={"ID":"6100b515-cfe9-47ef-b672-d698ae677bd0","Type":"ContainerStarted","Data":"d936a17660ff760d2f345ea596d7db41654eef82fd99308f8567c112c984d1ce"}
Jul 28 15:34:56.701494 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:34:56.701434    2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-tr647" podStartSLOduration=67.584718421 podStartE2EDuration="1m16.701416166s" totalImagesPullingTime="9.116697745s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 15:33:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 15:34:56.700609113 +0000 UTC m=+76.885804285" watchObservedRunningTime="2026-07-28 15:34:56.701416166 +0000 UTC m=+76.886611336"
Jul 28 15:34:56.701957 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:34:56.701924    2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-57df544f56-959nt" podStartSLOduration=3.396924127 podStartE2EDuration="12.701915211s" totalImagesPullingTime="9.304991084s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 15:34:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 15:34:55.716556896 +0000 UTC m=+75.901752066" watchObservedRunningTime="2026-07-28 15:34:56.701915211 +0000 UTC m=+76.887110381"
Jul 28 15:34:56.730239 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:34:56.730177    2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-74d48dfc45-9c7c5" podStartSLOduration=3.527268368 podStartE2EDuration="12.730157533s" totalImagesPullingTime="9.202889165s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 15:34:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 15:34:56.728864743 +0000 UTC m=+76.914059925" watchObservedRunningTime="2026-07-28 15:34:56.730157533 +0000 UTC m=+76.915352686"
Jul 28 15:34:56.751281 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:34:56.751219    2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-ckpr5" podStartSLOduration=67.744069303 podStartE2EDuration="1m16.751199049s" totalImagesPullingTime="9.007129746s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 15:33:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 15:34:56.750949053 +0000 UTC m=+76.936144223" watchObservedRunningTime="2026-07-28 15:34:56.751199049 +0000 UTC m=+76.936394218"
Jul 28 15:34:56.768501 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:34:56.768454    2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-85ncq" podStartSLOduration=67.70644409 podStartE2EDuration="1m16.768437864s" totalImagesPullingTime="9.061993774s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 15:33:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 15:34:56.766816851 +0000 UTC m=+76.952012022" watchObservedRunningTime="2026-07-28 15:34:56.768437864 +0000 UTC m=+76.953633034"
Jul 28 15:34:56.870893 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:34:56.870864    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22nh9_ebf75c0d-1597-4a1c-b668-dc2a041eb35e/konnectivity-agent/0.log" limit=1024
Jul 28 15:34:56.973419 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:34:56.973335    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22nh9_ebf75c0d-1597-4a1c-b668-dc2a041eb35e/konnectivity-agent/0.log" limit=1024
Jul 28 15:34:57.697524 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:34:57.697481    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-cbpwg" event={"ID":"d918112c-b663-4576-bf8b-3225bc38edb4","Type":"ContainerStarted","Data":"835945c61d3453b5f4dc0bee85d300652be471060a96b0be701d8db4936e5291"}
Jul 28 15:34:57.716331 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:34:57.716275    2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-cbpwg" podStartSLOduration=8.618660128 podStartE2EDuration="9.716258102s" totalImagesPullingTime="1.097597974s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 15:34:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 15:34:57.714420059 +0000 UTC m=+77.899615229" watchObservedRunningTime="2026-07-28 15:34:57.716258102 +0000 UTC m=+77.901453270"
Jul 28 15:34:58.348112 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:34:58.348065    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22nh9_ebf75c0d-1597-4a1c-b668-dc2a041eb35e/konnectivity-agent/0.log" limit=1024
Jul 28 15:34:58.449602 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:34:58.449568    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22nh9_ebf75c0d-1597-4a1c-b668-dc2a041eb35e/konnectivity-agent/0.log" limit=1024
Jul 28 15:34:58.701387 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:34:58.701305    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-cbpwg"
Jul 28 15:34:58.706869 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:34:58.706841    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-cbpwg"
Jul 28 15:34:59.666799 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:34:59.666746    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22nh9_ebf75c0d-1597-4a1c-b668-dc2a041eb35e/konnectivity-agent/0.log" limit=1024
Jul 28 15:34:59.710951 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:34:59.710919    2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-84d69f794d-662rc"]
Jul 28 15:34:59.716259 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:34:59.716234    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-84d69f794d-662rc"
Jul 28 15:34:59.719153 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:34:59.719003    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\""
Jul 28 15:34:59.719153 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:34:59.719017    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\""
Jul 28 15:34:59.719153 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:34:59.719039    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-g5kf6\""
Jul 28 15:34:59.719153 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:34:59.719054    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Jul 28 15:34:59.720189 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:34:59.720170    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Jul 28 15:34:59.720272 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:34:59.720199    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Jul 28 15:34:59.725215 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:34:59.725188    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-84d69f794d-662rc"]
Jul 28 15:34:59.752572 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:34:59.752539    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22nh9_ebf75c0d-1597-4a1c-b668-dc2a041eb35e/konnectivity-agent/0.log" limit=1024
Jul 28 15:34:59.821211 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:34:59.821165    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e8dfa62b-ad30-49fa-a0ed-08aeed3407f8-metrics-client-ca\") pod \"prometheus-operator-84d69f794d-662rc\" (UID: \"e8dfa62b-ad30-49fa-a0ed-08aeed3407f8\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-662rc"
Jul 28 15:34:59.821414 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:34:59.821237    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/e8dfa62b-ad30-49fa-a0ed-08aeed3407f8-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-84d69f794d-662rc\" (UID: \"e8dfa62b-ad30-49fa-a0ed-08aeed3407f8\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-662rc"
Jul 28 15:34:59.821414 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:34:59.821288    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/e8dfa62b-ad30-49fa-a0ed-08aeed3407f8-prometheus-operator-tls\") pod \"prometheus-operator-84d69f794d-662rc\" (UID: \"e8dfa62b-ad30-49fa-a0ed-08aeed3407f8\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-662rc"
Jul 28 15:34:59.821414 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:34:59.821326    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8gl88\" (UniqueName: \"kubernetes.io/projected/e8dfa62b-ad30-49fa-a0ed-08aeed3407f8-kube-api-access-8gl88\") pod \"prometheus-operator-84d69f794d-662rc\" (UID: \"e8dfa62b-ad30-49fa-a0ed-08aeed3407f8\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-662rc"
Jul 28 15:34:59.874730 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:34:59.874693    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22nh9_ebf75c0d-1597-4a1c-b668-dc2a041eb35e/konnectivity-agent/0.log" limit=1024
Jul 28 15:34:59.922889 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:34:59.922797    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e8dfa62b-ad30-49fa-a0ed-08aeed3407f8-metrics-client-ca\") pod \"prometheus-operator-84d69f794d-662rc\" (UID: \"e8dfa62b-ad30-49fa-a0ed-08aeed3407f8\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-662rc"
Jul 28 15:34:59.922889 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:34:59.922868    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/e8dfa62b-ad30-49fa-a0ed-08aeed3407f8-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-84d69f794d-662rc\" (UID: \"e8dfa62b-ad30-49fa-a0ed-08aeed3407f8\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-662rc"
Jul 28 15:34:59.923124 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:34:59.922913    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/e8dfa62b-ad30-49fa-a0ed-08aeed3407f8-prometheus-operator-tls\") pod \"prometheus-operator-84d69f794d-662rc\" (UID: \"e8dfa62b-ad30-49fa-a0ed-08aeed3407f8\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-662rc"
Jul 28 15:34:59.923124 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:34:59.923060    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8gl88\" (UniqueName: \"kubernetes.io/projected/e8dfa62b-ad30-49fa-a0ed-08aeed3407f8-kube-api-access-8gl88\") pod \"prometheus-operator-84d69f794d-662rc\" (UID: \"e8dfa62b-ad30-49fa-a0ed-08aeed3407f8\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-662rc"
Jul 28 15:34:59.923868 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:34:59.923844    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e8dfa62b-ad30-49fa-a0ed-08aeed3407f8-metrics-client-ca\") pod \"prometheus-operator-84d69f794d-662rc\" (UID: \"e8dfa62b-ad30-49fa-a0ed-08aeed3407f8\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-662rc"
Jul 28 15:34:59.926138 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:34:59.926106    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/e8dfa62b-ad30-49fa-a0ed-08aeed3407f8-prometheus-operator-tls\") pod \"prometheus-operator-84d69f794d-662rc\" (UID: \"e8dfa62b-ad30-49fa-a0ed-08aeed3407f8\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-662rc"
Jul 28 15:34:59.926578 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:34:59.926558    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/e8dfa62b-ad30-49fa-a0ed-08aeed3407f8-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-84d69f794d-662rc\" (UID: \"e8dfa62b-ad30-49fa-a0ed-08aeed3407f8\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-662rc"
Jul 28 15:34:59.932587 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:34:59.932562    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8gl88\" (UniqueName: \"kubernetes.io/projected/e8dfa62b-ad30-49fa-a0ed-08aeed3407f8-kube-api-access-8gl88\") pod \"prometheus-operator-84d69f794d-662rc\" (UID: \"e8dfa62b-ad30-49fa-a0ed-08aeed3407f8\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-662rc"
Jul 28 15:35:00.028857 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:35:00.028809    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-84d69f794d-662rc"
Jul 28 15:35:00.722631 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:35:00.722588    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22nh9_ebf75c0d-1597-4a1c-b668-dc2a041eb35e/konnectivity-agent/0.log" limit=1024
Jul 28 15:35:00.793355 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:35:00.793305    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22nh9_ebf75c0d-1597-4a1c-b668-dc2a041eb35e/konnectivity-agent/0.log" limit=1024
Jul 28 15:35:02.154629 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:35:02.154497    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22nh9_ebf75c0d-1597-4a1c-b668-dc2a041eb35e/konnectivity-agent/0.log" limit=1024
Jul 28 15:35:02.215757 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:35:02.215705    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22nh9_ebf75c0d-1597-4a1c-b668-dc2a041eb35e/konnectivity-agent/0.log" limit=1024
Jul 28 15:35:02.430196 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:35:02.430122    2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-74d48dfc45-9c7c5"]
Jul 28 15:35:03.263472 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:35:03.263437    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22nh9_ebf75c0d-1597-4a1c-b668-dc2a041eb35e/konnectivity-agent/0.log" limit=1024
Jul 28 15:35:03.449511 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:35:03.449475    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22nh9_ebf75c0d-1597-4a1c-b668-dc2a041eb35e/konnectivity-agent/0.log" limit=1024
Jul 28 15:35:03.547559 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:35:03.547475    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22nh9_ebf75c0d-1597-4a1c-b668-dc2a041eb35e/konnectivity-agent/0.log" limit=1024
Jul 28 15:35:03.612941 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:35:03.612898    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22nh9_ebf75c0d-1597-4a1c-b668-dc2a041eb35e/konnectivity-agent/0.log" limit=1024
Jul 28 15:35:04.368283 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:35:04.368246    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22nh9_ebf75c0d-1597-4a1c-b668-dc2a041eb35e/konnectivity-agent/0.log" limit=1024
Jul 28 15:35:04.469101 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:35:04.469061    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22nh9_ebf75c0d-1597-4a1c-b668-dc2a041eb35e/konnectivity-agent/0.log" limit=1024
Jul 28 15:35:04.503507 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:35:04.503470    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-74d48dfc45-9c7c5"
Jul 28 15:35:05.788998 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:35:05.788915    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-84d69f794d-662rc"]
Jul 28 15:35:05.791560 ip-10-0-131-121 kubenswrapper[2568]: W0728 15:35:05.791533    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode8dfa62b_ad30_49fa_a0ed_08aeed3407f8.slice/crio-1dbb0922c2adead214e6babfa555d101ecdf24aabebce110d1b6a47d6b0b25ae WatchSource:0}: Error finding container 1dbb0922c2adead214e6babfa555d101ecdf24aabebce110d1b6a47d6b0b25ae: Status 404 returned error can't find the container with id 1dbb0922c2adead214e6babfa555d101ecdf24aabebce110d1b6a47d6b0b25ae
Jul 28 15:35:06.735897 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:35:06.735804    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-84d69f794d-662rc" event={"ID":"e8dfa62b-ad30-49fa-a0ed-08aeed3407f8","Type":"ContainerStarted","Data":"1dbb0922c2adead214e6babfa555d101ecdf24aabebce110d1b6a47d6b0b25ae"}
Jul 28 15:35:06.737688 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:35:06.737647    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-7c97dc4555-x5gtq" event={"ID":"1e75cd2a-7fb1-4d78-aef3-523a82c5dde6","Type":"ContainerStarted","Data":"e1517e1d3de9e1c46f762c239d1bea793526332d38588127cdb2b2ee84cde81b"}
Jul 28 15:35:06.737999 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:35:06.737880    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-7c97dc4555-x5gtq"
Jul 28 15:35:06.754707 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:35:06.754675    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-7c97dc4555-x5gtq"
Jul 28 15:35:06.775810 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:35:06.775612    2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/downloads-7c97dc4555-x5gtq" podStartSLOduration=3.289012745 podStartE2EDuration="22.775591872s" totalImagesPullingTime="19.486579127s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 15:34:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 15:35:06.757956981 +0000 UTC m=+86.943152151" watchObservedRunningTime="2026-07-28 15:35:06.775591872 +0000 UTC m=+86.960787043"
Jul 28 15:35:07.565916 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:35:07.565861    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22nh9_ebf75c0d-1597-4a1c-b668-dc2a041eb35e/konnectivity-agent/0.log" limit=1024
Jul 28 15:35:07.666708 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:35:07.666666    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22nh9_ebf75c0d-1597-4a1c-b668-dc2a041eb35e/konnectivity-agent/0.log" limit=1024
Jul 28 15:35:07.738751 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:35:07.738684    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22nh9_ebf75c0d-1597-4a1c-b668-dc2a041eb35e/konnectivity-agent/0.log" limit=1024
Jul 28 15:35:08.191306 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:35:08.191205    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22nh9_ebf75c0d-1597-4a1c-b668-dc2a041eb35e/konnectivity-agent/0.log" limit=1024
Jul 28 15:35:08.747742 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:35:08.747706    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-84d69f794d-662rc" event={"ID":"e8dfa62b-ad30-49fa-a0ed-08aeed3407f8","Type":"ContainerStarted","Data":"e1ab3e39d466c875683463f96ad97c3d27ac5287edd151f1dc853dbaab46d7ee"}
Jul 28 15:35:08.748294 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:35:08.747771    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-84d69f794d-662rc" event={"ID":"e8dfa62b-ad30-49fa-a0ed-08aeed3407f8","Type":"ContainerStarted","Data":"0178bf73d718d41209afe3d0dcabaf07c4581982e18a7c924b9f17214c4f50ed"}
Jul 28 15:35:08.766388 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:35:08.766329    2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-84d69f794d-662rc" podStartSLOduration=7.704326076 podStartE2EDuration="9.766308104s" totalImagesPullingTime="2.061982028s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 15:34:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 15:35:08.765423461 +0000 UTC m=+88.950618643" watchObservedRunningTime="2026-07-28 15:35:08.766308104 +0000 UTC m=+88.951503275"
Jul 28 15:35:09.636085 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:35:09.636055    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-65444944d6-t5hmt"
Jul 28 15:35:09.772759 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:35:09.772707    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22nh9_ebf75c0d-1597-4a1c-b668-dc2a041eb35e/konnectivity-agent/0.log" limit=1024
Jul 28 15:35:09.871586 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:35:09.871519    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22nh9_ebf75c0d-1597-4a1c-b668-dc2a041eb35e/konnectivity-agent/0.log" limit=1024
Jul 28 15:35:09.972856 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:35:09.972738    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22nh9_ebf75c0d-1597-4a1c-b668-dc2a041eb35e/konnectivity-agent/0.log" limit=1024
Jul 28 15:35:10.831870 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:35:10.831824    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22nh9_ebf75c0d-1597-4a1c-b668-dc2a041eb35e/konnectivity-agent/0.log" limit=1024
Jul 28 15:35:10.972492 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:35:10.972451    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22nh9_ebf75c0d-1597-4a1c-b668-dc2a041eb35e/konnectivity-agent/0.log" limit=1024
Jul 28 15:35:11.071797 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:35:11.071718    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22nh9_ebf75c0d-1597-4a1c-b668-dc2a041eb35e/konnectivity-agent/0.log" limit=1024
Jul 28 15:35:11.088166 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:35:11.088076    2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-sbtm9"]
Jul 28 15:35:11.123770 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:35:11.123735    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-sbtm9"
Jul 28 15:35:11.126492 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:35:11.126458    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Jul 28 15:35:11.126633 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:35:11.126585    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-dxmd6\""
Jul 28 15:35:11.126690 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:35:11.126655    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Jul 28 15:35:11.126690 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:35:11.126681    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Jul 28 15:35:11.168654 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:35:11.168611    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22nh9_ebf75c0d-1597-4a1c-b668-dc2a041eb35e/konnectivity-agent/0.log" limit=1024
Jul 28 15:35:11.217680 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:35:11.217647    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/eb97dc56-9614-4c55-bfad-38869c6bbbad-node-exporter-textfile\") pod \"node-exporter-sbtm9\" (UID: \"eb97dc56-9614-4c55-bfad-38869c6bbbad\") " pod="openshift-monitoring/node-exporter-sbtm9"
Jul 28 15:35:11.217904 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:35:11.217696    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/eb97dc56-9614-4c55-bfad-38869c6bbbad-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-sbtm9\" (UID: \"eb97dc56-9614-4c55-bfad-38869c6bbbad\") " pod="openshift-monitoring/node-exporter-sbtm9"
Jul 28 15:35:11.217904 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:35:11.217734    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/eb97dc56-9614-4c55-bfad-38869c6bbbad-node-exporter-accelerators-collector-config\") pod \"node-exporter-sbtm9\" (UID: \"eb97dc56-9614-4c55-bfad-38869c6bbbad\") " pod="openshift-monitoring/node-exporter-sbtm9"
Jul 28 15:35:11.217904 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:35:11.217766    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/eb97dc56-9614-4c55-bfad-38869c6bbbad-metrics-client-ca\") pod \"node-exporter-sbtm9\" (UID: \"eb97dc56-9614-4c55-bfad-38869c6bbbad\") " pod="openshift-monitoring/node-exporter-sbtm9"
Jul 28 15:35:11.217904 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:35:11.217862    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/eb97dc56-9614-4c55-bfad-38869c6bbbad-root\") pod \"node-exporter-sbtm9\" (UID: \"eb97dc56-9614-4c55-bfad-38869c6bbbad\") " pod="openshift-monitoring/node-exporter-sbtm9"
Jul 28 15:35:11.217904 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:35:11.217902    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/eb97dc56-9614-4c55-bfad-38869c6bbbad-node-exporter-wtmp\") pod \"node-exporter-sbtm9\" (UID: \"eb97dc56-9614-4c55-bfad-38869c6bbbad\") " pod="openshift-monitoring/node-exporter-sbtm9"
Jul 28 15:35:11.218196 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:35:11.217965    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/eb97dc56-9614-4c55-bfad-38869c6bbbad-node-exporter-tls\") pod \"node-exporter-sbtm9\" (UID: \"eb97dc56-9614-4c55-bfad-38869c6bbbad\") " pod="openshift-monitoring/node-exporter-sbtm9"
Jul 28 15:35:11.218196 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:35:11.217996    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/eb97dc56-9614-4c55-bfad-38869c6bbbad-sys\") pod \"node-exporter-sbtm9\" (UID: \"eb97dc56-9614-4c55-bfad-38869c6bbbad\") " pod="openshift-monitoring/node-exporter-sbtm9"
Jul 28 15:35:11.218196 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:35:11.218074    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zf9tn\" (UniqueName: \"kubernetes.io/projected/eb97dc56-9614-4c55-bfad-38869c6bbbad-kube-api-access-zf9tn\") pod \"node-exporter-sbtm9\" (UID: \"eb97dc56-9614-4c55-bfad-38869c6bbbad\") " pod="openshift-monitoring/node-exporter-sbtm9"
Jul 28 15:35:11.271332 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:35:11.271301    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22nh9_ebf75c0d-1597-4a1c-b668-dc2a041eb35e/konnectivity-agent/0.log" limit=1024
Jul 28 15:35:11.319447 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:35:11.319403    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/eb97dc56-9614-4c55-bfad-38869c6bbbad-node-exporter-tls\") pod \"node-exporter-sbtm9\" (UID: \"eb97dc56-9614-4c55-bfad-38869c6bbbad\") " pod="openshift-monitoring/node-exporter-sbtm9"
Jul 28 15:35:11.319629 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:35:11.319462    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/eb97dc56-9614-4c55-bfad-38869c6bbbad-sys\") pod \"node-exporter-sbtm9\" (UID: \"eb97dc56-9614-4c55-bfad-38869c6bbbad\") " pod="openshift-monitoring/node-exporter-sbtm9"
Jul 28 15:35:11.319629 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:35:11.319500    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zf9tn\" (UniqueName: \"kubernetes.io/projected/eb97dc56-9614-4c55-bfad-38869c6bbbad-kube-api-access-zf9tn\") pod \"node-exporter-sbtm9\" (UID: \"eb97dc56-9614-4c55-bfad-38869c6bbbad\") " pod="openshift-monitoring/node-exporter-sbtm9"
Jul 28 15:35:11.319629 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:35:11.319551    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/eb97dc56-9614-4c55-bfad-38869c6bbbad-node-exporter-textfile\") pod \"node-exporter-sbtm9\" (UID: \"eb97dc56-9614-4c55-bfad-38869c6bbbad\") " pod="openshift-monitoring/node-exporter-sbtm9"
Jul 28 15:35:11.319629 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:35:11.319586    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/eb97dc56-9614-4c55-bfad-38869c6bbbad-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-sbtm9\" (UID: \"eb97dc56-9614-4c55-bfad-38869c6bbbad\") " pod="openshift-monitoring/node-exporter-sbtm9"
Jul 28 15:35:11.319629 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:35:11.319587    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/eb97dc56-9614-4c55-bfad-38869c6bbbad-sys\") pod \"node-exporter-sbtm9\" (UID: \"eb97dc56-9614-4c55-bfad-38869c6bbbad\") " pod="openshift-monitoring/node-exporter-sbtm9"
Jul 28 15:35:11.319900 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:35:11.319655    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/eb97dc56-9614-4c55-bfad-38869c6bbbad-node-exporter-accelerators-collector-config\") pod \"node-exporter-sbtm9\" (UID: \"eb97dc56-9614-4c55-bfad-38869c6bbbad\") " pod="openshift-monitoring/node-exporter-sbtm9"
Jul 28 15:35:11.319900 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:35:11.319695    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/eb97dc56-9614-4c55-bfad-38869c6bbbad-metrics-client-ca\") pod \"node-exporter-sbtm9\" (UID: \"eb97dc56-9614-4c55-bfad-38869c6bbbad\") " pod="openshift-monitoring/node-exporter-sbtm9"
Jul 28 15:35:11.319900 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:35:11.319731    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/eb97dc56-9614-4c55-bfad-38869c6bbbad-root\") pod \"node-exporter-sbtm9\" (UID: \"eb97dc56-9614-4c55-bfad-38869c6bbbad\") " pod="openshift-monitoring/node-exporter-sbtm9"
Jul 28 15:35:11.319900 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:35:11.319753    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/eb97dc56-9614-4c55-bfad-38869c6bbbad-node-exporter-wtmp\") pod \"node-exporter-sbtm9\" (UID: \"eb97dc56-9614-4c55-bfad-38869c6bbbad\") " pod="openshift-monitoring/node-exporter-sbtm9"
Jul 28 15:35:11.320089 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:35:11.319927    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/eb97dc56-9614-4c55-bfad-38869c6bbbad-node-exporter-wtmp\") pod \"node-exporter-sbtm9\" (UID: \"eb97dc56-9614-4c55-bfad-38869c6bbbad\") " pod="openshift-monitoring/node-exporter-sbtm9"
Jul 28 15:35:11.320089 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:35:11.320039    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/eb97dc56-9614-4c55-bfad-38869c6bbbad-root\") pod \"node-exporter-sbtm9\" (UID: \"eb97dc56-9614-4c55-bfad-38869c6bbbad\") " pod="openshift-monitoring/node-exporter-sbtm9"
Jul 28 15:35:11.320351 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:35:11.320323    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/eb97dc56-9614-4c55-bfad-38869c6bbbad-node-exporter-textfile\") pod \"node-exporter-sbtm9\" (UID: \"eb97dc56-9614-4c55-bfad-38869c6bbbad\") " pod="openshift-monitoring/node-exporter-sbtm9"
Jul 28 15:35:11.320461 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:35:11.320414    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/eb97dc56-9614-4c55-bfad-38869c6bbbad-node-exporter-accelerators-collector-config\") pod \"node-exporter-sbtm9\" (UID: \"eb97dc56-9614-4c55-bfad-38869c6bbbad\") " pod="openshift-monitoring/node-exporter-sbtm9"
Jul 28 15:35:11.321173 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:35:11.321097    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/eb97dc56-9614-4c55-bfad-38869c6bbbad-metrics-client-ca\") pod \"node-exporter-sbtm9\" (UID: \"eb97dc56-9614-4c55-bfad-38869c6bbbad\") " pod="openshift-monitoring/node-exporter-sbtm9"
Jul 28 15:35:11.322804 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:35:11.322749    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/eb97dc56-9614-4c55-bfad-38869c6bbbad-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-sbtm9\" (UID: \"eb97dc56-9614-4c55-bfad-38869c6bbbad\") " pod="openshift-monitoring/node-exporter-sbtm9"
Jul 28 15:35:11.323414 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:35:11.323394    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/eb97dc56-9614-4c55-bfad-38869c6bbbad-node-exporter-tls\") pod \"node-exporter-sbtm9\" (UID: \"eb97dc56-9614-4c55-bfad-38869c6bbbad\") " pod="openshift-monitoring/node-exporter-sbtm9"
Jul 28 15:35:11.328985 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:35:11.328958    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zf9tn\" (UniqueName: \"kubernetes.io/projected/eb97dc56-9614-4c55-bfad-38869c6bbbad-kube-api-access-zf9tn\") pod \"node-exporter-sbtm9\" (UID: \"eb97dc56-9614-4c55-bfad-38869c6bbbad\") " pod="openshift-monitoring/node-exporter-sbtm9"
Jul 28 15:35:11.373260 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:35:11.373178    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22nh9_ebf75c0d-1597-4a1c-b668-dc2a041eb35e/konnectivity-agent/0.log" limit=1024
Jul 28 15:35:11.437314 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:35:11.437278    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-sbtm9"
Jul 28 15:35:11.471303 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:35:11.470872    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22nh9_ebf75c0d-1597-4a1c-b668-dc2a041eb35e/konnectivity-agent/0.log" limit=1024
Jul 28 15:35:11.471303 ip-10-0-131-121 kubenswrapper[2568]: W0728 15:35:11.471045    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeb97dc56_9614_4c55_bfad_38869c6bbbad.slice/crio-b99f8f62fcd8de0f38a9a14153629c3361bfebe081b0565db69c497d15c742b9 WatchSource:0}: Error finding container b99f8f62fcd8de0f38a9a14153629c3361bfebe081b0565db69c497d15c742b9: Status 404 returned error can't find the container with id b99f8f62fcd8de0f38a9a14153629c3361bfebe081b0565db69c497d15c742b9
Jul 28 15:35:11.761354 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:35:11.761264    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-sbtm9" event={"ID":"eb97dc56-9614-4c55-bfad-38869c6bbbad","Type":"ContainerStarted","Data":"b99f8f62fcd8de0f38a9a14153629c3361bfebe081b0565db69c497d15c742b9"}
Jul 28 15:35:12.056426 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:35:12.056360    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22nh9_ebf75c0d-1597-4a1c-b668-dc2a041eb35e/konnectivity-agent/0.log" limit=1024
Jul 28 15:35:12.153203 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:35:12.153166    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22nh9_ebf75c0d-1597-4a1c-b668-dc2a041eb35e/konnectivity-agent/0.log" limit=1024
Jul 28 15:35:12.252395 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:35:12.252309    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22nh9_ebf75c0d-1597-4a1c-b668-dc2a041eb35e/konnectivity-agent/0.log" limit=1024
Jul 28 15:35:12.659093 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:35:12.659064    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22nh9_ebf75c0d-1597-4a1c-b668-dc2a041eb35e/konnectivity-agent/0.log" limit=1024
Jul 28 15:35:12.753886 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:35:12.753852    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22nh9_ebf75c0d-1597-4a1c-b668-dc2a041eb35e/konnectivity-agent/0.log" limit=1024
Jul 28 15:35:13.223057 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:35:13.223024    2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-867fc7464c-c4h8h"]
Jul 28 15:35:13.235924 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:35:13.235890    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-867fc7464c-c4h8h"
Jul 28 15:35:13.239007 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:35:13.238890    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-7sgh6\""
Jul 28 15:35:13.239169 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:35:13.239067    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\""
Jul 28 15:35:13.239224 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:35:13.239166    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\""
Jul 28 15:35:13.239276 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:35:13.239221    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\""
Jul 28 15:35:13.239675 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:35:13.239373    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\""
Jul 28 15:35:13.239675 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:35:13.239419    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/29c85cac-34d3-4dae-8d46-6090c978a936-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-867fc7464c-c4h8h\" (UID: \"29c85cac-34d3-4dae-8d46-6090c978a936\") " pod="openshift-monitoring/thanos-querier-867fc7464c-c4h8h"
Jul 28 15:35:13.239675 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:35:13.239454    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\""
Jul 28 15:35:13.239675 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:35:13.239466    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/29c85cac-34d3-4dae-8d46-6090c978a936-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-867fc7464c-c4h8h\" (UID: \"29c85cac-34d3-4dae-8d46-6090c978a936\") " pod="openshift-monitoring/thanos-querier-867fc7464c-c4h8h"
Jul 28 15:35:13.239675 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:35:13.239498    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/29c85cac-34d3-4dae-8d46-6090c978a936-metrics-client-ca\") pod \"thanos-querier-867fc7464c-c4h8h\" (UID: \"29c85cac-34d3-4dae-8d46-6090c978a936\") " pod="openshift-monitoring/thanos-querier-867fc7464c-c4h8h"
Jul 28 15:35:13.239675 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:35:13.239472    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-9sqa2ue242vdd\""
Jul 28 15:35:13.239675 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:35:13.239597    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/29c85cac-34d3-4dae-8d46-6090c978a936-secret-thanos-querier-tls\") pod \"thanos-querier-867fc7464c-c4h8h\" (UID: \"29c85cac-34d3-4dae-8d46-6090c978a936\") " pod="openshift-monitoring/thanos-querier-867fc7464c-c4h8h"
Jul 28 15:35:13.239675 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:35:13.239631    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/29c85cac-34d3-4dae-8d46-6090c978a936-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-867fc7464c-c4h8h\" (UID: \"29c85cac-34d3-4dae-8d46-6090c978a936\") " pod="openshift-monitoring/thanos-querier-867fc7464c-c4h8h"
Jul 28 15:35:13.239675 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:35:13.239682    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/29c85cac-34d3-4dae-8d46-6090c978a936-secret-grpc-tls\") pod \"thanos-querier-867fc7464c-c4h8h\" (UID: \"29c85cac-34d3-4dae-8d46-6090c978a936\") " pod="openshift-monitoring/thanos-querier-867fc7464c-c4h8h"
Jul 28 15:35:13.240194 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:35:13.239737    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-867fc7464c-c4h8h"]
Jul 28 15:35:13.240194 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:35:13.239750    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qmfx9\" (UniqueName: \"kubernetes.io/projected/29c85cac-34d3-4dae-8d46-6090c978a936-kube-api-access-qmfx9\") pod \"thanos-querier-867fc7464c-c4h8h\" (UID: \"29c85cac-34d3-4dae-8d46-6090c978a936\") " pod="openshift-monitoring/thanos-querier-867fc7464c-c4h8h"
Jul 28 15:35:13.240194 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:35:13.239849    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/29c85cac-34d3-4dae-8d46-6090c978a936-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-867fc7464c-c4h8h\" (UID: \"29c85cac-34d3-4dae-8d46-6090c978a936\") " pod="openshift-monitoring/thanos-querier-867fc7464c-c4h8h"
Jul 28 15:35:13.340630 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:35:13.340594    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/29c85cac-34d3-4dae-8d46-6090c978a936-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-867fc7464c-c4h8h\" (UID: \"29c85cac-34d3-4dae-8d46-6090c978a936\") " pod="openshift-monitoring/thanos-querier-867fc7464c-c4h8h"
Jul 28 15:35:13.340838 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:35:13.340638    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/29c85cac-34d3-4dae-8d46-6090c978a936-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-867fc7464c-c4h8h\" (UID: \"29c85cac-34d3-4dae-8d46-6090c978a936\") " pod="openshift-monitoring/thanos-querier-867fc7464c-c4h8h"
Jul 28 15:35:13.340838 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:35:13.340676    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/29c85cac-34d3-4dae-8d46-6090c978a936-metrics-client-ca\") pod \"thanos-querier-867fc7464c-c4h8h\" (UID: \"29c85cac-34d3-4dae-8d46-6090c978a936\") " pod="openshift-monitoring/thanos-querier-867fc7464c-c4h8h"
Jul 28 15:35:13.340838 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:35:13.340724    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/29c85cac-34d3-4dae-8d46-6090c978a936-secret-thanos-querier-tls\") pod \"thanos-querier-867fc7464c-c4h8h\" (UID: \"29c85cac-34d3-4dae-8d46-6090c978a936\") " pod="openshift-monitoring/thanos-querier-867fc7464c-c4h8h"
Jul 28 15:35:13.340838 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:35:13.340750    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/29c85cac-34d3-4dae-8d46-6090c978a936-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-867fc7464c-c4h8h\" (UID: \"29c85cac-34d3-4dae-8d46-6090c978a936\") " pod="openshift-monitoring/thanos-querier-867fc7464c-c4h8h"
Jul 28 15:35:13.340838 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:35:13.340823    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/29c85cac-34d3-4dae-8d46-6090c978a936-secret-grpc-tls\") pod \"thanos-querier-867fc7464c-c4h8h\" (UID: \"29c85cac-34d3-4dae-8d46-6090c978a936\") " pod="openshift-monitoring/thanos-querier-867fc7464c-c4h8h"
Jul 28 15:35:13.341085 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:35:13.340862    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qmfx9\" (UniqueName: \"kubernetes.io/projected/29c85cac-34d3-4dae-8d46-6090c978a936-kube-api-access-qmfx9\") pod \"thanos-querier-867fc7464c-c4h8h\" (UID: \"29c85cac-34d3-4dae-8d46-6090c978a936\") " pod="openshift-monitoring/thanos-querier-867fc7464c-c4h8h"
Jul 28 15:35:13.341085 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:35:13.340897    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/29c85cac-34d3-4dae-8d46-6090c978a936-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-867fc7464c-c4h8h\" (UID: \"29c85cac-34d3-4dae-8d46-6090c978a936\") " pod="openshift-monitoring/thanos-querier-867fc7464c-c4h8h"
Jul 28 15:35:13.341617 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:35:13.341564    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/29c85cac-34d3-4dae-8d46-6090c978a936-metrics-client-ca\") pod \"thanos-querier-867fc7464c-c4h8h\" (UID: \"29c85cac-34d3-4dae-8d46-6090c978a936\") " pod="openshift-monitoring/thanos-querier-867fc7464c-c4h8h"
Jul 28 15:35:13.344157 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:35:13.344105    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/29c85cac-34d3-4dae-8d46-6090c978a936-secret-thanos-querier-tls\") pod \"thanos-querier-867fc7464c-c4h8h\" (UID: \"29c85cac-34d3-4dae-8d46-6090c978a936\") " pod="openshift-monitoring/thanos-querier-867fc7464c-c4h8h"
Jul 28 15:35:13.344885 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:35:13.344311    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/29c85cac-34d3-4dae-8d46-6090c978a936-secret-grpc-tls\") pod \"thanos-querier-867fc7464c-c4h8h\" (UID: \"29c85cac-34d3-4dae-8d46-6090c978a936\") " pod="openshift-monitoring/thanos-querier-867fc7464c-c4h8h"
Jul 28 15:35:13.344885 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:35:13.344817    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/29c85cac-34d3-4dae-8d46-6090c978a936-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-867fc7464c-c4h8h\" (UID: \"29c85cac-34d3-4dae-8d46-6090c978a936\") " pod="openshift-monitoring/thanos-querier-867fc7464c-c4h8h"
Jul 28 15:35:13.344885 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:35:13.344825    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/29c85cac-34d3-4dae-8d46-6090c978a936-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-867fc7464c-c4h8h\" (UID: \"29c85cac-34d3-4dae-8d46-6090c978a936\") " pod="openshift-monitoring/thanos-querier-867fc7464c-c4h8h"
Jul 28 15:35:13.345138 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:35:13.345116    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/29c85cac-34d3-4dae-8d46-6090c978a936-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-867fc7464c-c4h8h\" (UID: \"29c85cac-34d3-4dae-8d46-6090c978a936\") " pod="openshift-monitoring/thanos-querier-867fc7464c-c4h8h"
Jul 28 15:35:13.345466 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:35:13.345448    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/29c85cac-34d3-4dae-8d46-6090c978a936-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-867fc7464c-c4h8h\" (UID: \"29c85cac-34d3-4dae-8d46-6090c978a936\") " pod="openshift-monitoring/thanos-querier-867fc7464c-c4h8h"
Jul 28 15:35:13.350407 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:35:13.350378    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qmfx9\" (UniqueName: \"kubernetes.io/projected/29c85cac-34d3-4dae-8d46-6090c978a936-kube-api-access-qmfx9\") pod \"thanos-querier-867fc7464c-c4h8h\" (UID: \"29c85cac-34d3-4dae-8d46-6090c978a936\") " pod="openshift-monitoring/thanos-querier-867fc7464c-c4h8h"
Jul 28 15:35:13.453501 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:35:13.453456    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22nh9_ebf75c0d-1597-4a1c-b668-dc2a041eb35e/konnectivity-agent/0.log" limit=1024
Jul 28 15:35:13.549989 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:35:13.549953    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-867fc7464c-c4h8h"
Jul 28 15:35:13.570004 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:35:13.569972    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22nh9_ebf75c0d-1597-4a1c-b668-dc2a041eb35e/konnectivity-agent/0.log" limit=1024
Jul 28 15:35:13.731741 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:35:13.731703    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-867fc7464c-c4h8h"]
Jul 28 15:35:13.735036 ip-10-0-131-121 kubenswrapper[2568]: W0728 15:35:13.735005    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod29c85cac_34d3_4dae_8d46_6090c978a936.slice/crio-911b789c5896d7ac0b1f723aac384ed0de8b4bfe7cefb37c206ca4c6423429d9 WatchSource:0}: Error finding container 911b789c5896d7ac0b1f723aac384ed0de8b4bfe7cefb37c206ca4c6423429d9: Status 404 returned error can't find the container with id 911b789c5896d7ac0b1f723aac384ed0de8b4bfe7cefb37c206ca4c6423429d9
Jul 28 15:35:13.770997 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:35:13.770957    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-867fc7464c-c4h8h" event={"ID":"29c85cac-34d3-4dae-8d46-6090c978a936","Type":"ContainerStarted","Data":"911b789c5896d7ac0b1f723aac384ed0de8b4bfe7cefb37c206ca4c6423429d9"}
Jul 28 15:35:13.772953 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:35:13.772926    2568 generic.go:356] "Generic (PLEG): container finished" podID="eb97dc56-9614-4c55-bfad-38869c6bbbad" containerID="ec0fc358d030aea625a202c1da5c7b7d52aa1a3dd1f2b212bd2feb6197e8eebe" exitCode=0
Jul 28 15:35:13.773082 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:35:13.772970    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-sbtm9" event={"ID":"eb97dc56-9614-4c55-bfad-38869c6bbbad","Type":"ContainerDied","Data":"ec0fc358d030aea625a202c1da5c7b7d52aa1a3dd1f2b212bd2feb6197e8eebe"}
Jul 28 15:35:14.028389 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:35:14.028304    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22nh9_ebf75c0d-1597-4a1c-b668-dc2a041eb35e/konnectivity-agent/0.log" limit=1024
Jul 28 15:35:14.127502 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:35:14.127469    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22nh9_ebf75c0d-1597-4a1c-b668-dc2a041eb35e/konnectivity-agent/0.log" limit=1024
Jul 28 15:35:14.226154 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:35:14.226116    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22nh9_ebf75c0d-1597-4a1c-b668-dc2a041eb35e/konnectivity-agent/0.log" limit=1024
Jul 28 15:35:14.302326 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:35:14.302242    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22nh9_ebf75c0d-1597-4a1c-b668-dc2a041eb35e/konnectivity-agent/0.log" limit=1024
Jul 28 15:35:14.781924 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:35:14.781831    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-sbtm9" event={"ID":"eb97dc56-9614-4c55-bfad-38869c6bbbad","Type":"ContainerStarted","Data":"0ba812fe762edbfcd270678ae3c5bb696beb232ba11996e2bdb06bf0deb64b7e"}
Jul 28 15:35:14.781924 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:35:14.781891    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-sbtm9" event={"ID":"eb97dc56-9614-4c55-bfad-38869c6bbbad","Type":"ContainerStarted","Data":"aa588dad8223a7d667dd9500d5a0c1f99a737e4360949a698ac98ad279c65c2f"}
Jul 28 15:35:14.804130 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:35:14.804070    2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-sbtm9" podStartSLOduration=1.7703935579999999 podStartE2EDuration="3.804051837s" totalImagesPullingTime="1.916552589s" totalInitContainerRuntime="117.10569ms" isStatefulPod=true podCreationTimestamp="2026-07-28 15:35:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 15:35:14.801989055 +0000 UTC m=+94.987184225" watchObservedRunningTime="2026-07-28 15:35:14.804051837 +0000 UTC m=+94.989247007"
Jul 28 15:35:15.134142 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:35:15.134009    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22nh9_ebf75c0d-1597-4a1c-b668-dc2a041eb35e/konnectivity-agent/0.log" limit=1024
Jul 28 15:35:15.234660 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:35:15.234573    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22nh9_ebf75c0d-1597-4a1c-b668-dc2a041eb35e/konnectivity-agent/0.log" limit=1024
Jul 28 15:35:15.467976 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:35:15.467889    2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-6f489fff9f-cxft6"]
Jul 28 15:35:15.495707 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:35:15.495672    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-6f489fff9f-cxft6"]
Jul 28 15:35:15.495907 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:35:15.495805    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-6f489fff9f-cxft6"
Jul 28 15:35:15.500347 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:35:15.500309    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\""
Jul 28 15:35:15.500841 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:35:15.500817    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\""
Jul 28 15:35:15.502555 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:35:15.501083    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-k4rlm\""
Jul 28 15:35:15.502555 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:35:15.501473    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\""
Jul 28 15:35:15.502555 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:35:15.501651    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Jul 28 15:35:15.502555 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:35:15.501846    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-c34qdbkanvkr8\""
Jul 28 15:35:15.559519 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:35:15.559480    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/150c669a-1db3-4dbe-90b0-cf0f9fb4ab15-metrics-server-audit-profiles\") pod \"metrics-server-6f489fff9f-cxft6\" (UID: \"150c669a-1db3-4dbe-90b0-cf0f9fb4ab15\") " pod="openshift-monitoring/metrics-server-6f489fff9f-cxft6"
Jul 28 15:35:15.559725 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:35:15.559534    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/150c669a-1db3-4dbe-90b0-cf0f9fb4ab15-secret-metrics-server-client-certs\") pod \"metrics-server-6f489fff9f-cxft6\" (UID: \"150c669a-1db3-4dbe-90b0-cf0f9fb4ab15\") " pod="openshift-monitoring/metrics-server-6f489fff9f-cxft6"
Jul 28 15:35:15.559725 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:35:15.559559    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/150c669a-1db3-4dbe-90b0-cf0f9fb4ab15-audit-log\") pod \"metrics-server-6f489fff9f-cxft6\" (UID: \"150c669a-1db3-4dbe-90b0-cf0f9fb4ab15\") " pod="openshift-monitoring/metrics-server-6f489fff9f-cxft6"
Jul 28 15:35:15.559725 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:35:15.559592    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/150c669a-1db3-4dbe-90b0-cf0f9fb4ab15-secret-metrics-server-tls\") pod \"metrics-server-6f489fff9f-cxft6\" (UID: \"150c669a-1db3-4dbe-90b0-cf0f9fb4ab15\") " pod="openshift-monitoring/metrics-server-6f489fff9f-cxft6"
Jul 28 15:35:15.559725 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:35:15.559646    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/150c669a-1db3-4dbe-90b0-cf0f9fb4ab15-client-ca-bundle\") pod \"metrics-server-6f489fff9f-cxft6\" (UID: \"150c669a-1db3-4dbe-90b0-cf0f9fb4ab15\") " pod="openshift-monitoring/metrics-server-6f489fff9f-cxft6"
Jul 28 15:35:15.559725 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:35:15.559683    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kvs6j\" (UniqueName: \"kubernetes.io/projected/150c669a-1db3-4dbe-90b0-cf0f9fb4ab15-kube-api-access-kvs6j\") pod \"metrics-server-6f489fff9f-cxft6\" (UID: \"150c669a-1db3-4dbe-90b0-cf0f9fb4ab15\") " pod="openshift-monitoring/metrics-server-6f489fff9f-cxft6"
Jul 28 15:35:15.559725 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:35:15.559706    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/150c669a-1db3-4dbe-90b0-cf0f9fb4ab15-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-6f489fff9f-cxft6\" (UID: \"150c669a-1db3-4dbe-90b0-cf0f9fb4ab15\") " pod="openshift-monitoring/metrics-server-6f489fff9f-cxft6"
Jul 28 15:35:15.660838 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:35:15.660800    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/150c669a-1db3-4dbe-90b0-cf0f9fb4ab15-metrics-server-audit-profiles\") pod \"metrics-server-6f489fff9f-cxft6\" (UID: \"150c669a-1db3-4dbe-90b0-cf0f9fb4ab15\") " pod="openshift-monitoring/metrics-server-6f489fff9f-cxft6"
Jul 28 15:35:15.660838 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:35:15.660851    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/150c669a-1db3-4dbe-90b0-cf0f9fb4ab15-secret-metrics-server-client-certs\") pod \"metrics-server-6f489fff9f-cxft6\" (UID: \"150c669a-1db3-4dbe-90b0-cf0f9fb4ab15\") " pod="openshift-monitoring/metrics-server-6f489fff9f-cxft6"
Jul 28 15:35:15.661098 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:35:15.660886    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/150c669a-1db3-4dbe-90b0-cf0f9fb4ab15-audit-log\") pod \"metrics-server-6f489fff9f-cxft6\" (UID: \"150c669a-1db3-4dbe-90b0-cf0f9fb4ab15\") " pod="openshift-monitoring/metrics-server-6f489fff9f-cxft6"
Jul 28 15:35:15.661098 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:35:15.660935    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/150c669a-1db3-4dbe-90b0-cf0f9fb4ab15-secret-metrics-server-tls\") pod \"metrics-server-6f489fff9f-cxft6\" (UID: \"150c669a-1db3-4dbe-90b0-cf0f9fb4ab15\") " pod="openshift-monitoring/metrics-server-6f489fff9f-cxft6"
Jul 28 15:35:15.661098 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:35:15.660962    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/150c669a-1db3-4dbe-90b0-cf0f9fb4ab15-client-ca-bundle\") pod \"metrics-server-6f489fff9f-cxft6\" (UID: \"150c669a-1db3-4dbe-90b0-cf0f9fb4ab15\") " pod="openshift-monitoring/metrics-server-6f489fff9f-cxft6"
Jul 28 15:35:15.661098 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:35:15.660986    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kvs6j\" (UniqueName: \"kubernetes.io/projected/150c669a-1db3-4dbe-90b0-cf0f9fb4ab15-kube-api-access-kvs6j\") pod \"metrics-server-6f489fff9f-cxft6\" (UID: \"150c669a-1db3-4dbe-90b0-cf0f9fb4ab15\") " pod="openshift-monitoring/metrics-server-6f489fff9f-cxft6"
Jul 28 15:35:15.661098 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:35:15.661010    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/150c669a-1db3-4dbe-90b0-cf0f9fb4ab15-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-6f489fff9f-cxft6\" (UID: \"150c669a-1db3-4dbe-90b0-cf0f9fb4ab15\") " pod="openshift-monitoring/metrics-server-6f489fff9f-cxft6"
Jul 28 15:35:15.661642 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:35:15.661602    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/150c669a-1db3-4dbe-90b0-cf0f9fb4ab15-audit-log\") pod \"metrics-server-6f489fff9f-cxft6\" (UID: \"150c669a-1db3-4dbe-90b0-cf0f9fb4ab15\") " pod="openshift-monitoring/metrics-server-6f489fff9f-cxft6"
Jul 28 15:35:15.661766 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:35:15.661662    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/150c669a-1db3-4dbe-90b0-cf0f9fb4ab15-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-6f489fff9f-cxft6\" (UID: \"150c669a-1db3-4dbe-90b0-cf0f9fb4ab15\") " pod="openshift-monitoring/metrics-server-6f489fff9f-cxft6"
Jul 28 15:35:15.661965 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:35:15.661942    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/150c669a-1db3-4dbe-90b0-cf0f9fb4ab15-metrics-server-audit-profiles\") pod \"metrics-server-6f489fff9f-cxft6\" (UID: \"150c669a-1db3-4dbe-90b0-cf0f9fb4ab15\") " pod="openshift-monitoring/metrics-server-6f489fff9f-cxft6"
Jul 28 15:35:15.664033 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:35:15.664004    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/150c669a-1db3-4dbe-90b0-cf0f9fb4ab15-secret-metrics-server-client-certs\") pod \"metrics-server-6f489fff9f-cxft6\" (UID: \"150c669a-1db3-4dbe-90b0-cf0f9fb4ab15\") " pod="openshift-monitoring/metrics-server-6f489fff9f-cxft6"
Jul 28 15:35:15.664139 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:35:15.664054    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/150c669a-1db3-4dbe-90b0-cf0f9fb4ab15-client-ca-bundle\") pod \"metrics-server-6f489fff9f-cxft6\" (UID: \"150c669a-1db3-4dbe-90b0-cf0f9fb4ab15\") " pod="openshift-monitoring/metrics-server-6f489fff9f-cxft6"
Jul 28 15:35:15.664139 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:35:15.664114    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/150c669a-1db3-4dbe-90b0-cf0f9fb4ab15-secret-metrics-server-tls\") pod \"metrics-server-6f489fff9f-cxft6\" (UID: \"150c669a-1db3-4dbe-90b0-cf0f9fb4ab15\") " pod="openshift-monitoring/metrics-server-6f489fff9f-cxft6"
Jul 28 15:35:15.674595 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:35:15.674561    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kvs6j\" (UniqueName: \"kubernetes.io/projected/150c669a-1db3-4dbe-90b0-cf0f9fb4ab15-kube-api-access-kvs6j\") pod \"metrics-server-6f489fff9f-cxft6\" (UID: \"150c669a-1db3-4dbe-90b0-cf0f9fb4ab15\") " pod="openshift-monitoring/metrics-server-6f489fff9f-cxft6"
Jul 28 15:35:15.811089 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:35:15.811054    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-6f489fff9f-cxft6"
Jul 28 15:35:15.868518 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:35:15.868484    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22nh9_ebf75c0d-1597-4a1c-b668-dc2a041eb35e/konnectivity-agent/0.log" limit=1024
Jul 28 15:35:15.943116 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:35:15.943084    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22nh9_ebf75c0d-1597-4a1c-b668-dc2a041eb35e/konnectivity-agent/0.log" limit=1024
Jul 28 15:35:15.960391 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:35:15.960363    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-6f489fff9f-cxft6"]
Jul 28 15:35:16.050712 ip-10-0-131-121 kubenswrapper[2568]: W0728 15:35:16.050683    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod150c669a_1db3_4dbe_90b0_cf0f9fb4ab15.slice/crio-8d4c18906a6524f45d47db2e74a82a052d1f971cd4e043a2a1b58ba39715c0c7 WatchSource:0}: Error finding container 8d4c18906a6524f45d47db2e74a82a052d1f971cd4e043a2a1b58ba39715c0c7: Status 404 returned error can't find the container with id 8d4c18906a6524f45d47db2e74a82a052d1f971cd4e043a2a1b58ba39715c0c7
Jul 28 15:35:16.053524 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:35:16.053501    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22nh9_ebf75c0d-1597-4a1c-b668-dc2a041eb35e/konnectivity-agent/0.log" limit=1024
Jul 28 15:35:16.156395 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:35:16.156362    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22nh9_ebf75c0d-1597-4a1c-b668-dc2a041eb35e/konnectivity-agent/0.log" limit=1024
Jul 28 15:35:16.267173 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:35:16.267147    2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-7c86964b5b-jrqrl"]
Jul 28 15:35:16.278621 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:35:16.278599    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-7c86964b5b-jrqrl"
Jul 28 15:35:16.281042 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:35:16.281022    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-9wjgd\""
Jul 28 15:35:16.281338 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:35:16.281319    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\""
Jul 28 15:35:16.281439 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:35:16.281372    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\""
Jul 28 15:35:16.281505 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:35:16.281372    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\""
Jul 28 15:35:16.281623 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:35:16.281603    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\""
Jul 28 15:35:16.281722 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:35:16.281704    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\""
Jul 28 15:35:16.287870 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:35:16.287177    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-7c86964b5b-jrqrl"]
Jul 28 15:35:16.287870 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:35:16.287854    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-8i12ta5c71j38\""
Jul 28 15:35:16.297346 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:35:16.297298    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22nh9_ebf75c0d-1597-4a1c-b668-dc2a041eb35e/konnectivity-agent/0.log" limit=1024
Jul 28 15:35:16.366568 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:35:16.366544    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/45c39b24-8978-4302-910a-5a590a422a55-federate-client-tls\") pod \"telemeter-client-7c86964b5b-jrqrl\" (UID: \"45c39b24-8978-4302-910a-5a590a422a55\") " pod="openshift-monitoring/telemeter-client-7c86964b5b-jrqrl"
Jul 28 15:35:16.366695 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:35:16.366575    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/45c39b24-8978-4302-910a-5a590a422a55-secret-telemeter-client\") pod \"telemeter-client-7c86964b5b-jrqrl\" (UID: \"45c39b24-8978-4302-910a-5a590a422a55\") " pod="openshift-monitoring/telemeter-client-7c86964b5b-jrqrl"
Jul 28 15:35:16.366695 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:35:16.366595    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4xpnw\" (UniqueName: \"kubernetes.io/projected/45c39b24-8978-4302-910a-5a590a422a55-kube-api-access-4xpnw\") pod \"telemeter-client-7c86964b5b-jrqrl\" (UID: \"45c39b24-8978-4302-910a-5a590a422a55\") " pod="openshift-monitoring/telemeter-client-7c86964b5b-jrqrl"
Jul 28 15:35:16.366695 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:35:16.366619    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/45c39b24-8978-4302-910a-5a590a422a55-telemeter-client-tls\") pod \"telemeter-client-7c86964b5b-jrqrl\" (UID: \"45c39b24-8978-4302-910a-5a590a422a55\") " pod="openshift-monitoring/telemeter-client-7c86964b5b-jrqrl"
Jul 28 15:35:16.366847 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:35:16.366700    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/45c39b24-8978-4302-910a-5a590a422a55-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-7c86964b5b-jrqrl\" (UID: \"45c39b24-8978-4302-910a-5a590a422a55\") " pod="openshift-monitoring/telemeter-client-7c86964b5b-jrqrl"
Jul 28 15:35:16.366847 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:35:16.366752    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/45c39b24-8978-4302-910a-5a590a422a55-serving-certs-ca-bundle\") pod \"telemeter-client-7c86964b5b-jrqrl\" (UID: \"45c39b24-8978-4302-910a-5a590a422a55\") " pod="openshift-monitoring/telemeter-client-7c86964b5b-jrqrl"
Jul 28 15:35:16.366922 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:35:16.366843    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/45c39b24-8978-4302-910a-5a590a422a55-telemeter-trusted-ca-bundle\") pod \"telemeter-client-7c86964b5b-jrqrl\" (UID: \"45c39b24-8978-4302-910a-5a590a422a55\") " pod="openshift-monitoring/telemeter-client-7c86964b5b-jrqrl"
Jul 28 15:35:16.366922 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:35:16.366876    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/45c39b24-8978-4302-910a-5a590a422a55-metrics-client-ca\") pod \"telemeter-client-7c86964b5b-jrqrl\" (UID: \"45c39b24-8978-4302-910a-5a590a422a55\") " pod="openshift-monitoring/telemeter-client-7c86964b5b-jrqrl"
Jul 28 15:35:16.468026 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:35:16.467990    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/45c39b24-8978-4302-910a-5a590a422a55-federate-client-tls\") pod \"telemeter-client-7c86964b5b-jrqrl\" (UID: \"45c39b24-8978-4302-910a-5a590a422a55\") " pod="openshift-monitoring/telemeter-client-7c86964b5b-jrqrl"
Jul 28 15:35:16.468239 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:35:16.468033    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/45c39b24-8978-4302-910a-5a590a422a55-secret-telemeter-client\") pod \"telemeter-client-7c86964b5b-jrqrl\" (UID: \"45c39b24-8978-4302-910a-5a590a422a55\") " pod="openshift-monitoring/telemeter-client-7c86964b5b-jrqrl"
Jul 28 15:35:16.468239 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:35:16.468065    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4xpnw\" (UniqueName: \"kubernetes.io/projected/45c39b24-8978-4302-910a-5a590a422a55-kube-api-access-4xpnw\") pod \"telemeter-client-7c86964b5b-jrqrl\" (UID: \"45c39b24-8978-4302-910a-5a590a422a55\") " pod="openshift-monitoring/telemeter-client-7c86964b5b-jrqrl"
Jul 28 15:35:16.468239 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:35:16.468122    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/45c39b24-8978-4302-910a-5a590a422a55-telemeter-client-tls\") pod \"telemeter-client-7c86964b5b-jrqrl\" (UID: \"45c39b24-8978-4302-910a-5a590a422a55\") " pod="openshift-monitoring/telemeter-client-7c86964b5b-jrqrl"
Jul 28 15:35:16.468239 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:35:16.468186    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/45c39b24-8978-4302-910a-5a590a422a55-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-7c86964b5b-jrqrl\" (UID: \"45c39b24-8978-4302-910a-5a590a422a55\") " pod="openshift-monitoring/telemeter-client-7c86964b5b-jrqrl"
Jul 28 15:35:16.468239 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:35:16.468220    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/45c39b24-8978-4302-910a-5a590a422a55-serving-certs-ca-bundle\") pod \"telemeter-client-7c86964b5b-jrqrl\" (UID: \"45c39b24-8978-4302-910a-5a590a422a55\") " pod="openshift-monitoring/telemeter-client-7c86964b5b-jrqrl"
Jul 28 15:35:16.468489 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:35:16.468273    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/45c39b24-8978-4302-910a-5a590a422a55-telemeter-trusted-ca-bundle\") pod \"telemeter-client-7c86964b5b-jrqrl\" (UID: \"45c39b24-8978-4302-910a-5a590a422a55\") " pod="openshift-monitoring/telemeter-client-7c86964b5b-jrqrl"
Jul 28 15:35:16.468489 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:35:16.468296    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/45c39b24-8978-4302-910a-5a590a422a55-metrics-client-ca\") pod \"telemeter-client-7c86964b5b-jrqrl\" (UID: \"45c39b24-8978-4302-910a-5a590a422a55\") " pod="openshift-monitoring/telemeter-client-7c86964b5b-jrqrl"
Jul 28 15:35:16.469101 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:35:16.469074    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/45c39b24-8978-4302-910a-5a590a422a55-serving-certs-ca-bundle\") pod \"telemeter-client-7c86964b5b-jrqrl\" (UID: \"45c39b24-8978-4302-910a-5a590a422a55\") " pod="openshift-monitoring/telemeter-client-7c86964b5b-jrqrl"
Jul 28 15:35:16.469219 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:35:16.469079    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/45c39b24-8978-4302-910a-5a590a422a55-metrics-client-ca\") pod \"telemeter-client-7c86964b5b-jrqrl\" (UID: \"45c39b24-8978-4302-910a-5a590a422a55\") " pod="openshift-monitoring/telemeter-client-7c86964b5b-jrqrl"
Jul 28 15:35:16.469313 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:35:16.469289    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/45c39b24-8978-4302-910a-5a590a422a55-telemeter-trusted-ca-bundle\") pod \"telemeter-client-7c86964b5b-jrqrl\" (UID: \"45c39b24-8978-4302-910a-5a590a422a55\") " pod="openshift-monitoring/telemeter-client-7c86964b5b-jrqrl"
Jul 28 15:35:16.470735 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:35:16.470713    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/45c39b24-8978-4302-910a-5a590a422a55-telemeter-client-tls\") pod \"telemeter-client-7c86964b5b-jrqrl\" (UID: \"45c39b24-8978-4302-910a-5a590a422a55\") " pod="openshift-monitoring/telemeter-client-7c86964b5b-jrqrl"
Jul 28 15:35:16.470858 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:35:16.470835    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/45c39b24-8978-4302-910a-5a590a422a55-secret-telemeter-client\") pod \"telemeter-client-7c86964b5b-jrqrl\" (UID: \"45c39b24-8978-4302-910a-5a590a422a55\") " pod="openshift-monitoring/telemeter-client-7c86964b5b-jrqrl"
Jul 28 15:35:16.471083 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:35:16.471063    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/45c39b24-8978-4302-910a-5a590a422a55-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-7c86964b5b-jrqrl\" (UID: \"45c39b24-8978-4302-910a-5a590a422a55\") " pod="openshift-monitoring/telemeter-client-7c86964b5b-jrqrl"
Jul 28 15:35:16.471134 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:35:16.471119    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/45c39b24-8978-4302-910a-5a590a422a55-federate-client-tls\") pod \"telemeter-client-7c86964b5b-jrqrl\" (UID: \"45c39b24-8978-4302-910a-5a590a422a55\") " pod="openshift-monitoring/telemeter-client-7c86964b5b-jrqrl"
Jul 28 15:35:16.478371 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:35:16.478352    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4xpnw\" (UniqueName: \"kubernetes.io/projected/45c39b24-8978-4302-910a-5a590a422a55-kube-api-access-4xpnw\") pod \"telemeter-client-7c86964b5b-jrqrl\" (UID: \"45c39b24-8978-4302-910a-5a590a422a55\") " pod="openshift-monitoring/telemeter-client-7c86964b5b-jrqrl"
Jul 28 15:35:16.591424 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:35:16.591328    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-7c86964b5b-jrqrl"
Jul 28 15:35:16.747408 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:35:16.747384    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-7c86964b5b-jrqrl"]
Jul 28 15:35:16.749652 ip-10-0-131-121 kubenswrapper[2568]: W0728 15:35:16.749624    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod45c39b24_8978_4302_910a_5a590a422a55.slice/crio-dd79a1939f7473cdb9ee078f550fd556d854b0094b6dac8096bb41448de5e5a0 WatchSource:0}: Error finding container dd79a1939f7473cdb9ee078f550fd556d854b0094b6dac8096bb41448de5e5a0: Status 404 returned error can't find the container with id dd79a1939f7473cdb9ee078f550fd556d854b0094b6dac8096bb41448de5e5a0
Jul 28 15:35:16.791362 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:35:16.791328    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-7c86964b5b-jrqrl" event={"ID":"45c39b24-8978-4302-910a-5a590a422a55","Type":"ContainerStarted","Data":"dd79a1939f7473cdb9ee078f550fd556d854b0094b6dac8096bb41448de5e5a0"}
Jul 28 15:35:16.793479 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:35:16.793451    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-867fc7464c-c4h8h" event={"ID":"29c85cac-34d3-4dae-8d46-6090c978a936","Type":"ContainerStarted","Data":"05f93280ab40a0395753aa536f5f177f19fa5b2fd63559dc1d66d693dfd05f91"}
Jul 28 15:35:16.793630 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:35:16.793514    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-867fc7464c-c4h8h" event={"ID":"29c85cac-34d3-4dae-8d46-6090c978a936","Type":"ContainerStarted","Data":"2e2b62758af7ace53c602ad2f9f75c4b185fa2059c937bc01e5ecec33d6fb423"}
Jul 28 15:35:16.793630 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:35:16.793548    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-867fc7464c-c4h8h" event={"ID":"29c85cac-34d3-4dae-8d46-6090c978a936","Type":"ContainerStarted","Data":"ed58ec45d22de2696c11848e111551441b11ebd45761c6dd64b8b15c2129e527"}
Jul 28 15:35:16.794544 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:35:16.794520    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-6f489fff9f-cxft6" event={"ID":"150c669a-1db3-4dbe-90b0-cf0f9fb4ab15","Type":"ContainerStarted","Data":"8d4c18906a6524f45d47db2e74a82a052d1f971cd4e043a2a1b58ba39715c0c7"}
Jul 28 15:35:17.375484 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:35:17.375450    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22nh9_ebf75c0d-1597-4a1c-b668-dc2a041eb35e/konnectivity-agent/0.log" limit=1024
Jul 28 15:35:18.810408 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:35:18.810219    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-867fc7464c-c4h8h" event={"ID":"29c85cac-34d3-4dae-8d46-6090c978a936","Type":"ContainerStarted","Data":"66943b14585ff7493475ea49fd946bf874a9d1faa97e53af0de4db5375c70642"}
Jul 28 15:35:18.810408 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:35:18.810352    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-867fc7464c-c4h8h" event={"ID":"29c85cac-34d3-4dae-8d46-6090c978a936","Type":"ContainerStarted","Data":"dbd1bbc9b4e2bee6723a893c8c0a7c106ace73dbc855d333b7068142d6f2b11b"}
Jul 28 15:35:18.810408 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:35:18.810398    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-867fc7464c-c4h8h" event={"ID":"29c85cac-34d3-4dae-8d46-6090c978a936","Type":"ContainerStarted","Data":"72b9c3e0de394e4be75ae333949c1173d9d9aed2189e7871e5b7292f43c60e99"}
Jul 28 15:35:18.810989 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:35:18.810449    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-867fc7464c-c4h8h"
Jul 28 15:35:18.812563 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:35:18.812533    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-6f489fff9f-cxft6" event={"ID":"150c669a-1db3-4dbe-90b0-cf0f9fb4ab15","Type":"ContainerStarted","Data":"30c8c1e7d0b33877eaf1d9dd3486ae4269294dc83ede6429c35ed74137ab03c3"}
Jul 28 15:35:18.834798 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:35:18.834685    2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-867fc7464c-c4h8h" podStartSLOduration=1.585262449 podStartE2EDuration="5.834661325s" totalImagesPullingTime="4.249398876s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 15:35:13 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 15:35:18.832877611 +0000 UTC m=+99.018072781" watchObservedRunningTime="2026-07-28 15:35:18.834661325 +0000 UTC m=+99.019856495"
Jul 28 15:35:18.855803 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:35:18.855676    2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-6f489fff9f-cxft6" podStartSLOduration=1.633849562 podStartE2EDuration="3.855651694s" totalImagesPullingTime="2.221802132s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 15:35:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 15:35:18.854681305 +0000 UTC m=+99.039876477" watchObservedRunningTime="2026-07-28 15:35:18.855651694 +0000 UTC m=+99.040846864"
Jul 28 15:35:20.822366 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:35:20.822337    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-7c86964b5b-jrqrl" event={"ID":"45c39b24-8978-4302-910a-5a590a422a55","Type":"ContainerStarted","Data":"021d97fb074856ac5e66e660e41bcaead9b2344f35dd04d6d13150b78fc115fa"}
Jul 28 15:35:22.831124 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:35:22.831089    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-7c86964b5b-jrqrl" event={"ID":"45c39b24-8978-4302-910a-5a590a422a55","Type":"ContainerStarted","Data":"847a5297653ddaec0fd102c1593b345cd001b14a24053d1a6ac29e640522ccb9"}
Jul 28 15:35:22.831507 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:35:22.831148    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-7c86964b5b-jrqrl" event={"ID":"45c39b24-8978-4302-910a-5a590a422a55","Type":"ContainerStarted","Data":"26903e4f295633f3f0efca793e955cdf2d7f30c0f72dd0f377e7d00b90599330"}
Jul 28 15:35:22.856104 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:35:22.856051    2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-7c86964b5b-jrqrl" podStartSLOduration=1.919746964 podStartE2EDuration="6.856032028s" totalImagesPullingTime="4.936285064s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 15:35:16 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 15:35:22.854391271 +0000 UTC m=+103.039586441" watchObservedRunningTime="2026-07-28 15:35:22.856032028 +0000 UTC m=+103.041227197"
Jul 28 15:35:24.823749 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:35:24.823723    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-867fc7464c-c4h8h"
Jul 28 15:35:27.450981 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:35:27.450932    2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-74d48dfc45-9c7c5" podUID="f81be785-6cfa-452b-b654-3f8c54893ab1" containerName="console" containerID="cri-o://a89fd78b58ce0580be23d548a877751539262ec7d553a9cb29b2f1d83ded9672" gracePeriod=15
Jul 28 15:35:27.725496 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:35:27.725422    2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-74d48dfc45-9c7c5"
Jul 28 15:35:27.772362 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:35:27.772319    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/f81be785-6cfa-452b-b654-3f8c54893ab1-oauth-serving-cert\") pod \"f81be785-6cfa-452b-b654-3f8c54893ab1\" (UID: \"f81be785-6cfa-452b-b654-3f8c54893ab1\") "
Jul 28 15:35:27.772534 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:35:27.772369    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/f81be785-6cfa-452b-b654-3f8c54893ab1-console-config\") pod \"f81be785-6cfa-452b-b654-3f8c54893ab1\" (UID: \"f81be785-6cfa-452b-b654-3f8c54893ab1\") "
Jul 28 15:35:27.772534 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:35:27.772464    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/f81be785-6cfa-452b-b654-3f8c54893ab1-console-serving-cert\") pod \"f81be785-6cfa-452b-b654-3f8c54893ab1\" (UID: \"f81be785-6cfa-452b-b654-3f8c54893ab1\") "
Jul 28 15:35:27.772534 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:35:27.772507    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/f81be785-6cfa-452b-b654-3f8c54893ab1-service-ca\") pod \"f81be785-6cfa-452b-b654-3f8c54893ab1\" (UID: \"f81be785-6cfa-452b-b654-3f8c54893ab1\") "
Jul 28 15:35:27.772534 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:35:27.772531    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-px687\" (UniqueName: \"kubernetes.io/projected/f81be785-6cfa-452b-b654-3f8c54893ab1-kube-api-access-px687\") pod \"f81be785-6cfa-452b-b654-3f8c54893ab1\" (UID: \"f81be785-6cfa-452b-b654-3f8c54893ab1\") "
Jul 28 15:35:27.772718 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:35:27.772606    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/f81be785-6cfa-452b-b654-3f8c54893ab1-console-oauth-config\") pod \"f81be785-6cfa-452b-b654-3f8c54893ab1\" (UID: \"f81be785-6cfa-452b-b654-3f8c54893ab1\") "
Jul 28 15:35:27.772831 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:35:27.772808    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f81be785-6cfa-452b-b654-3f8c54893ab1-console-config" (OuterVolumeSpecName: "console-config") pod "f81be785-6cfa-452b-b654-3f8c54893ab1" (UID: "f81be785-6cfa-452b-b654-3f8c54893ab1"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 15:35:27.772831 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:35:27.772815    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f81be785-6cfa-452b-b654-3f8c54893ab1-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "f81be785-6cfa-452b-b654-3f8c54893ab1" (UID: "f81be785-6cfa-452b-b654-3f8c54893ab1"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 15:35:27.772994 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:35:27.772857    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f81be785-6cfa-452b-b654-3f8c54893ab1-service-ca" (OuterVolumeSpecName: "service-ca") pod "f81be785-6cfa-452b-b654-3f8c54893ab1" (UID: "f81be785-6cfa-452b-b654-3f8c54893ab1"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 15:35:27.772994 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:35:27.772893    2568 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/f81be785-6cfa-452b-b654-3f8c54893ab1-oauth-serving-cert\") on node \"ip-10-0-131-121.ec2.internal\" DevicePath \"\""
Jul 28 15:35:27.772994 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:35:27.772910    2568 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/f81be785-6cfa-452b-b654-3f8c54893ab1-console-config\") on node \"ip-10-0-131-121.ec2.internal\" DevicePath \"\""
Jul 28 15:35:27.774837 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:35:27.774802    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f81be785-6cfa-452b-b654-3f8c54893ab1-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "f81be785-6cfa-452b-b654-3f8c54893ab1" (UID: "f81be785-6cfa-452b-b654-3f8c54893ab1"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 15:35:27.774958 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:35:27.774838    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f81be785-6cfa-452b-b654-3f8c54893ab1-kube-api-access-px687" (OuterVolumeSpecName: "kube-api-access-px687") pod "f81be785-6cfa-452b-b654-3f8c54893ab1" (UID: "f81be785-6cfa-452b-b654-3f8c54893ab1"). InnerVolumeSpecName "kube-api-access-px687". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 15:35:27.774958 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:35:27.774868    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f81be785-6cfa-452b-b654-3f8c54893ab1-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "f81be785-6cfa-452b-b654-3f8c54893ab1" (UID: "f81be785-6cfa-452b-b654-3f8c54893ab1"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 15:35:27.852673 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:35:27.852637    2568 generic.go:356] "Generic (PLEG): container finished" podID="f81be785-6cfa-452b-b654-3f8c54893ab1" containerID="a89fd78b58ce0580be23d548a877751539262ec7d553a9cb29b2f1d83ded9672" exitCode=0
Jul 28 15:35:27.852869 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:35:27.852681    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-74d48dfc45-9c7c5" event={"ID":"f81be785-6cfa-452b-b654-3f8c54893ab1","Type":"ContainerDied","Data":"a89fd78b58ce0580be23d548a877751539262ec7d553a9cb29b2f1d83ded9672"}
Jul 28 15:35:27.852869 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:35:27.852689    2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-74d48dfc45-9c7c5"
Jul 28 15:35:27.852869 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:35:27.852718    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-74d48dfc45-9c7c5" event={"ID":"f81be785-6cfa-452b-b654-3f8c54893ab1","Type":"ContainerDied","Data":"49f764c327385e640c0a6767b8e1c088a4e1d73040ff24781fedd3de57d2cef0"}
Jul 28 15:35:27.852869 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:35:27.852738    2568 scope.go:117] "RemoveContainer" containerID="a89fd78b58ce0580be23d548a877751539262ec7d553a9cb29b2f1d83ded9672"
Jul 28 15:35:27.861629 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:35:27.861610    2568 scope.go:117] "RemoveContainer" containerID="a89fd78b58ce0580be23d548a877751539262ec7d553a9cb29b2f1d83ded9672"
Jul 28 15:35:27.861950 ip-10-0-131-121 kubenswrapper[2568]: E0728 15:35:27.861930    2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a89fd78b58ce0580be23d548a877751539262ec7d553a9cb29b2f1d83ded9672\": container with ID starting with a89fd78b58ce0580be23d548a877751539262ec7d553a9cb29b2f1d83ded9672 not found: ID does not exist" containerID="a89fd78b58ce0580be23d548a877751539262ec7d553a9cb29b2f1d83ded9672"
Jul 28 15:35:27.862019 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:35:27.861962    2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a89fd78b58ce0580be23d548a877751539262ec7d553a9cb29b2f1d83ded9672"} err="failed to get container status \"a89fd78b58ce0580be23d548a877751539262ec7d553a9cb29b2f1d83ded9672\": rpc error: code = NotFound desc = could not find container \"a89fd78b58ce0580be23d548a877751539262ec7d553a9cb29b2f1d83ded9672\": container with ID starting with a89fd78b58ce0580be23d548a877751539262ec7d553a9cb29b2f1d83ded9672 not found: ID does not exist"
Jul 28 15:35:27.873545 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:35:27.873514    2568 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/f81be785-6cfa-452b-b654-3f8c54893ab1-console-oauth-config\") on node \"ip-10-0-131-121.ec2.internal\" DevicePath \"\""
Jul 28 15:35:27.873545 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:35:27.873539    2568 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/f81be785-6cfa-452b-b654-3f8c54893ab1-console-serving-cert\") on node \"ip-10-0-131-121.ec2.internal\" DevicePath \"\""
Jul 28 15:35:27.873545 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:35:27.873550    2568 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/f81be785-6cfa-452b-b654-3f8c54893ab1-service-ca\") on node \"ip-10-0-131-121.ec2.internal\" DevicePath \"\""
Jul 28 15:35:27.873820 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:35:27.873558    2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-px687\" (UniqueName: \"kubernetes.io/projected/f81be785-6cfa-452b-b654-3f8c54893ab1-kube-api-access-px687\") on node \"ip-10-0-131-121.ec2.internal\" DevicePath \"\""
Jul 28 15:35:27.874705 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:35:27.874680    2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-74d48dfc45-9c7c5"]
Jul 28 15:35:27.878938 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:35:27.878912    2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-74d48dfc45-9c7c5"]
Jul 28 15:35:28.295044 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:35:28.295007    2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f81be785-6cfa-452b-b654-3f8c54893ab1" path="/var/lib/kubelet/pods/f81be785-6cfa-452b-b654-3f8c54893ab1/volumes"
Jul 28 15:35:28.704742 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:35:28.704669    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-85ncq"
Jul 28 15:35:30.228654 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:35:30.228621    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22nh9_ebf75c0d-1597-4a1c-b668-dc2a041eb35e/konnectivity-agent/0.log" limit=1024
Jul 28 15:35:30.291740 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:35:30.291708    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22nh9_ebf75c0d-1597-4a1c-b668-dc2a041eb35e/konnectivity-agent/0.log" limit=1024
Jul 28 15:35:34.709545 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:35:34.709510    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22nh9_ebf75c0d-1597-4a1c-b668-dc2a041eb35e/konnectivity-agent/0.log" limit=1024
Jul 28 15:35:34.809336 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:35:34.809294    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22nh9_ebf75c0d-1597-4a1c-b668-dc2a041eb35e/konnectivity-agent/0.log" limit=1024
Jul 28 15:35:34.912473 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:35:34.912441    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22nh9_ebf75c0d-1597-4a1c-b668-dc2a041eb35e/konnectivity-agent/0.log" limit=1024
Jul 28 15:35:35.812122 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:35:35.812084    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/metrics-server-6f489fff9f-cxft6"
Jul 28 15:35:35.812511 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:35:35.812149    2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/metrics-server-6f489fff9f-cxft6"
Jul 28 15:35:55.816419 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:35:55.816387    2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/metrics-server-6f489fff9f-cxft6"
Jul 28 15:35:55.822501 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:35:55.822473    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-6f489fff9f-cxft6"
Jul 28 15:36:04.923579 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:36:04.923543    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22nh9_ebf75c0d-1597-4a1c-b668-dc2a041eb35e/konnectivity-agent/0.log" limit=1024
Jul 28 15:36:27.269490 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:36:27.269450    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22nh9_ebf75c0d-1597-4a1c-b668-dc2a041eb35e/konnectivity-agent/0.log" limit=1024
Jul 28 15:36:27.340716 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:36:27.340683    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22nh9_ebf75c0d-1597-4a1c-b668-dc2a041eb35e/konnectivity-agent/0.log" limit=1024
Jul 28 15:36:27.505092 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:36:27.505057    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22nh9_ebf75c0d-1597-4a1c-b668-dc2a041eb35e/konnectivity-agent/0.log" limit=1024
Jul 28 15:36:27.565677 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:36:27.565599    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22nh9_ebf75c0d-1597-4a1c-b668-dc2a041eb35e/konnectivity-agent/0.log" limit=1024
Jul 28 15:36:36.834898 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:36:36.834816    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22nh9_ebf75c0d-1597-4a1c-b668-dc2a041eb35e/konnectivity-agent/0.log" limit=1024
Jul 28 15:36:38.005118 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:36:38.005085    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22nh9_ebf75c0d-1597-4a1c-b668-dc2a041eb35e/konnectivity-agent/0.log" limit=1024
Jul 28 15:36:38.109543 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:36:38.109500    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22nh9_ebf75c0d-1597-4a1c-b668-dc2a041eb35e/konnectivity-agent/0.log" limit=1024
Jul 28 15:36:38.180399 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:36:38.180359    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22nh9_ebf75c0d-1597-4a1c-b668-dc2a041eb35e/konnectivity-agent/0.log" limit=1024
Jul 28 15:36:39.211909 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:36:39.211872    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22nh9_ebf75c0d-1597-4a1c-b668-dc2a041eb35e/konnectivity-agent/0.log" limit=1024
Jul 28 15:36:39.311470 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:36:39.311435    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22nh9_ebf75c0d-1597-4a1c-b668-dc2a041eb35e/konnectivity-agent/0.log" limit=1024
Jul 28 15:36:45.141343 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:36:45.141312    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22nh9_ebf75c0d-1597-4a1c-b668-dc2a041eb35e/konnectivity-agent/0.log" limit=1024
Jul 28 15:36:45.246059 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:36:45.246026    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22nh9_ebf75c0d-1597-4a1c-b668-dc2a041eb35e/konnectivity-agent/0.log" limit=1024
Jul 28 15:37:34.987884 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:37:34.987846    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22nh9_ebf75c0d-1597-4a1c-b668-dc2a041eb35e/konnectivity-agent/0.log" limit=1024
Jul 28 15:37:53.061520 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:37:53.061466    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22nh9_ebf75c0d-1597-4a1c-b668-dc2a041eb35e/konnectivity-agent/0.log" limit=1024
Jul 28 15:38:40.181183 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:38:40.181158    2568 kubelet.go:1648] "Image garbage collection succeeded"
Jul 28 15:38:47.293035 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:38:47.292998    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22nh9_ebf75c0d-1597-4a1c-b668-dc2a041eb35e/konnectivity-agent/0.log" limit=1024
Jul 28 15:38:48.704415 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:38:48.704349    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22nh9_ebf75c0d-1597-4a1c-b668-dc2a041eb35e/konnectivity-agent/0.log" limit=1024
Jul 28 15:39:09.757873 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:39:09.757830    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22nh9_ebf75c0d-1597-4a1c-b668-dc2a041eb35e/konnectivity-agent/0.log" limit=1024
Jul 28 15:39:09.838690 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:39:09.838659    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22nh9_ebf75c0d-1597-4a1c-b668-dc2a041eb35e/konnectivity-agent/0.log" limit=1024
Jul 28 15:39:15.112682 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:39:15.112650    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22nh9_ebf75c0d-1597-4a1c-b668-dc2a041eb35e/konnectivity-agent/0.log" limit=1024
Jul 28 15:39:15.378970 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:39:15.378880    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22nh9_ebf75c0d-1597-4a1c-b668-dc2a041eb35e/konnectivity-agent/0.log" limit=1024
Jul 28 15:39:15.597905 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:39:15.597870    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22nh9_ebf75c0d-1597-4a1c-b668-dc2a041eb35e/konnectivity-agent/0.log" limit=1024
Jul 28 15:39:15.712012 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:39:15.711918    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22nh9_ebf75c0d-1597-4a1c-b668-dc2a041eb35e/konnectivity-agent/0.log" limit=1024
Jul 28 15:39:15.722584 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:39:15.722555    2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-keda/keda-admission-56f9798bb4-n4glj"]
Jul 28 15:39:15.722879 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:39:15.722866    2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f81be785-6cfa-452b-b654-3f8c54893ab1" containerName="console"
Jul 28 15:39:15.722932 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:39:15.722881    2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="f81be785-6cfa-452b-b654-3f8c54893ab1" containerName="console"
Jul 28 15:39:15.722932 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:39:15.722928    2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="f81be785-6cfa-452b-b654-3f8c54893ab1" containerName="console"
Jul 28 15:39:15.725089 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:39:15.725074    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-admission-56f9798bb4-n4glj"
Jul 28 15:39:15.727563 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:39:15.727537    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"keda-ocp-cabundle\""
Jul 28 15:39:15.727696 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:39:15.727537    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-admission-webhooks-certs\""
Jul 28 15:39:15.727696 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:39:15.727658    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"openshift-service-ca.crt\""
Jul 28 15:39:15.727835 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:39:15.727618    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"kube-root-ca.crt\""
Jul 28 15:39:15.728619 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:39:15.728598    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-operator-dockercfg-xxg4f\""
Jul 28 15:39:15.735739 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:39:15.735715    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-admission-56f9798bb4-n4glj"]
Jul 28 15:39:15.809530 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:39:15.809490    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22nh9_ebf75c0d-1597-4a1c-b668-dc2a041eb35e/konnectivity-agent/0.log" limit=1024
Jul 28 15:39:15.856885 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:39:15.856846    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/56406774-aab1-488a-9335-37cd0aeb4499-certificates\") pod \"keda-admission-56f9798bb4-n4glj\" (UID: \"56406774-aab1-488a-9335-37cd0aeb4499\") " pod="openshift-keda/keda-admission-56f9798bb4-n4glj"
Jul 28 15:39:15.857164 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:39:15.857145    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t8ggt\" (UniqueName: \"kubernetes.io/projected/56406774-aab1-488a-9335-37cd0aeb4499-kube-api-access-t8ggt\") pod \"keda-admission-56f9798bb4-n4glj\" (UID: \"56406774-aab1-488a-9335-37cd0aeb4499\") " pod="openshift-keda/keda-admission-56f9798bb4-n4glj"
Jul 28 15:39:15.910890 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:39:15.910850    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22nh9_ebf75c0d-1597-4a1c-b668-dc2a041eb35e/konnectivity-agent/0.log" limit=1024
Jul 28 15:39:15.959214 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:39:15.959172    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t8ggt\" (UniqueName: \"kubernetes.io/projected/56406774-aab1-488a-9335-37cd0aeb4499-kube-api-access-t8ggt\") pod \"keda-admission-56f9798bb4-n4glj\" (UID: \"56406774-aab1-488a-9335-37cd0aeb4499\") " pod="openshift-keda/keda-admission-56f9798bb4-n4glj"
Jul 28 15:39:15.960009 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:39:15.959983    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/56406774-aab1-488a-9335-37cd0aeb4499-certificates\") pod \"keda-admission-56f9798bb4-n4glj\" (UID: \"56406774-aab1-488a-9335-37cd0aeb4499\") " pod="openshift-keda/keda-admission-56f9798bb4-n4glj"
Jul 28 15:39:15.960361 ip-10-0-131-121 kubenswrapper[2568]: E0728 15:39:15.960271    2568 projected.go:266] Couldn't get secret openshift-keda/keda-admission-webhooks-certs: secret "keda-admission-webhooks-certs" not found
Jul 28 15:39:15.960494 ip-10-0-131-121 kubenswrapper[2568]: E0728 15:39:15.960484    2568 projected.go:196] Error preparing data for projected volume certificates for pod openshift-keda/keda-admission-56f9798bb4-n4glj: secret "keda-admission-webhooks-certs" not found
Jul 28 15:39:15.960661 ip-10-0-131-121 kubenswrapper[2568]: E0728 15:39:15.960651    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/56406774-aab1-488a-9335-37cd0aeb4499-certificates podName:56406774-aab1-488a-9335-37cd0aeb4499 nodeName:}" failed. No retries permitted until 2026-07-28 15:39:16.460629116 +0000 UTC m=+336.645824285 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "certificates" (UniqueName: "kubernetes.io/projected/56406774-aab1-488a-9335-37cd0aeb4499-certificates") pod "keda-admission-56f9798bb4-n4glj" (UID: "56406774-aab1-488a-9335-37cd0aeb4499") : secret "keda-admission-webhooks-certs" not found
Jul 28 15:39:15.969767 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:39:15.969697    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t8ggt\" (UniqueName: \"kubernetes.io/projected/56406774-aab1-488a-9335-37cd0aeb4499-kube-api-access-t8ggt\") pod \"keda-admission-56f9798bb4-n4glj\" (UID: \"56406774-aab1-488a-9335-37cd0aeb4499\") " pod="openshift-keda/keda-admission-56f9798bb4-n4glj"
Jul 28 15:39:16.113479 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:39:16.113443    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22nh9_ebf75c0d-1597-4a1c-b668-dc2a041eb35e/konnectivity-agent/0.log" limit=1024
Jul 28 15:39:16.211269 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:39:16.211235    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22nh9_ebf75c0d-1597-4a1c-b668-dc2a041eb35e/konnectivity-agent/0.log" limit=1024
Jul 28 15:39:16.311427 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:39:16.311393    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22nh9_ebf75c0d-1597-4a1c-b668-dc2a041eb35e/konnectivity-agent/0.log" limit=1024
Jul 28 15:39:16.412162 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:39:16.412125    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22nh9_ebf75c0d-1597-4a1c-b668-dc2a041eb35e/konnectivity-agent/0.log" limit=1024
Jul 28 15:39:16.464295 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:39:16.464256    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/56406774-aab1-488a-9335-37cd0aeb4499-certificates\") pod \"keda-admission-56f9798bb4-n4glj\" (UID: \"56406774-aab1-488a-9335-37cd0aeb4499\") " pod="openshift-keda/keda-admission-56f9798bb4-n4glj"
Jul 28 15:39:16.466787 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:39:16.466750    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/56406774-aab1-488a-9335-37cd0aeb4499-certificates\") pod \"keda-admission-56f9798bb4-n4glj\" (UID: \"56406774-aab1-488a-9335-37cd0aeb4499\") " pod="openshift-keda/keda-admission-56f9798bb4-n4glj"
Jul 28 15:39:16.517256 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:39:16.517221    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22nh9_ebf75c0d-1597-4a1c-b668-dc2a041eb35e/konnectivity-agent/0.log" limit=1024
Jul 28 15:39:16.628144 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:39:16.628058    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22nh9_ebf75c0d-1597-4a1c-b668-dc2a041eb35e/konnectivity-agent/0.log" limit=1024
Jul 28 15:39:16.636974 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:39:16.636947    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-admission-56f9798bb4-n4glj"
Jul 28 15:39:16.768343 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:39:16.768322    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-admission-56f9798bb4-n4glj"]
Jul 28 15:39:16.770790 ip-10-0-131-121 kubenswrapper[2568]: W0728 15:39:16.770752    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod56406774_aab1_488a_9335_37cd0aeb4499.slice/crio-1cc51a5e08e5b9cf83c7a1edaa7aa79c5a266bfae6e92222086d019babb859f8 WatchSource:0}: Error finding container 1cc51a5e08e5b9cf83c7a1edaa7aa79c5a266bfae6e92222086d019babb859f8: Status 404 returned error can't find the container with id 1cc51a5e08e5b9cf83c7a1edaa7aa79c5a266bfae6e92222086d019babb859f8
Jul 28 15:39:16.772087 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:39:16.772069    2568 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 28 15:39:17.703741 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:39:17.703690    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-admission-56f9798bb4-n4glj" event={"ID":"56406774-aab1-488a-9335-37cd0aeb4499","Type":"ContainerStarted","Data":"1cc51a5e08e5b9cf83c7a1edaa7aa79c5a266bfae6e92222086d019babb859f8"}
Jul 28 15:39:20.717489 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:39:20.717446    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-admission-56f9798bb4-n4glj" event={"ID":"56406774-aab1-488a-9335-37cd0aeb4499","Type":"ContainerStarted","Data":"d6bb49074c98aac1d06397875b8021ddcd7323394a26230bb20c258ec8cd3bfa"}
Jul 28 15:39:20.717915 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:39:20.717558    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-keda/keda-admission-56f9798bb4-n4glj"
Jul 28 15:39:20.733539 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:39:20.733491    2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-keda/keda-admission-56f9798bb4-n4glj" podStartSLOduration=2.61001752 podStartE2EDuration="5.733472721s" totalImagesPullingTime="3.123455201s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 15:39:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 15:39:20.733200072 +0000 UTC m=+340.918395232" watchObservedRunningTime="2026-07-28 15:39:20.733472721 +0000 UTC m=+340.918667893"
Jul 28 15:39:30.980855 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:39:30.980817    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22nh9_ebf75c0d-1597-4a1c-b668-dc2a041eb35e/konnectivity-agent/0.log" limit=1024
Jul 28 15:39:41.724207 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:39:41.724134    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-keda/keda-admission-56f9798bb4-n4glj"
Jul 28 15:40:31.393471 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:40:31.393420    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22nh9_ebf75c0d-1597-4a1c-b668-dc2a041eb35e/konnectivity-agent/0.log" limit=1024
Jul 28 15:40:31.449866 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:40:31.449824    2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/llmisvc-controller-manager-84fc959947-wmbcg"]
Jul 28 15:40:31.453569 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:40:31.453541    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/llmisvc-controller-manager-84fc959947-wmbcg"
Jul 28 15:40:31.455963 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:40:31.455925    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"llmisvc-webhook-server-cert\""
Jul 28 15:40:31.455963 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:40:31.455945    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"openshift-service-ca.crt\""
Jul 28 15:40:31.456196 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:40:31.455933    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"llmisvc-controller-manager-dockercfg-v6kcd\""
Jul 28 15:40:31.457047 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:40:31.457027    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"kube-root-ca.crt\""
Jul 28 15:40:31.463300 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:40:31.463239    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/llmisvc-controller-manager-84fc959947-wmbcg"]
Jul 28 15:40:31.477727 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:40:31.477694    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22nh9_ebf75c0d-1597-4a1c-b668-dc2a041eb35e/konnectivity-agent/0.log" limit=1024
Jul 28 15:40:31.544484 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:40:31.544452    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22nh9_ebf75c0d-1597-4a1c-b668-dc2a041eb35e/konnectivity-agent/0.log" limit=1024
Jul 28 15:40:31.569587 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:40:31.569546    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b9tp8\" (UniqueName: \"kubernetes.io/projected/60525da1-3d0a-4186-be33-0f94468ebc99-kube-api-access-b9tp8\") pod \"llmisvc-controller-manager-84fc959947-wmbcg\" (UID: \"60525da1-3d0a-4186-be33-0f94468ebc99\") " pod="kserve/llmisvc-controller-manager-84fc959947-wmbcg"
Jul 28 15:40:31.569738 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:40:31.569610    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/60525da1-3d0a-4186-be33-0f94468ebc99-cert\") pod \"llmisvc-controller-manager-84fc959947-wmbcg\" (UID: \"60525da1-3d0a-4186-be33-0f94468ebc99\") " pod="kserve/llmisvc-controller-manager-84fc959947-wmbcg"
Jul 28 15:40:31.670327 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:40:31.670236    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/60525da1-3d0a-4186-be33-0f94468ebc99-cert\") pod \"llmisvc-controller-manager-84fc959947-wmbcg\" (UID: \"60525da1-3d0a-4186-be33-0f94468ebc99\") " pod="kserve/llmisvc-controller-manager-84fc959947-wmbcg"
Jul 28 15:40:31.670508 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:40:31.670325    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b9tp8\" (UniqueName: \"kubernetes.io/projected/60525da1-3d0a-4186-be33-0f94468ebc99-kube-api-access-b9tp8\") pod \"llmisvc-controller-manager-84fc959947-wmbcg\" (UID: \"60525da1-3d0a-4186-be33-0f94468ebc99\") " pod="kserve/llmisvc-controller-manager-84fc959947-wmbcg"
Jul 28 15:40:31.670508 ip-10-0-131-121 kubenswrapper[2568]: E0728 15:40:31.670391    2568 secret.go:189] Couldn't get secret kserve/llmisvc-webhook-server-cert: secret "llmisvc-webhook-server-cert" not found
Jul 28 15:40:31.670508 ip-10-0-131-121 kubenswrapper[2568]: E0728 15:40:31.670469    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/60525da1-3d0a-4186-be33-0f94468ebc99-cert podName:60525da1-3d0a-4186-be33-0f94468ebc99 nodeName:}" failed. No retries permitted until 2026-07-28 15:40:32.170452575 +0000 UTC m=+412.355647737 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/60525da1-3d0a-4186-be33-0f94468ebc99-cert") pod "llmisvc-controller-manager-84fc959947-wmbcg" (UID: "60525da1-3d0a-4186-be33-0f94468ebc99") : secret "llmisvc-webhook-server-cert" not found
Jul 28 15:40:31.679383 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:40:31.679348    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b9tp8\" (UniqueName: \"kubernetes.io/projected/60525da1-3d0a-4186-be33-0f94468ebc99-kube-api-access-b9tp8\") pod \"llmisvc-controller-manager-84fc959947-wmbcg\" (UID: \"60525da1-3d0a-4186-be33-0f94468ebc99\") " pod="kserve/llmisvc-controller-manager-84fc959947-wmbcg"
Jul 28 15:40:31.702727 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:40:31.702700    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22nh9_ebf75c0d-1597-4a1c-b668-dc2a041eb35e/konnectivity-agent/0.log" limit=1024
Jul 28 15:40:31.883761 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:40:31.883730    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22nh9_ebf75c0d-1597-4a1c-b668-dc2a041eb35e/konnectivity-agent/0.log" limit=1024
Jul 28 15:40:31.983765 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:40:31.983687    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22nh9_ebf75c0d-1597-4a1c-b668-dc2a041eb35e/konnectivity-agent/0.log" limit=1024
Jul 28 15:40:32.084401 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:40:32.084368    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22nh9_ebf75c0d-1597-4a1c-b668-dc2a041eb35e/konnectivity-agent/0.log" limit=1024
Jul 28 15:40:32.175143 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:40:32.175109    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/60525da1-3d0a-4186-be33-0f94468ebc99-cert\") pod \"llmisvc-controller-manager-84fc959947-wmbcg\" (UID: \"60525da1-3d0a-4186-be33-0f94468ebc99\") " pod="kserve/llmisvc-controller-manager-84fc959947-wmbcg"
Jul 28 15:40:32.177590 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:40:32.177566    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/60525da1-3d0a-4186-be33-0f94468ebc99-cert\") pod \"llmisvc-controller-manager-84fc959947-wmbcg\" (UID: \"60525da1-3d0a-4186-be33-0f94468ebc99\") " pod="kserve/llmisvc-controller-manager-84fc959947-wmbcg"
Jul 28 15:40:32.366500 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:40:32.366465    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/llmisvc-controller-manager-84fc959947-wmbcg"
Jul 28 15:40:32.497898 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:40:32.497870    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/llmisvc-controller-manager-84fc959947-wmbcg"]
Jul 28 15:40:32.500174 ip-10-0-131-121 kubenswrapper[2568]: W0728 15:40:32.500149    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod60525da1_3d0a_4186_be33_0f94468ebc99.slice/crio-4a618ce032292cd83547491d6b068456d33a535b39e0be9f8e5bf5b36e9d44ff WatchSource:0}: Error finding container 4a618ce032292cd83547491d6b068456d33a535b39e0be9f8e5bf5b36e9d44ff: Status 404 returned error can't find the container with id 4a618ce032292cd83547491d6b068456d33a535b39e0be9f8e5bf5b36e9d44ff
Jul 28 15:40:32.980417 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:40:32.980381    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/llmisvc-controller-manager-84fc959947-wmbcg" event={"ID":"60525da1-3d0a-4186-be33-0f94468ebc99","Type":"ContainerStarted","Data":"4a618ce032292cd83547491d6b068456d33a535b39e0be9f8e5bf5b36e9d44ff"}
Jul 28 15:40:35.302035 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:40:35.302002    2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-77cd488857-w8w89"]
Jul 28 15:40:35.305544 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:40:35.305519    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-77cd488857-w8w89"
Jul 28 15:40:35.309395 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:40:35.309364    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Jul 28 15:40:35.309944 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:40:35.309915    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Jul 28 15:40:35.310660 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:40:35.310130    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Jul 28 15:40:35.310660 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:40:35.310309    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-4ssqh\""
Jul 28 15:40:35.310660 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:40:35.310519    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Jul 28 15:40:35.314795 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:40:35.310980    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Jul 28 15:40:35.320312 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:40:35.320288    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\""
Jul 28 15:40:35.320793 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:40:35.320733    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-77cd488857-w8w89"]
Jul 28 15:40:35.355022 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:40:35.354983    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22nh9_ebf75c0d-1597-4a1c-b668-dc2a041eb35e/konnectivity-agent/0.log" limit=1024
Jul 28 15:40:35.401792 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:40:35.401733    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4f26j\" (UniqueName: \"kubernetes.io/projected/e4c44c7d-70ae-4890-81c7-939ec40e8b51-kube-api-access-4f26j\") pod \"console-77cd488857-w8w89\" (UID: \"e4c44c7d-70ae-4890-81c7-939ec40e8b51\") " pod="openshift-console/console-77cd488857-w8w89"
Jul 28 15:40:35.401992 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:40:35.401808    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/e4c44c7d-70ae-4890-81c7-939ec40e8b51-console-serving-cert\") pod \"console-77cd488857-w8w89\" (UID: \"e4c44c7d-70ae-4890-81c7-939ec40e8b51\") " pod="openshift-console/console-77cd488857-w8w89"
Jul 28 15:40:35.401992 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:40:35.401855    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/e4c44c7d-70ae-4890-81c7-939ec40e8b51-service-ca\") pod \"console-77cd488857-w8w89\" (UID: \"e4c44c7d-70ae-4890-81c7-939ec40e8b51\") " pod="openshift-console/console-77cd488857-w8w89"
Jul 28 15:40:35.401992 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:40:35.401888    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e4c44c7d-70ae-4890-81c7-939ec40e8b51-trusted-ca-bundle\") pod \"console-77cd488857-w8w89\" (UID: \"e4c44c7d-70ae-4890-81c7-939ec40e8b51\") " pod="openshift-console/console-77cd488857-w8w89"
Jul 28 15:40:35.401992 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:40:35.401922    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/e4c44c7d-70ae-4890-81c7-939ec40e8b51-oauth-serving-cert\") pod \"console-77cd488857-w8w89\" (UID: \"e4c44c7d-70ae-4890-81c7-939ec40e8b51\") " pod="openshift-console/console-77cd488857-w8w89"
Jul 28 15:40:35.401992 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:40:35.401954    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/e4c44c7d-70ae-4890-81c7-939ec40e8b51-console-oauth-config\") pod \"console-77cd488857-w8w89\" (UID: \"e4c44c7d-70ae-4890-81c7-939ec40e8b51\") " pod="openshift-console/console-77cd488857-w8w89"
Jul 28 15:40:35.401992 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:40:35.401988    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/e4c44c7d-70ae-4890-81c7-939ec40e8b51-console-config\") pod \"console-77cd488857-w8w89\" (UID: \"e4c44c7d-70ae-4890-81c7-939ec40e8b51\") " pod="openshift-console/console-77cd488857-w8w89"
Jul 28 15:40:35.406490 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:40:35.406460    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22nh9_ebf75c0d-1597-4a1c-b668-dc2a041eb35e/konnectivity-agent/0.log" limit=1024
Jul 28 15:40:35.503080 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:40:35.503023    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4f26j\" (UniqueName: \"kubernetes.io/projected/e4c44c7d-70ae-4890-81c7-939ec40e8b51-kube-api-access-4f26j\") pod \"console-77cd488857-w8w89\" (UID: \"e4c44c7d-70ae-4890-81c7-939ec40e8b51\") " pod="openshift-console/console-77cd488857-w8w89"
Jul 28 15:40:35.503080 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:40:35.503089    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/e4c44c7d-70ae-4890-81c7-939ec40e8b51-console-serving-cert\") pod \"console-77cd488857-w8w89\" (UID: \"e4c44c7d-70ae-4890-81c7-939ec40e8b51\") " pod="openshift-console/console-77cd488857-w8w89"
Jul 28 15:40:35.503340 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:40:35.503124    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/e4c44c7d-70ae-4890-81c7-939ec40e8b51-service-ca\") pod \"console-77cd488857-w8w89\" (UID: \"e4c44c7d-70ae-4890-81c7-939ec40e8b51\") " pod="openshift-console/console-77cd488857-w8w89"
Jul 28 15:40:35.503340 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:40:35.503155    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e4c44c7d-70ae-4890-81c7-939ec40e8b51-trusted-ca-bundle\") pod \"console-77cd488857-w8w89\" (UID: \"e4c44c7d-70ae-4890-81c7-939ec40e8b51\") " pod="openshift-console/console-77cd488857-w8w89"
Jul 28 15:40:35.503340 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:40:35.503189    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/e4c44c7d-70ae-4890-81c7-939ec40e8b51-oauth-serving-cert\") pod \"console-77cd488857-w8w89\" (UID: \"e4c44c7d-70ae-4890-81c7-939ec40e8b51\") " pod="openshift-console/console-77cd488857-w8w89"
Jul 28 15:40:35.503340 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:40:35.503216    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/e4c44c7d-70ae-4890-81c7-939ec40e8b51-console-oauth-config\") pod \"console-77cd488857-w8w89\" (UID: \"e4c44c7d-70ae-4890-81c7-939ec40e8b51\") " pod="openshift-console/console-77cd488857-w8w89"
Jul 28 15:40:35.503340 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:40:35.503239    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/e4c44c7d-70ae-4890-81c7-939ec40e8b51-console-config\") pod \"console-77cd488857-w8w89\" (UID: \"e4c44c7d-70ae-4890-81c7-939ec40e8b51\") " pod="openshift-console/console-77cd488857-w8w89"
Jul 28 15:40:35.504112 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:40:35.504060    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/e4c44c7d-70ae-4890-81c7-939ec40e8b51-service-ca\") pod \"console-77cd488857-w8w89\" (UID: \"e4c44c7d-70ae-4890-81c7-939ec40e8b51\") " pod="openshift-console/console-77cd488857-w8w89"
Jul 28 15:40:35.504112 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:40:35.504097    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/e4c44c7d-70ae-4890-81c7-939ec40e8b51-console-config\") pod \"console-77cd488857-w8w89\" (UID: \"e4c44c7d-70ae-4890-81c7-939ec40e8b51\") " pod="openshift-console/console-77cd488857-w8w89"
Jul 28 15:40:35.504360 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:40:35.504339    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e4c44c7d-70ae-4890-81c7-939ec40e8b51-trusted-ca-bundle\") pod \"console-77cd488857-w8w89\" (UID: \"e4c44c7d-70ae-4890-81c7-939ec40e8b51\") " pod="openshift-console/console-77cd488857-w8w89"
Jul 28 15:40:35.504731 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:40:35.504707    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/e4c44c7d-70ae-4890-81c7-939ec40e8b51-oauth-serving-cert\") pod \"console-77cd488857-w8w89\" (UID: \"e4c44c7d-70ae-4890-81c7-939ec40e8b51\") " pod="openshift-console/console-77cd488857-w8w89"
Jul 28 15:40:35.506334 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:40:35.506311    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/e4c44c7d-70ae-4890-81c7-939ec40e8b51-console-serving-cert\") pod \"console-77cd488857-w8w89\" (UID: \"e4c44c7d-70ae-4890-81c7-939ec40e8b51\") " pod="openshift-console/console-77cd488857-w8w89"
Jul 28 15:40:35.506447 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:40:35.506374    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/e4c44c7d-70ae-4890-81c7-939ec40e8b51-console-oauth-config\") pod \"console-77cd488857-w8w89\" (UID: \"e4c44c7d-70ae-4890-81c7-939ec40e8b51\") " pod="openshift-console/console-77cd488857-w8w89"
Jul 28 15:40:35.512432 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:40:35.512402    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4f26j\" (UniqueName: \"kubernetes.io/projected/e4c44c7d-70ae-4890-81c7-939ec40e8b51-kube-api-access-4f26j\") pod \"console-77cd488857-w8w89\" (UID: \"e4c44c7d-70ae-4890-81c7-939ec40e8b51\") " pod="openshift-console/console-77cd488857-w8w89"
Jul 28 15:40:35.622920 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:40:35.622805    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-77cd488857-w8w89"
Jul 28 15:40:35.878228 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:40:35.878202    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-77cd488857-w8w89"]
Jul 28 15:40:35.880707 ip-10-0-131-121 kubenswrapper[2568]: W0728 15:40:35.880679    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode4c44c7d_70ae_4890_81c7_939ec40e8b51.slice/crio-7cef78f31d71b2d59b320a8529a827749a53b77eb463a22ff388ae2e7d6041d6 WatchSource:0}: Error finding container 7cef78f31d71b2d59b320a8529a827749a53b77eb463a22ff388ae2e7d6041d6: Status 404 returned error can't find the container with id 7cef78f31d71b2d59b320a8529a827749a53b77eb463a22ff388ae2e7d6041d6
Jul 28 15:40:35.996361 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:40:35.996324    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-77cd488857-w8w89" event={"ID":"e4c44c7d-70ae-4890-81c7-939ec40e8b51","Type":"ContainerStarted","Data":"a48263cf46088751aaf7fee988dcf867ee5360a59f4de4d9f917832bffac7937"}
Jul 28 15:40:35.996557 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:40:35.996387    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-77cd488857-w8w89" event={"ID":"e4c44c7d-70ae-4890-81c7-939ec40e8b51","Type":"ContainerStarted","Data":"7cef78f31d71b2d59b320a8529a827749a53b77eb463a22ff388ae2e7d6041d6"}
Jul 28 15:40:35.997832 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:40:35.997806    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/llmisvc-controller-manager-84fc959947-wmbcg" event={"ID":"60525da1-3d0a-4186-be33-0f94468ebc99","Type":"ContainerStarted","Data":"d9626f42d0fe8576601d6b1a65948a90da6e27a173e61cd573e67456a2942132"}
Jul 28 15:40:35.997983 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:40:35.997860    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/llmisvc-controller-manager-84fc959947-wmbcg"
Jul 28 15:40:36.014326 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:40:36.014266    2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-77cd488857-w8w89" podStartSLOduration=1.014247131 podStartE2EDuration="1.014247131s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 15:40:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 15:40:36.012586967 +0000 UTC m=+416.197782136" watchObservedRunningTime="2026-07-28 15:40:36.014247131 +0000 UTC m=+416.199442302"
Jul 28 15:40:36.030706 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:40:36.030652    2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/llmisvc-controller-manager-84fc959947-wmbcg" podStartSLOduration=1.735017273 podStartE2EDuration="5.030631387s" totalImagesPullingTime="3.295614114s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 15:40:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 15:40:36.029458284 +0000 UTC m=+416.214653455" watchObservedRunningTime="2026-07-28 15:40:36.030631387 +0000 UTC m=+416.215826555"
Jul 28 15:40:41.527752 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:40:41.527692    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22nh9_ebf75c0d-1597-4a1c-b668-dc2a041eb35e/konnectivity-agent/0.log" limit=1024
Jul 28 15:40:41.581920 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:40:41.581880    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22nh9_ebf75c0d-1597-4a1c-b668-dc2a041eb35e/konnectivity-agent/0.log" limit=1024
Jul 28 15:40:43.477425 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:40:43.477384    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22nh9_ebf75c0d-1597-4a1c-b668-dc2a041eb35e/konnectivity-agent/0.log" limit=1024
Jul 28 15:40:43.530011 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:40:43.529979    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22nh9_ebf75c0d-1597-4a1c-b668-dc2a041eb35e/konnectivity-agent/0.log" limit=1024
Jul 28 15:40:45.623728 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:40:45.623692    2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-77cd488857-w8w89"
Jul 28 15:40:45.624210 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:40:45.623796    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-77cd488857-w8w89"
Jul 28 15:40:45.628661 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:40:45.628638    2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-77cd488857-w8w89"
Jul 28 15:40:46.041711 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:40:46.041684    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-77cd488857-w8w89"
Jul 28 15:40:46.179484 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:40:46.179454    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22nh9_ebf75c0d-1597-4a1c-b668-dc2a041eb35e/konnectivity-agent/0.log" limit=1024
Jul 28 15:40:46.230359 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:40:46.230312    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22nh9_ebf75c0d-1597-4a1c-b668-dc2a041eb35e/konnectivity-agent/0.log" limit=1024
Jul 28 15:40:46.288189 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:40:46.288153    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22nh9_ebf75c0d-1597-4a1c-b668-dc2a041eb35e/konnectivity-agent/0.log" limit=1024
Jul 28 15:40:48.340658 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:40:48.340623    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22nh9_ebf75c0d-1597-4a1c-b668-dc2a041eb35e/konnectivity-agent/0.log" limit=1024
Jul 28 15:41:02.157796 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:41:02.157685    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22nh9_ebf75c0d-1597-4a1c-b668-dc2a041eb35e/konnectivity-agent/0.log" limit=1024
Jul 28 15:41:07.004204 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:41:07.004171    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/llmisvc-controller-manager-84fc959947-wmbcg"
Jul 28 15:41:48.495916 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:41:48.495880    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22nh9_ebf75c0d-1597-4a1c-b668-dc2a041eb35e/konnectivity-agent/0.log" limit=1024
Jul 28 15:41:48.543122 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:41:48.543085    2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/model-serving-api-7b954ffc95-p9c4h"]
Jul 28 15:41:48.545383 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:41:48.545360    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/model-serving-api-7b954ffc95-p9c4h"
Jul 28 15:41:48.547845 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:41:48.547816    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"model-serving-api-dockercfg-5ndph\""
Jul 28 15:41:48.547845 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:41:48.547816    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"model-serving-api-tls\""
Jul 28 15:41:48.554193 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:41:48.554165    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/model-serving-api-7b954ffc95-p9c4h"]
Jul 28 15:41:48.563412 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:41:48.563381    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22nh9_ebf75c0d-1597-4a1c-b668-dc2a041eb35e/konnectivity-agent/0.log" limit=1024
Jul 28 15:41:48.626476 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:41:48.626444    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22nh9_ebf75c0d-1597-4a1c-b668-dc2a041eb35e/konnectivity-agent/0.log" limit=1024
Jul 28 15:41:48.729360 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:41:48.729328    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5f5b50cc-e574-4f74-b27e-0e5c35923b80-tls-certs\") pod \"model-serving-api-7b954ffc95-p9c4h\" (UID: \"5f5b50cc-e574-4f74-b27e-0e5c35923b80\") " pod="kserve/model-serving-api-7b954ffc95-p9c4h"
Jul 28 15:41:48.729528 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:41:48.729372    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-54bsz\" (UniqueName: \"kubernetes.io/projected/5f5b50cc-e574-4f74-b27e-0e5c35923b80-kube-api-access-54bsz\") pod \"model-serving-api-7b954ffc95-p9c4h\" (UID: \"5f5b50cc-e574-4f74-b27e-0e5c35923b80\") " pod="kserve/model-serving-api-7b954ffc95-p9c4h"
Jul 28 15:41:48.828488 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:41:48.828458    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22nh9_ebf75c0d-1597-4a1c-b668-dc2a041eb35e/konnectivity-agent/0.log" limit=1024
Jul 28 15:41:48.830501 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:41:48.830481    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5f5b50cc-e574-4f74-b27e-0e5c35923b80-tls-certs\") pod \"model-serving-api-7b954ffc95-p9c4h\" (UID: \"5f5b50cc-e574-4f74-b27e-0e5c35923b80\") " pod="kserve/model-serving-api-7b954ffc95-p9c4h"
Jul 28 15:41:48.830572 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:41:48.830526    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-54bsz\" (UniqueName: \"kubernetes.io/projected/5f5b50cc-e574-4f74-b27e-0e5c35923b80-kube-api-access-54bsz\") pod \"model-serving-api-7b954ffc95-p9c4h\" (UID: \"5f5b50cc-e574-4f74-b27e-0e5c35923b80\") " pod="kserve/model-serving-api-7b954ffc95-p9c4h"
Jul 28 15:41:48.832935 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:41:48.832905    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5f5b50cc-e574-4f74-b27e-0e5c35923b80-tls-certs\") pod \"model-serving-api-7b954ffc95-p9c4h\" (UID: \"5f5b50cc-e574-4f74-b27e-0e5c35923b80\") " pod="kserve/model-serving-api-7b954ffc95-p9c4h"
Jul 28 15:41:48.838471 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:41:48.838445    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-54bsz\" (UniqueName: \"kubernetes.io/projected/5f5b50cc-e574-4f74-b27e-0e5c35923b80-kube-api-access-54bsz\") pod \"model-serving-api-7b954ffc95-p9c4h\" (UID: \"5f5b50cc-e574-4f74-b27e-0e5c35923b80\") " pod="kserve/model-serving-api-7b954ffc95-p9c4h"
Jul 28 15:41:48.856231 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:41:48.856193    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/model-serving-api-7b954ffc95-p9c4h"
Jul 28 15:41:48.987795 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:41:48.987745    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/model-serving-api-7b954ffc95-p9c4h"]
Jul 28 15:41:48.994893 ip-10-0-131-121 kubenswrapper[2568]: W0728 15:41:48.994865    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5f5b50cc_e574_4f74_b27e_0e5c35923b80.slice/crio-91bb6598230872804772bf561a7b9154743a3efea070ff1b3901968600121b00 WatchSource:0}: Error finding container 91bb6598230872804772bf561a7b9154743a3efea070ff1b3901968600121b00: Status 404 returned error can't find the container with id 91bb6598230872804772bf561a7b9154743a3efea070ff1b3901968600121b00
Jul 28 15:41:49.161551 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:41:49.161475    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22nh9_ebf75c0d-1597-4a1c-b668-dc2a041eb35e/konnectivity-agent/0.log" limit=1024
Jul 28 15:41:49.277165 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:41:49.277129    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/model-serving-api-7b954ffc95-p9c4h" event={"ID":"5f5b50cc-e574-4f74-b27e-0e5c35923b80","Type":"ContainerStarted","Data":"91bb6598230872804772bf561a7b9154743a3efea070ff1b3901968600121b00"}
Jul 28 15:41:49.502990 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:41:49.502901    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22nh9_ebf75c0d-1597-4a1c-b668-dc2a041eb35e/konnectivity-agent/0.log" limit=1024
Jul 28 15:41:51.287470 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:41:51.287436    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/model-serving-api-7b954ffc95-p9c4h" event={"ID":"5f5b50cc-e574-4f74-b27e-0e5c35923b80","Type":"ContainerStarted","Data":"9be07a8ba645d403c8dfca6e1b0adf15edb47fc7cb88f97aa4f3765ded1c257f"}
Jul 28 15:41:51.287950 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:41:51.287543    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/model-serving-api-7b954ffc95-p9c4h"
Jul 28 15:41:51.304820 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:41:51.304729    2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/model-serving-api-7b954ffc95-p9c4h" podStartSLOduration=1.154211418 podStartE2EDuration="3.304708502s" totalImagesPullingTime="2.150497084s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 15:41:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 15:41:51.304276029 +0000 UTC m=+491.489471198" watchObservedRunningTime="2026-07-28 15:41:51.304708502 +0000 UTC m=+491.489903672"
Jul 28 15:42:02.297304 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:42:02.297269    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/model-serving-api-7b954ffc95-p9c4h"
Jul 28 15:42:07.760407 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:42:07.760362    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22nh9_ebf75c0d-1597-4a1c-b668-dc2a041eb35e/konnectivity-agent/0.log" limit=1024
Jul 28 15:42:07.815908 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:42:07.815878    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22nh9_ebf75c0d-1597-4a1c-b668-dc2a041eb35e/konnectivity-agent/0.log" limit=1024
Jul 28 15:42:53.279314 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:42:53.279233    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22nh9_ebf75c0d-1597-4a1c-b668-dc2a041eb35e/konnectivity-agent/0.log" limit=1024
Jul 28 15:42:53.483573 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:42:53.483540    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22nh9_ebf75c0d-1597-4a1c-b668-dc2a041eb35e/konnectivity-agent/0.log" limit=1024
Jul 28 15:43:23.976839 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:43:23.976801    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22nh9_ebf75c0d-1597-4a1c-b668-dc2a041eb35e/konnectivity-agent/0.log" limit=1024
Jul 28 15:43:24.077452 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:43:24.077421    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22nh9_ebf75c0d-1597-4a1c-b668-dc2a041eb35e/konnectivity-agent/0.log" limit=1024
Jul 28 15:43:37.129852 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:43:37.129757    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22nh9_ebf75c0d-1597-4a1c-b668-dc2a041eb35e/konnectivity-agent/0.log" limit=1024
Jul 28 15:43:37.318288 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:43:37.318257    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22nh9_ebf75c0d-1597-4a1c-b668-dc2a041eb35e/konnectivity-agent/0.log" limit=1024
Jul 28 15:43:40.311508 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:43:40.311476    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22nh9_ebf75c0d-1597-4a1c-b668-dc2a041eb35e/konnectivity-agent/0.log" limit=1024
Jul 28 15:44:30.488856 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:44:30.488823    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22nh9_ebf75c0d-1597-4a1c-b668-dc2a041eb35e/konnectivity-agent/0.log" limit=1024
Jul 28 15:44:32.375654 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:44:32.375617    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22nh9_ebf75c0d-1597-4a1c-b668-dc2a041eb35e/konnectivity-agent/0.log" limit=1024
Jul 28 15:44:59.815937 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:44:59.815900    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22nh9_ebf75c0d-1597-4a1c-b668-dc2a041eb35e/konnectivity-agent/0.log" limit=1024
Jul 28 15:44:59.866598 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:44:59.866562    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22nh9_ebf75c0d-1597-4a1c-b668-dc2a041eb35e/konnectivity-agent/0.log" limit=1024
Jul 28 15:45:22.409150 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:45:22.409119    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22nh9_ebf75c0d-1597-4a1c-b668-dc2a041eb35e/konnectivity-agent/0.log" limit=1024
Jul 28 15:45:22.473510 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:45:22.473475    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22nh9_ebf75c0d-1597-4a1c-b668-dc2a041eb35e/konnectivity-agent/0.log" limit=1024
Jul 28 15:45:22.544087 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:45:22.544060    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22nh9_ebf75c0d-1597-4a1c-b668-dc2a041eb35e/konnectivity-agent/0.log" limit=1024
Jul 28 15:45:22.768044 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:45:22.768012    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22nh9_ebf75c0d-1597-4a1c-b668-dc2a041eb35e/konnectivity-agent/0.log" limit=1024
Jul 28 15:45:23.159363 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:45:23.159283    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22nh9_ebf75c0d-1597-4a1c-b668-dc2a041eb35e/konnectivity-agent/0.log" limit=1024
Jul 28 15:47:02.511347 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:47:02.511256    2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/model-chainer-raw-0be34-9dd95d454-bdzdk"]
Jul 28 15:47:02.514477 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:47:02.514458    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/model-chainer-raw-0be34-9dd95d454-bdzdk"
Jul 28 15:47:02.519292 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:47:02.519157    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"model-chainer-raw-0be34-serving-cert\""
Jul 28 15:47:02.519292 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:47:02.519241    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 28 15:47:02.519292 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:47:02.519250    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"model-chainer-raw-0be34-kube-rbac-proxy-sar-config\""
Jul 28 15:47:02.519595 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:47:02.519421    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-gcwwc\""
Jul 28 15:47:02.523961 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:47:02.523932    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/model-chainer-raw-0be34-9dd95d454-bdzdk"]
Jul 28 15:47:02.536849 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:47:02.536815    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22nh9_ebf75c0d-1597-4a1c-b668-dc2a041eb35e/konnectivity-agent/0.log" limit=1024
Jul 28 15:47:02.610936 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:47:02.610887    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c8563ba5-080f-495d-a7c6-a32ca10dbf59-openshift-service-ca-bundle\") pod \"model-chainer-raw-0be34-9dd95d454-bdzdk\" (UID: \"c8563ba5-080f-495d-a7c6-a32ca10dbf59\") " pod="kserve-ci-e2e-test/model-chainer-raw-0be34-9dd95d454-bdzdk"
Jul 28 15:47:02.611114 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:47:02.610977    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c8563ba5-080f-495d-a7c6-a32ca10dbf59-proxy-tls\") pod \"model-chainer-raw-0be34-9dd95d454-bdzdk\" (UID: \"c8563ba5-080f-495d-a7c6-a32ca10dbf59\") " pod="kserve-ci-e2e-test/model-chainer-raw-0be34-9dd95d454-bdzdk"
Jul 28 15:47:02.674161 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:47:02.674123    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22nh9_ebf75c0d-1597-4a1c-b668-dc2a041eb35e/konnectivity-agent/0.log" limit=1024
Jul 28 15:47:02.711705 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:47:02.711670    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c8563ba5-080f-495d-a7c6-a32ca10dbf59-openshift-service-ca-bundle\") pod \"model-chainer-raw-0be34-9dd95d454-bdzdk\" (UID: \"c8563ba5-080f-495d-a7c6-a32ca10dbf59\") " pod="kserve-ci-e2e-test/model-chainer-raw-0be34-9dd95d454-bdzdk"
Jul 28 15:47:02.711858 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:47:02.711726    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c8563ba5-080f-495d-a7c6-a32ca10dbf59-proxy-tls\") pod \"model-chainer-raw-0be34-9dd95d454-bdzdk\" (UID: \"c8563ba5-080f-495d-a7c6-a32ca10dbf59\") " pod="kserve-ci-e2e-test/model-chainer-raw-0be34-9dd95d454-bdzdk"
Jul 28 15:47:02.712340 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:47:02.712311    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c8563ba5-080f-495d-a7c6-a32ca10dbf59-openshift-service-ca-bundle\") pod \"model-chainer-raw-0be34-9dd95d454-bdzdk\" (UID: \"c8563ba5-080f-495d-a7c6-a32ca10dbf59\") " pod="kserve-ci-e2e-test/model-chainer-raw-0be34-9dd95d454-bdzdk"
Jul 28 15:47:02.714162 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:47:02.714141    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c8563ba5-080f-495d-a7c6-a32ca10dbf59-proxy-tls\") pod \"model-chainer-raw-0be34-9dd95d454-bdzdk\" (UID: \"c8563ba5-080f-495d-a7c6-a32ca10dbf59\") " pod="kserve-ci-e2e-test/model-chainer-raw-0be34-9dd95d454-bdzdk"
Jul 28 15:47:02.826571 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:47:02.826526    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/model-chainer-raw-0be34-9dd95d454-bdzdk"
Jul 28 15:47:02.961739 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:47:02.961688    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/model-chainer-raw-0be34-9dd95d454-bdzdk"]
Jul 28 15:47:02.964588 ip-10-0-131-121 kubenswrapper[2568]: W0728 15:47:02.964546    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc8563ba5_080f_495d_a7c6_a32ca10dbf59.slice/crio-2532931bcbcc881e5386475919bb20f3f92d5fcebce543bf4e65451fc8aad8c1 WatchSource:0}: Error finding container 2532931bcbcc881e5386475919bb20f3f92d5fcebce543bf4e65451fc8aad8c1: Status 404 returned error can't find the container with id 2532931bcbcc881e5386475919bb20f3f92d5fcebce543bf4e65451fc8aad8c1
Jul 28 15:47:02.966540 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:47:02.966522    2568 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 28 15:47:03.508445 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:47:03.508407    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/model-chainer-raw-0be34-9dd95d454-bdzdk" event={"ID":"c8563ba5-080f-495d-a7c6-a32ca10dbf59","Type":"ContainerStarted","Data":"2532931bcbcc881e5386475919bb20f3f92d5fcebce543bf4e65451fc8aad8c1"}
Jul 28 15:47:05.517874 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:47:05.517834    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/model-chainer-raw-0be34-9dd95d454-bdzdk" event={"ID":"c8563ba5-080f-495d-a7c6-a32ca10dbf59","Type":"ContainerStarted","Data":"9db9810ad53fad91822805183723548d49f5d33447605e4907fafbd18468a3e5"}
Jul 28 15:47:05.518278 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:47:05.517967    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/model-chainer-raw-0be34-9dd95d454-bdzdk"
Jul 28 15:47:05.534809 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:47:05.534734    2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/model-chainer-raw-0be34-9dd95d454-bdzdk" podStartSLOduration=1.085089618 podStartE2EDuration="3.534713782s" totalImagesPullingTime="2.449624164s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 15:47:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 15:47:05.532994319 +0000 UTC m=+805.718189489" watchObservedRunningTime="2026-07-28 15:47:05.534713782 +0000 UTC m=+805.719908952"
Jul 28 15:47:11.527184 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:47:11.527154    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/model-chainer-raw-0be34-9dd95d454-bdzdk"
Jul 28 15:47:12.563642 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:47:12.563605    2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/model-chainer-raw-0be34-9dd95d454-bdzdk"]
Jul 28 15:47:12.564068 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:47:12.563822    2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/model-chainer-raw-0be34-9dd95d454-bdzdk" podUID="c8563ba5-080f-495d-a7c6-a32ca10dbf59" containerName="model-chainer-raw-0be34" containerID="cri-o://9db9810ad53fad91822805183723548d49f5d33447605e4907fafbd18468a3e5" gracePeriod=30
Jul 28 15:47:12.593445 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:47:12.593416    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22nh9_ebf75c0d-1597-4a1c-b668-dc2a041eb35e/konnectivity-agent/0.log" limit=1024
Jul 28 15:47:12.731691 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:47:12.731647    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22nh9_ebf75c0d-1597-4a1c-b668-dc2a041eb35e/konnectivity-agent/0.log" limit=1024
Jul 28 15:47:12.804661 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:47:12.804625    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22nh9_ebf75c0d-1597-4a1c-b668-dc2a041eb35e/konnectivity-agent/0.log" limit=1024
Jul 28 15:47:12.868372 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:47:12.868287    2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-c70d2-predictor-7c7cb8b49-8qmv2"]
Jul 28 15:47:12.870558 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:47:12.870533    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22nh9_ebf75c0d-1597-4a1c-b668-dc2a041eb35e/konnectivity-agent/0.log" limit=1024
Jul 28 15:47:12.872065 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:47:12.872043    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-c70d2-predictor-7c7cb8b49-8qmv2"
Jul 28 15:47:12.874298 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:47:12.874275    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 28 15:47:12.874551 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:47:12.874316    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 28 15:47:12.874675 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:47:12.874334    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-graph-raw-hpa-c70d2-predictor-serving-cert\""
Jul 28 15:47:12.874675 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:47:12.874638    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 28 15:47:12.874956 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:47:12.874939    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-graph-raw-hpa-c70d2-kube-rbac-proxy-sar-config\""
Jul 28 15:47:12.885891 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:47:12.885867    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-c70d2-predictor-7c7cb8b49-8qmv2"]
Jul 28 15:47:12.896023 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:47:12.895999    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7cc2f9f3-d61c-470e-99ec-ddaef02da3aa-kserve-provision-location\") pod \"isvc-xgboost-graph-raw-hpa-c70d2-predictor-7c7cb8b49-8qmv2\" (UID: \"7cc2f9f3-d61c-470e-99ec-ddaef02da3aa\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-c70d2-predictor-7c7cb8b49-8qmv2"
Jul 28 15:47:12.896174 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:47:12.896032    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7cc2f9f3-d61c-470e-99ec-ddaef02da3aa-cabundle-cert\") pod \"isvc-xgboost-graph-raw-hpa-c70d2-predictor-7c7cb8b49-8qmv2\" (UID: \"7cc2f9f3-d61c-470e-99ec-ddaef02da3aa\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-c70d2-predictor-7c7cb8b49-8qmv2"
Jul 28 15:47:12.896174 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:47:12.896051    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7cc2f9f3-d61c-470e-99ec-ddaef02da3aa-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-graph-raw-hpa-c70d2-predictor-7c7cb8b49-8qmv2\" (UID: \"7cc2f9f3-d61c-470e-99ec-ddaef02da3aa\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-c70d2-predictor-7c7cb8b49-8qmv2"
Jul 28 15:47:12.896174 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:47:12.896096    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7cc2f9f3-d61c-470e-99ec-ddaef02da3aa-proxy-tls\") pod \"isvc-xgboost-graph-raw-hpa-c70d2-predictor-7c7cb8b49-8qmv2\" (UID: \"7cc2f9f3-d61c-470e-99ec-ddaef02da3aa\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-c70d2-predictor-7c7cb8b49-8qmv2"
Jul 28 15:47:12.896174 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:47:12.896171    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c46md\" (UniqueName: \"kubernetes.io/projected/7cc2f9f3-d61c-470e-99ec-ddaef02da3aa-kube-api-access-c46md\") pod \"isvc-xgboost-graph-raw-hpa-c70d2-predictor-7c7cb8b49-8qmv2\" (UID: \"7cc2f9f3-d61c-470e-99ec-ddaef02da3aa\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-c70d2-predictor-7c7cb8b49-8qmv2"
Jul 28 15:47:12.940001 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:47:12.939969    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22nh9_ebf75c0d-1597-4a1c-b668-dc2a041eb35e/konnectivity-agent/0.log" limit=1024
Jul 28 15:47:12.997496 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:47:12.997464    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7cc2f9f3-d61c-470e-99ec-ddaef02da3aa-kserve-provision-location\") pod \"isvc-xgboost-graph-raw-hpa-c70d2-predictor-7c7cb8b49-8qmv2\" (UID: \"7cc2f9f3-d61c-470e-99ec-ddaef02da3aa\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-c70d2-predictor-7c7cb8b49-8qmv2"
Jul 28 15:47:12.997682 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:47:12.997503    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7cc2f9f3-d61c-470e-99ec-ddaef02da3aa-cabundle-cert\") pod \"isvc-xgboost-graph-raw-hpa-c70d2-predictor-7c7cb8b49-8qmv2\" (UID: \"7cc2f9f3-d61c-470e-99ec-ddaef02da3aa\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-c70d2-predictor-7c7cb8b49-8qmv2"
Jul 28 15:47:12.997682 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:47:12.997529    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7cc2f9f3-d61c-470e-99ec-ddaef02da3aa-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-graph-raw-hpa-c70d2-predictor-7c7cb8b49-8qmv2\" (UID: \"7cc2f9f3-d61c-470e-99ec-ddaef02da3aa\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-c70d2-predictor-7c7cb8b49-8qmv2"
Jul 28 15:47:12.997808 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:47:12.997675    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7cc2f9f3-d61c-470e-99ec-ddaef02da3aa-proxy-tls\") pod \"isvc-xgboost-graph-raw-hpa-c70d2-predictor-7c7cb8b49-8qmv2\" (UID: \"7cc2f9f3-d61c-470e-99ec-ddaef02da3aa\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-c70d2-predictor-7c7cb8b49-8qmv2"
Jul 28 15:47:12.997868 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:47:12.997798    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c46md\" (UniqueName: \"kubernetes.io/projected/7cc2f9f3-d61c-470e-99ec-ddaef02da3aa-kube-api-access-c46md\") pod \"isvc-xgboost-graph-raw-hpa-c70d2-predictor-7c7cb8b49-8qmv2\" (UID: \"7cc2f9f3-d61c-470e-99ec-ddaef02da3aa\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-c70d2-predictor-7c7cb8b49-8qmv2"
Jul 28 15:47:12.997938 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:47:12.997913    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7cc2f9f3-d61c-470e-99ec-ddaef02da3aa-kserve-provision-location\") pod \"isvc-xgboost-graph-raw-hpa-c70d2-predictor-7c7cb8b49-8qmv2\" (UID: \"7cc2f9f3-d61c-470e-99ec-ddaef02da3aa\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-c70d2-predictor-7c7cb8b49-8qmv2"
Jul 28 15:47:12.998571 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:47:12.998549    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7cc2f9f3-d61c-470e-99ec-ddaef02da3aa-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-graph-raw-hpa-c70d2-predictor-7c7cb8b49-8qmv2\" (UID: \"7cc2f9f3-d61c-470e-99ec-ddaef02da3aa\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-c70d2-predictor-7c7cb8b49-8qmv2"
Jul 28 15:47:12.998675 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:47:12.998601    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7cc2f9f3-d61c-470e-99ec-ddaef02da3aa-cabundle-cert\") pod \"isvc-xgboost-graph-raw-hpa-c70d2-predictor-7c7cb8b49-8qmv2\" (UID: \"7cc2f9f3-d61c-470e-99ec-ddaef02da3aa\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-c70d2-predictor-7c7cb8b49-8qmv2"
Jul 28 15:47:13.001057 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:47:13.001037    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7cc2f9f3-d61c-470e-99ec-ddaef02da3aa-proxy-tls\") pod \"isvc-xgboost-graph-raw-hpa-c70d2-predictor-7c7cb8b49-8qmv2\" (UID: \"7cc2f9f3-d61c-470e-99ec-ddaef02da3aa\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-c70d2-predictor-7c7cb8b49-8qmv2"
Jul 28 15:47:13.006217 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:47:13.006195    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c46md\" (UniqueName: \"kubernetes.io/projected/7cc2f9f3-d61c-470e-99ec-ddaef02da3aa-kube-api-access-c46md\") pod \"isvc-xgboost-graph-raw-hpa-c70d2-predictor-7c7cb8b49-8qmv2\" (UID: \"7cc2f9f3-d61c-470e-99ec-ddaef02da3aa\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-c70d2-predictor-7c7cb8b49-8qmv2"
Jul 28 15:47:13.039501 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:47:13.039462    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22nh9_ebf75c0d-1597-4a1c-b668-dc2a041eb35e/konnectivity-agent/0.log" limit=1024
Jul 28 15:47:13.186872 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:47:13.186578    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-c70d2-predictor-7c7cb8b49-8qmv2"
Jul 28 15:47:13.320200 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:47:13.320171    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22nh9_ebf75c0d-1597-4a1c-b668-dc2a041eb35e/konnectivity-agent/0.log" limit=1024
Jul 28 15:47:13.326933 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:47:13.326909    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-c70d2-predictor-7c7cb8b49-8qmv2"]
Jul 28 15:47:13.550121 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:47:13.550085    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-c70d2-predictor-7c7cb8b49-8qmv2" event={"ID":"7cc2f9f3-d61c-470e-99ec-ddaef02da3aa","Type":"ContainerStarted","Data":"0df9a550bf4be152dbe3b114ed931c6e859af210bac8d1a6baff53926480da65"}
Jul 28 15:47:16.526763 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:47:16.526712    2568 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/model-chainer-raw-0be34-9dd95d454-bdzdk" podUID="c8563ba5-080f-495d-a7c6-a32ca10dbf59" containerName="model-chainer-raw-0be34" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 15:47:16.563931 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:47:16.563897    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-c70d2-predictor-7c7cb8b49-8qmv2" event={"ID":"7cc2f9f3-d61c-470e-99ec-ddaef02da3aa","Type":"ContainerStarted","Data":"4935ad2cc36872c67dd41443fcd0d90fa2860415ec3774ccdcf4239bc46d4d0f"}
Jul 28 15:47:20.581692 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:47:20.581661    2568 generic.go:356] "Generic (PLEG): container finished" podID="7cc2f9f3-d61c-470e-99ec-ddaef02da3aa" containerID="4935ad2cc36872c67dd41443fcd0d90fa2860415ec3774ccdcf4239bc46d4d0f" exitCode=0
Jul 28 15:47:20.582081 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:47:20.581727    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-c70d2-predictor-7c7cb8b49-8qmv2" event={"ID":"7cc2f9f3-d61c-470e-99ec-ddaef02da3aa","Type":"ContainerDied","Data":"4935ad2cc36872c67dd41443fcd0d90fa2860415ec3774ccdcf4239bc46d4d0f"}
Jul 28 15:47:21.528206 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:47:21.528144    2568 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/model-chainer-raw-0be34-9dd95d454-bdzdk" podUID="c8563ba5-080f-495d-a7c6-a32ca10dbf59" containerName="model-chainer-raw-0be34" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 15:47:26.527188 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:47:26.527147    2568 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/model-chainer-raw-0be34-9dd95d454-bdzdk" podUID="c8563ba5-080f-495d-a7c6-a32ca10dbf59" containerName="model-chainer-raw-0be34" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 15:47:26.527742 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:47:26.527285    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/model-chainer-raw-0be34-9dd95d454-bdzdk"
Jul 28 15:47:31.527579 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:47:31.527537    2568 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/model-chainer-raw-0be34-9dd95d454-bdzdk" podUID="c8563ba5-080f-495d-a7c6-a32ca10dbf59" containerName="model-chainer-raw-0be34" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 15:47:36.526689 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:47:36.526647    2568 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/model-chainer-raw-0be34-9dd95d454-bdzdk" podUID="c8563ba5-080f-495d-a7c6-a32ca10dbf59" containerName="model-chainer-raw-0be34" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 15:47:38.665826 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:47:38.665767    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-c70d2-predictor-7c7cb8b49-8qmv2" event={"ID":"7cc2f9f3-d61c-470e-99ec-ddaef02da3aa","Type":"ContainerStarted","Data":"9d59ae03224d072246f9b08aa672209cf8417e0ac88d7e2fd922acc08dd76eca"}
Jul 28 15:47:41.527289 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:47:41.527252    2568 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/model-chainer-raw-0be34-9dd95d454-bdzdk" podUID="c8563ba5-080f-495d-a7c6-a32ca10dbf59" containerName="model-chainer-raw-0be34" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 15:47:41.680529 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:47:41.680482    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-c70d2-predictor-7c7cb8b49-8qmv2" event={"ID":"7cc2f9f3-d61c-470e-99ec-ddaef02da3aa","Type":"ContainerStarted","Data":"8aa682b406654550c86c0306aa793d181723d21443a55fd4eb5319e0bc0e3735"}
Jul 28 15:47:41.680720 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:47:41.680575    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-c70d2-predictor-7c7cb8b49-8qmv2"
Jul 28 15:47:41.680804 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:47:41.680737    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-c70d2-predictor-7c7cb8b49-8qmv2"
Jul 28 15:47:41.682073 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:47:41.682045    2568 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-c70d2-predictor-7c7cb8b49-8qmv2" podUID="7cc2f9f3-d61c-470e-99ec-ddaef02da3aa" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.26:8080: connect: connection refused"
Jul 28 15:47:41.703645 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:47:41.703593    2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-c70d2-predictor-7c7cb8b49-8qmv2" podStartSLOduration=2.383358281 podStartE2EDuration="29.703575676s" totalImagesPullingTime="23.268514782s" totalInitContainerRuntime="4.051702613s" isStatefulPod=false podCreationTimestamp="2026-07-28 15:47:12 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 15:47:41.701432929 +0000 UTC m=+841.886628094" watchObservedRunningTime="2026-07-28 15:47:41.703575676 +0000 UTC m=+841.888770844"
Jul 28 15:47:42.687122 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:47:42.687082    2568 generic.go:356] "Generic (PLEG): container finished" podID="c8563ba5-080f-495d-a7c6-a32ca10dbf59" containerID="9db9810ad53fad91822805183723548d49f5d33447605e4907fafbd18468a3e5" exitCode=0
Jul 28 15:47:42.687585 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:47:42.687152    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/model-chainer-raw-0be34-9dd95d454-bdzdk" event={"ID":"c8563ba5-080f-495d-a7c6-a32ca10dbf59","Type":"ContainerDied","Data":"9db9810ad53fad91822805183723548d49f5d33447605e4907fafbd18468a3e5"}
Jul 28 15:47:42.687585 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:47:42.687427    2568 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-c70d2-predictor-7c7cb8b49-8qmv2" podUID="7cc2f9f3-d61c-470e-99ec-ddaef02da3aa" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.26:8080: connect: connection refused"
Jul 28 15:47:42.708571 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:47:42.708551    2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/model-chainer-raw-0be34-9dd95d454-bdzdk"
Jul 28 15:47:42.778024 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:47:42.777979    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c8563ba5-080f-495d-a7c6-a32ca10dbf59-proxy-tls\") pod \"c8563ba5-080f-495d-a7c6-a32ca10dbf59\" (UID: \"c8563ba5-080f-495d-a7c6-a32ca10dbf59\") "
Jul 28 15:47:42.778229 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:47:42.778084    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c8563ba5-080f-495d-a7c6-a32ca10dbf59-openshift-service-ca-bundle\") pod \"c8563ba5-080f-495d-a7c6-a32ca10dbf59\" (UID: \"c8563ba5-080f-495d-a7c6-a32ca10dbf59\") "
Jul 28 15:47:42.778456 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:47:42.778428    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c8563ba5-080f-495d-a7c6-a32ca10dbf59-openshift-service-ca-bundle" (OuterVolumeSpecName: "openshift-service-ca-bundle") pod "c8563ba5-080f-495d-a7c6-a32ca10dbf59" (UID: "c8563ba5-080f-495d-a7c6-a32ca10dbf59"). InnerVolumeSpecName "openshift-service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 15:47:42.780338 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:47:42.780313    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c8563ba5-080f-495d-a7c6-a32ca10dbf59-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "c8563ba5-080f-495d-a7c6-a32ca10dbf59" (UID: "c8563ba5-080f-495d-a7c6-a32ca10dbf59"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 15:47:42.879059 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:47:42.878968    2568 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c8563ba5-080f-495d-a7c6-a32ca10dbf59-openshift-service-ca-bundle\") on node \"ip-10-0-131-121.ec2.internal\" DevicePath \"\""
Jul 28 15:47:42.879059 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:47:42.879001    2568 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c8563ba5-080f-495d-a7c6-a32ca10dbf59-proxy-tls\") on node \"ip-10-0-131-121.ec2.internal\" DevicePath \"\""
Jul 28 15:47:43.692320 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:47:43.692296    2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/model-chainer-raw-0be34-9dd95d454-bdzdk"
Jul 28 15:47:43.692320 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:47:43.692310    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/model-chainer-raw-0be34-9dd95d454-bdzdk" event={"ID":"c8563ba5-080f-495d-a7c6-a32ca10dbf59","Type":"ContainerDied","Data":"2532931bcbcc881e5386475919bb20f3f92d5fcebce543bf4e65451fc8aad8c1"}
Jul 28 15:47:43.692799 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:47:43.692370    2568 scope.go:117] "RemoveContainer" containerID="9db9810ad53fad91822805183723548d49f5d33447605e4907fafbd18468a3e5"
Jul 28 15:47:43.716114 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:47:43.716088    2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/model-chainer-raw-0be34-9dd95d454-bdzdk"]
Jul 28 15:47:43.722491 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:47:43.722469    2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/model-chainer-raw-0be34-9dd95d454-bdzdk"]
Jul 28 15:47:44.294427 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:47:44.294382    2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c8563ba5-080f-495d-a7c6-a32ca10dbf59" path="/var/lib/kubelet/pods/c8563ba5-080f-495d-a7c6-a32ca10dbf59/volumes"
Jul 28 15:47:47.692292 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:47:47.692262    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-c70d2-predictor-7c7cb8b49-8qmv2"
Jul 28 15:47:47.692849 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:47:47.692816    2568 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-c70d2-predictor-7c7cb8b49-8qmv2" podUID="7cc2f9f3-d61c-470e-99ec-ddaef02da3aa" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.26:8080: connect: connection refused"
Jul 28 15:47:57.594165 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:47:57.594128    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22nh9_ebf75c0d-1597-4a1c-b668-dc2a041eb35e/konnectivity-agent/0.log" limit=1024
Jul 28 15:47:57.693411 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:47:57.693373    2568 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-c70d2-predictor-7c7cb8b49-8qmv2" podUID="7cc2f9f3-d61c-470e-99ec-ddaef02da3aa" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.26:8080: connect: connection refused"
Jul 28 15:48:07.692786 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:48:07.692707    2568 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-c70d2-predictor-7c7cb8b49-8qmv2" podUID="7cc2f9f3-d61c-470e-99ec-ddaef02da3aa" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.26:8080: connect: connection refused"
Jul 28 15:48:17.693027 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:48:17.692987    2568 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-c70d2-predictor-7c7cb8b49-8qmv2" podUID="7cc2f9f3-d61c-470e-99ec-ddaef02da3aa" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.26:8080: connect: connection refused"
Jul 28 15:48:27.693047 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:48:27.693008    2568 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-c70d2-predictor-7c7cb8b49-8qmv2" podUID="7cc2f9f3-d61c-470e-99ec-ddaef02da3aa" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.26:8080: connect: connection refused"
Jul 28 15:48:37.693066 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:48:37.692978    2568 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-c70d2-predictor-7c7cb8b49-8qmv2" podUID="7cc2f9f3-d61c-470e-99ec-ddaef02da3aa" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.26:8080: connect: connection refused"
Jul 28 15:48:46.646257 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:48:46.646200    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22nh9_ebf75c0d-1597-4a1c-b668-dc2a041eb35e/konnectivity-agent/0.log" limit=1024
Jul 28 15:48:47.693996 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:48:47.693952    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-c70d2-predictor-7c7cb8b49-8qmv2"
Jul 28 15:48:52.865161 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:48:52.865120    2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/model-chainer-raw-hpa-c70d2-5f8cf6db8b-wgdr9"]
Jul 28 15:48:52.865560 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:48:52.865184    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22nh9_ebf75c0d-1597-4a1c-b668-dc2a041eb35e/konnectivity-agent/0.log" limit=1024
Jul 28 15:48:52.865560 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:48:52.865482    2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c8563ba5-080f-495d-a7c6-a32ca10dbf59" containerName="model-chainer-raw-0be34"
Jul 28 15:48:52.865560 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:48:52.865495    2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="c8563ba5-080f-495d-a7c6-a32ca10dbf59" containerName="model-chainer-raw-0be34"
Jul 28 15:48:52.865560 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:48:52.865557    2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="c8563ba5-080f-495d-a7c6-a32ca10dbf59" containerName="model-chainer-raw-0be34"
Jul 28 15:48:52.868007 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:48:52.867991    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/model-chainer-raw-hpa-c70d2-5f8cf6db8b-wgdr9"
Jul 28 15:48:52.871422 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:48:52.871403    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"model-chainer-raw-hpa-c70d2-serving-cert\""
Jul 28 15:48:52.871498 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:48:52.871451    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"model-chainer-raw-hpa-c70d2-kube-rbac-proxy-sar-config\""
Jul 28 15:48:52.878193 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:48:52.878166    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/model-chainer-raw-hpa-c70d2-5f8cf6db8b-wgdr9"]
Jul 28 15:48:52.930487 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:48:52.930457    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22nh9_ebf75c0d-1597-4a1c-b668-dc2a041eb35e/konnectivity-agent/0.log" limit=1024
Jul 28 15:48:52.980276 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:48:52.980246    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ec023ae8-844f-48c3-abf7-15e62e25d89a-openshift-service-ca-bundle\") pod \"model-chainer-raw-hpa-c70d2-5f8cf6db8b-wgdr9\" (UID: \"ec023ae8-844f-48c3-abf7-15e62e25d89a\") " pod="kserve-ci-e2e-test/model-chainer-raw-hpa-c70d2-5f8cf6db8b-wgdr9"
Jul 28 15:48:52.980276 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:48:52.980277    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ec023ae8-844f-48c3-abf7-15e62e25d89a-proxy-tls\") pod \"model-chainer-raw-hpa-c70d2-5f8cf6db8b-wgdr9\" (UID: \"ec023ae8-844f-48c3-abf7-15e62e25d89a\") " pod="kserve-ci-e2e-test/model-chainer-raw-hpa-c70d2-5f8cf6db8b-wgdr9"
Jul 28 15:48:53.081655 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:48:53.081602    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ec023ae8-844f-48c3-abf7-15e62e25d89a-openshift-service-ca-bundle\") pod \"model-chainer-raw-hpa-c70d2-5f8cf6db8b-wgdr9\" (UID: \"ec023ae8-844f-48c3-abf7-15e62e25d89a\") " pod="kserve-ci-e2e-test/model-chainer-raw-hpa-c70d2-5f8cf6db8b-wgdr9"
Jul 28 15:48:53.081655 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:48:53.081660    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ec023ae8-844f-48c3-abf7-15e62e25d89a-proxy-tls\") pod \"model-chainer-raw-hpa-c70d2-5f8cf6db8b-wgdr9\" (UID: \"ec023ae8-844f-48c3-abf7-15e62e25d89a\") " pod="kserve-ci-e2e-test/model-chainer-raw-hpa-c70d2-5f8cf6db8b-wgdr9"
Jul 28 15:48:53.082276 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:48:53.082255    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ec023ae8-844f-48c3-abf7-15e62e25d89a-openshift-service-ca-bundle\") pod \"model-chainer-raw-hpa-c70d2-5f8cf6db8b-wgdr9\" (UID: \"ec023ae8-844f-48c3-abf7-15e62e25d89a\") " pod="kserve-ci-e2e-test/model-chainer-raw-hpa-c70d2-5f8cf6db8b-wgdr9"
Jul 28 15:48:53.084052 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:48:53.084035    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ec023ae8-844f-48c3-abf7-15e62e25d89a-proxy-tls\") pod \"model-chainer-raw-hpa-c70d2-5f8cf6db8b-wgdr9\" (UID: \"ec023ae8-844f-48c3-abf7-15e62e25d89a\") " pod="kserve-ci-e2e-test/model-chainer-raw-hpa-c70d2-5f8cf6db8b-wgdr9"
Jul 28 15:48:53.179550 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:48:53.179449    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/model-chainer-raw-hpa-c70d2-5f8cf6db8b-wgdr9"
Jul 28 15:48:53.313632 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:48:53.313607    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/model-chainer-raw-hpa-c70d2-5f8cf6db8b-wgdr9"]
Jul 28 15:48:53.316250 ip-10-0-131-121 kubenswrapper[2568]: W0728 15:48:53.316222    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podec023ae8_844f_48c3_abf7_15e62e25d89a.slice/crio-6e907a8386050a2900ff7f8e7301bed3dc5fadf343659419283e2326dc6a20f6 WatchSource:0}: Error finding container 6e907a8386050a2900ff7f8e7301bed3dc5fadf343659419283e2326dc6a20f6: Status 404 returned error can't find the container with id 6e907a8386050a2900ff7f8e7301bed3dc5fadf343659419283e2326dc6a20f6
Jul 28 15:48:53.981117 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:48:53.981082    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/model-chainer-raw-hpa-c70d2-5f8cf6db8b-wgdr9" event={"ID":"ec023ae8-844f-48c3-abf7-15e62e25d89a","Type":"ContainerStarted","Data":"201526d286d937b505f53e95bc5f194d8cc757350f3c4cf3e5810b61481306f7"}
Jul 28 15:48:53.981513 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:48:53.981141    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/model-chainer-raw-hpa-c70d2-5f8cf6db8b-wgdr9" event={"ID":"ec023ae8-844f-48c3-abf7-15e62e25d89a","Type":"ContainerStarted","Data":"6e907a8386050a2900ff7f8e7301bed3dc5fadf343659419283e2326dc6a20f6"}
Jul 28 15:48:53.981513 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:48:53.981182    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/model-chainer-raw-hpa-c70d2-5f8cf6db8b-wgdr9"
Jul 28 15:48:53.998976 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:48:53.998926    2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/model-chainer-raw-hpa-c70d2-5f8cf6db8b-wgdr9" podStartSLOduration=1.998908751 podStartE2EDuration="1.998908751s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 15:48:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 15:48:53.996758854 +0000 UTC m=+914.181954023" watchObservedRunningTime="2026-07-28 15:48:53.998908751 +0000 UTC m=+914.184103919"
Jul 28 15:48:59.991388 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:48:59.991348    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/model-chainer-raw-hpa-c70d2-5f8cf6db8b-wgdr9"
Jul 28 15:49:02.887138 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:49:02.887101    2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/model-chainer-raw-hpa-c70d2-5f8cf6db8b-wgdr9"]
Jul 28 15:49:02.887634 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:49:02.887401    2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/model-chainer-raw-hpa-c70d2-5f8cf6db8b-wgdr9" podUID="ec023ae8-844f-48c3-abf7-15e62e25d89a" containerName="model-chainer-raw-hpa-c70d2" containerID="cri-o://201526d286d937b505f53e95bc5f194d8cc757350f3c4cf3e5810b61481306f7" gracePeriod=30
Jul 28 15:49:02.911315 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:49:02.911284    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22nh9_ebf75c0d-1597-4a1c-b668-dc2a041eb35e/konnectivity-agent/0.log" limit=1024
Jul 28 15:49:03.060307 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:49:03.060275    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22nh9_ebf75c0d-1597-4a1c-b668-dc2a041eb35e/konnectivity-agent/0.log" limit=1024
Jul 28 15:49:03.091835 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:49:03.091801    2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/message-dumper-raw-22083-predictor-7fc855cfcc-kv9rd"]
Jul 28 15:49:03.094287 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:49:03.094268    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/message-dumper-raw-22083-predictor-7fc855cfcc-kv9rd"
Jul 28 15:49:03.096824 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:49:03.096798    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"message-dumper-raw-22083-predictor-serving-cert\""
Jul 28 15:49:03.096930 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:49:03.096808    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"message-dumper-raw-22083-kube-rbac-proxy-sar-config\""
Jul 28 15:49:03.105287 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:49:03.105260    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/message-dumper-raw-22083-predictor-7fc855cfcc-kv9rd"]
Jul 28 15:49:03.126748 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:49:03.126717    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22nh9_ebf75c0d-1597-4a1c-b668-dc2a041eb35e/konnectivity-agent/0.log" limit=1024
Jul 28 15:49:03.155222 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:49:03.155131    2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-c70d2-predictor-7c7cb8b49-8qmv2"]
Jul 28 15:49:03.155458 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:49:03.155434    2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-c70d2-predictor-7c7cb8b49-8qmv2" podUID="7cc2f9f3-d61c-470e-99ec-ddaef02da3aa" containerName="kserve-container" containerID="cri-o://9d59ae03224d072246f9b08aa672209cf8417e0ac88d7e2fd922acc08dd76eca" gracePeriod=30
Jul 28 15:49:03.155545 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:49:03.155506    2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-c70d2-predictor-7c7cb8b49-8qmv2" podUID="7cc2f9f3-d61c-470e-99ec-ddaef02da3aa" containerName="kube-rbac-proxy" containerID="cri-o://8aa682b406654550c86c0306aa793d181723d21443a55fd4eb5319e0bc0e3735" gracePeriod=30
Jul 28 15:49:03.167497 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:49:03.167471    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0d8bf4a5-d7b5-4204-9890-7cace9680330-proxy-tls\") pod \"message-dumper-raw-22083-predictor-7fc855cfcc-kv9rd\" (UID: \"0d8bf4a5-d7b5-4204-9890-7cace9680330\") " pod="kserve-ci-e2e-test/message-dumper-raw-22083-predictor-7fc855cfcc-kv9rd"
Jul 28 15:49:03.167709 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:49:03.167523    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0d8bf4a5-d7b5-4204-9890-7cace9680330-kube-rbac-proxy-sar-config\") pod \"message-dumper-raw-22083-predictor-7fc855cfcc-kv9rd\" (UID: \"0d8bf4a5-d7b5-4204-9890-7cace9680330\") " pod="kserve-ci-e2e-test/message-dumper-raw-22083-predictor-7fc855cfcc-kv9rd"
Jul 28 15:49:03.167709 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:49:03.167545    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nd6bm\" (UniqueName: \"kubernetes.io/projected/0d8bf4a5-d7b5-4204-9890-7cace9680330-kube-api-access-nd6bm\") pod \"message-dumper-raw-22083-predictor-7fc855cfcc-kv9rd\" (UID: \"0d8bf4a5-d7b5-4204-9890-7cace9680330\") " pod="kserve-ci-e2e-test/message-dumper-raw-22083-predictor-7fc855cfcc-kv9rd"
Jul 28 15:49:03.191890 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:49:03.191860    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22nh9_ebf75c0d-1597-4a1c-b668-dc2a041eb35e/konnectivity-agent/0.log" limit=1024
Jul 28 15:49:03.268167 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:49:03.268132    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nd6bm\" (UniqueName: \"kubernetes.io/projected/0d8bf4a5-d7b5-4204-9890-7cace9680330-kube-api-access-nd6bm\") pod \"message-dumper-raw-22083-predictor-7fc855cfcc-kv9rd\" (UID: \"0d8bf4a5-d7b5-4204-9890-7cace9680330\") " pod="kserve-ci-e2e-test/message-dumper-raw-22083-predictor-7fc855cfcc-kv9rd"
Jul 28 15:49:03.268324 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:49:03.268207    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0d8bf4a5-d7b5-4204-9890-7cace9680330-proxy-tls\") pod \"message-dumper-raw-22083-predictor-7fc855cfcc-kv9rd\" (UID: \"0d8bf4a5-d7b5-4204-9890-7cace9680330\") " pod="kserve-ci-e2e-test/message-dumper-raw-22083-predictor-7fc855cfcc-kv9rd"
Jul 28 15:49:03.268324 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:49:03.268250    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0d8bf4a5-d7b5-4204-9890-7cace9680330-kube-rbac-proxy-sar-config\") pod \"message-dumper-raw-22083-predictor-7fc855cfcc-kv9rd\" (UID: \"0d8bf4a5-d7b5-4204-9890-7cace9680330\") " pod="kserve-ci-e2e-test/message-dumper-raw-22083-predictor-7fc855cfcc-kv9rd"
Jul 28 15:49:03.268456 ip-10-0-131-121 kubenswrapper[2568]: E0728 15:49:03.268364    2568 secret.go:189] Couldn't get secret kserve-ci-e2e-test/message-dumper-raw-22083-predictor-serving-cert: secret "message-dumper-raw-22083-predictor-serving-cert" not found
Jul 28 15:49:03.268456 ip-10-0-131-121 kubenswrapper[2568]: E0728 15:49:03.268436    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0d8bf4a5-d7b5-4204-9890-7cace9680330-proxy-tls podName:0d8bf4a5-d7b5-4204-9890-7cace9680330 nodeName:}" failed. No retries permitted until 2026-07-28 15:49:03.768413983 +0000 UTC m=+923.953609147 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/0d8bf4a5-d7b5-4204-9890-7cace9680330-proxy-tls") pod "message-dumper-raw-22083-predictor-7fc855cfcc-kv9rd" (UID: "0d8bf4a5-d7b5-4204-9890-7cace9680330") : secret "message-dumper-raw-22083-predictor-serving-cert" not found
Jul 28 15:49:03.268834 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:49:03.268815    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0d8bf4a5-d7b5-4204-9890-7cace9680330-kube-rbac-proxy-sar-config\") pod \"message-dumper-raw-22083-predictor-7fc855cfcc-kv9rd\" (UID: \"0d8bf4a5-d7b5-4204-9890-7cace9680330\") " pod="kserve-ci-e2e-test/message-dumper-raw-22083-predictor-7fc855cfcc-kv9rd"
Jul 28 15:49:03.278651 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:49:03.278624    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nd6bm\" (UniqueName: \"kubernetes.io/projected/0d8bf4a5-d7b5-4204-9890-7cace9680330-kube-api-access-nd6bm\") pod \"message-dumper-raw-22083-predictor-7fc855cfcc-kv9rd\" (UID: \"0d8bf4a5-d7b5-4204-9890-7cace9680330\") " pod="kserve-ci-e2e-test/message-dumper-raw-22083-predictor-7fc855cfcc-kv9rd"
Jul 28 15:49:03.314122 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:49:03.314088    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22nh9_ebf75c0d-1597-4a1c-b668-dc2a041eb35e/konnectivity-agent/0.log" limit=1024
Jul 28 15:49:03.773315 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:49:03.773278    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0d8bf4a5-d7b5-4204-9890-7cace9680330-proxy-tls\") pod \"message-dumper-raw-22083-predictor-7fc855cfcc-kv9rd\" (UID: \"0d8bf4a5-d7b5-4204-9890-7cace9680330\") " pod="kserve-ci-e2e-test/message-dumper-raw-22083-predictor-7fc855cfcc-kv9rd"
Jul 28 15:49:03.775653 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:49:03.775631    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0d8bf4a5-d7b5-4204-9890-7cace9680330-proxy-tls\") pod \"message-dumper-raw-22083-predictor-7fc855cfcc-kv9rd\" (UID: \"0d8bf4a5-d7b5-4204-9890-7cace9680330\") " pod="kserve-ci-e2e-test/message-dumper-raw-22083-predictor-7fc855cfcc-kv9rd"
Jul 28 15:49:04.007530 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:49:04.007490    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/message-dumper-raw-22083-predictor-7fc855cfcc-kv9rd"
Jul 28 15:49:04.025031 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:49:04.024943    2568 generic.go:356] "Generic (PLEG): container finished" podID="7cc2f9f3-d61c-470e-99ec-ddaef02da3aa" containerID="8aa682b406654550c86c0306aa793d181723d21443a55fd4eb5319e0bc0e3735" exitCode=2
Jul 28 15:49:04.025157 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:49:04.025021    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-c70d2-predictor-7c7cb8b49-8qmv2" event={"ID":"7cc2f9f3-d61c-470e-99ec-ddaef02da3aa","Type":"ContainerDied","Data":"8aa682b406654550c86c0306aa793d181723d21443a55fd4eb5319e0bc0e3735"}
Jul 28 15:49:04.142422 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:49:04.142398    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/message-dumper-raw-22083-predictor-7fc855cfcc-kv9rd"]
Jul 28 15:49:04.144437 ip-10-0-131-121 kubenswrapper[2568]: W0728 15:49:04.144401    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0d8bf4a5_d7b5_4204_9890_7cace9680330.slice/crio-79a3b6f61fdeb39ddafad1adf321ed5d04fffcfa30ae14389fbc418375c5446d WatchSource:0}: Error finding container 79a3b6f61fdeb39ddafad1adf321ed5d04fffcfa30ae14389fbc418375c5446d: Status 404 returned error can't find the container with id 79a3b6f61fdeb39ddafad1adf321ed5d04fffcfa30ae14389fbc418375c5446d
Jul 28 15:49:04.991390 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:49:04.991349    2568 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/model-chainer-raw-hpa-c70d2-5f8cf6db8b-wgdr9" podUID="ec023ae8-844f-48c3-abf7-15e62e25d89a" containerName="model-chainer-raw-hpa-c70d2" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 15:49:05.032444 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:49:05.032411    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-raw-22083-predictor-7fc855cfcc-kv9rd" event={"ID":"0d8bf4a5-d7b5-4204-9890-7cace9680330","Type":"ContainerStarted","Data":"79a3b6f61fdeb39ddafad1adf321ed5d04fffcfa30ae14389fbc418375c5446d"}
Jul 28 15:49:06.038843 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:49:06.038800    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-raw-22083-predictor-7fc855cfcc-kv9rd" event={"ID":"0d8bf4a5-d7b5-4204-9890-7cace9680330","Type":"ContainerStarted","Data":"cfa0c39877be81ff5714a0b33ec63eb317214d0db0787c05c86b7525199d433b"}
Jul 28 15:49:06.039245 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:49:06.038879    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/message-dumper-raw-22083-predictor-7fc855cfcc-kv9rd"
Jul 28 15:49:06.039245 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:49:06.039066    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/message-dumper-raw-22083-predictor-7fc855cfcc-kv9rd"
Jul 28 15:49:06.039245 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:49:06.039109    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-raw-22083-predictor-7fc855cfcc-kv9rd" event={"ID":"0d8bf4a5-d7b5-4204-9890-7cace9680330","Type":"ContainerStarted","Data":"1fba9ebdfc404d34824238ab574b9d04efe079d473aab0e932395018b9ae8a3e"}
Jul 28 15:49:06.040887 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:49:06.040869    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/message-dumper-raw-22083-predictor-7fc855cfcc-kv9rd"
Jul 28 15:49:06.059393 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:49:06.059343    2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/message-dumper-raw-22083-predictor-7fc855cfcc-kv9rd" podStartSLOduration=1.964957394 podStartE2EDuration="3.059326414s" totalImagesPullingTime="1.09436902s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 15:49:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 15:49:06.05721188 +0000 UTC m=+926.242407049" watchObservedRunningTime="2026-07-28 15:49:06.059326414 +0000 UTC m=+926.244521582"
Jul 28 15:49:07.001984 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:49:07.001959    2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-c70d2-predictor-7c7cb8b49-8qmv2"
Jul 28 15:49:07.045264 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:49:07.045230    2568 generic.go:356] "Generic (PLEG): container finished" podID="7cc2f9f3-d61c-470e-99ec-ddaef02da3aa" containerID="9d59ae03224d072246f9b08aa672209cf8417e0ac88d7e2fd922acc08dd76eca" exitCode=0
Jul 28 15:49:07.045739 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:49:07.045291    2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-c70d2-predictor-7c7cb8b49-8qmv2"
Jul 28 15:49:07.045739 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:49:07.045305    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-c70d2-predictor-7c7cb8b49-8qmv2" event={"ID":"7cc2f9f3-d61c-470e-99ec-ddaef02da3aa","Type":"ContainerDied","Data":"9d59ae03224d072246f9b08aa672209cf8417e0ac88d7e2fd922acc08dd76eca"}
Jul 28 15:49:07.045739 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:49:07.045390    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-c70d2-predictor-7c7cb8b49-8qmv2" event={"ID":"7cc2f9f3-d61c-470e-99ec-ddaef02da3aa","Type":"ContainerDied","Data":"0df9a550bf4be152dbe3b114ed931c6e859af210bac8d1a6baff53926480da65"}
Jul 28 15:49:07.045739 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:49:07.045416    2568 scope.go:117] "RemoveContainer" containerID="8aa682b406654550c86c0306aa793d181723d21443a55fd4eb5319e0bc0e3735"
Jul 28 15:49:07.053690 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:49:07.053669    2568 scope.go:117] "RemoveContainer" containerID="9d59ae03224d072246f9b08aa672209cf8417e0ac88d7e2fd922acc08dd76eca"
Jul 28 15:49:07.061789 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:49:07.061754    2568 scope.go:117] "RemoveContainer" containerID="4935ad2cc36872c67dd41443fcd0d90fa2860415ec3774ccdcf4239bc46d4d0f"
Jul 28 15:49:07.069664 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:49:07.069642    2568 scope.go:117] "RemoveContainer" containerID="8aa682b406654550c86c0306aa793d181723d21443a55fd4eb5319e0bc0e3735"
Jul 28 15:49:07.070001 ip-10-0-131-121 kubenswrapper[2568]: E0728 15:49:07.069981    2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8aa682b406654550c86c0306aa793d181723d21443a55fd4eb5319e0bc0e3735\": container with ID starting with 8aa682b406654550c86c0306aa793d181723d21443a55fd4eb5319e0bc0e3735 not found: ID does not exist" containerID="8aa682b406654550c86c0306aa793d181723d21443a55fd4eb5319e0bc0e3735"
Jul 28 15:49:07.070062 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:49:07.070011    2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8aa682b406654550c86c0306aa793d181723d21443a55fd4eb5319e0bc0e3735"} err="failed to get container status \"8aa682b406654550c86c0306aa793d181723d21443a55fd4eb5319e0bc0e3735\": rpc error: code = NotFound desc = could not find container \"8aa682b406654550c86c0306aa793d181723d21443a55fd4eb5319e0bc0e3735\": container with ID starting with 8aa682b406654550c86c0306aa793d181723d21443a55fd4eb5319e0bc0e3735 not found: ID does not exist"
Jul 28 15:49:07.070062 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:49:07.070030    2568 scope.go:117] "RemoveContainer" containerID="9d59ae03224d072246f9b08aa672209cf8417e0ac88d7e2fd922acc08dd76eca"
Jul 28 15:49:07.070244 ip-10-0-131-121 kubenswrapper[2568]: E0728 15:49:07.070229    2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9d59ae03224d072246f9b08aa672209cf8417e0ac88d7e2fd922acc08dd76eca\": container with ID starting with 9d59ae03224d072246f9b08aa672209cf8417e0ac88d7e2fd922acc08dd76eca not found: ID does not exist" containerID="9d59ae03224d072246f9b08aa672209cf8417e0ac88d7e2fd922acc08dd76eca"
Jul 28 15:49:07.070282 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:49:07.070247    2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9d59ae03224d072246f9b08aa672209cf8417e0ac88d7e2fd922acc08dd76eca"} err="failed to get container status \"9d59ae03224d072246f9b08aa672209cf8417e0ac88d7e2fd922acc08dd76eca\": rpc error: code = NotFound desc = could not find container \"9d59ae03224d072246f9b08aa672209cf8417e0ac88d7e2fd922acc08dd76eca\": container with ID starting with 9d59ae03224d072246f9b08aa672209cf8417e0ac88d7e2fd922acc08dd76eca not found: ID does not exist"
Jul 28 15:49:07.070282 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:49:07.070260    2568 scope.go:117] "RemoveContainer" containerID="4935ad2cc36872c67dd41443fcd0d90fa2860415ec3774ccdcf4239bc46d4d0f"
Jul 28 15:49:07.070467 ip-10-0-131-121 kubenswrapper[2568]: E0728 15:49:07.070449    2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4935ad2cc36872c67dd41443fcd0d90fa2860415ec3774ccdcf4239bc46d4d0f\": container with ID starting with 4935ad2cc36872c67dd41443fcd0d90fa2860415ec3774ccdcf4239bc46d4d0f not found: ID does not exist" containerID="4935ad2cc36872c67dd41443fcd0d90fa2860415ec3774ccdcf4239bc46d4d0f"
Jul 28 15:49:07.070507 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:49:07.070475    2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4935ad2cc36872c67dd41443fcd0d90fa2860415ec3774ccdcf4239bc46d4d0f"} err="failed to get container status \"4935ad2cc36872c67dd41443fcd0d90fa2860415ec3774ccdcf4239bc46d4d0f\": rpc error: code = NotFound desc = could not find container \"4935ad2cc36872c67dd41443fcd0d90fa2860415ec3774ccdcf4239bc46d4d0f\": container with ID starting with 4935ad2cc36872c67dd41443fcd0d90fa2860415ec3774ccdcf4239bc46d4d0f not found: ID does not exist"
Jul 28 15:49:07.103867 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:49:07.103794    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7cc2f9f3-d61c-470e-99ec-ddaef02da3aa-cabundle-cert\") pod \"7cc2f9f3-d61c-470e-99ec-ddaef02da3aa\" (UID: \"7cc2f9f3-d61c-470e-99ec-ddaef02da3aa\") "
Jul 28 15:49:07.103867 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:49:07.103844    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7cc2f9f3-d61c-470e-99ec-ddaef02da3aa-proxy-tls\") pod \"7cc2f9f3-d61c-470e-99ec-ddaef02da3aa\" (UID: \"7cc2f9f3-d61c-470e-99ec-ddaef02da3aa\") "
Jul 28 15:49:07.104002 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:49:07.103868    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7cc2f9f3-d61c-470e-99ec-ddaef02da3aa-kube-rbac-proxy-sar-config\") pod \"7cc2f9f3-d61c-470e-99ec-ddaef02da3aa\" (UID: \"7cc2f9f3-d61c-470e-99ec-ddaef02da3aa\") "
Jul 28 15:49:07.104002 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:49:07.103925    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-c46md\" (UniqueName: \"kubernetes.io/projected/7cc2f9f3-d61c-470e-99ec-ddaef02da3aa-kube-api-access-c46md\") pod \"7cc2f9f3-d61c-470e-99ec-ddaef02da3aa\" (UID: \"7cc2f9f3-d61c-470e-99ec-ddaef02da3aa\") "
Jul 28 15:49:07.104002 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:49:07.103969    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7cc2f9f3-d61c-470e-99ec-ddaef02da3aa-kserve-provision-location\") pod \"7cc2f9f3-d61c-470e-99ec-ddaef02da3aa\" (UID: \"7cc2f9f3-d61c-470e-99ec-ddaef02da3aa\") "
Jul 28 15:49:07.104281 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:49:07.104252    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7cc2f9f3-d61c-470e-99ec-ddaef02da3aa-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "7cc2f9f3-d61c-470e-99ec-ddaef02da3aa" (UID: "7cc2f9f3-d61c-470e-99ec-ddaef02da3aa"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 15:49:07.104364 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:49:07.104335    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7cc2f9f3-d61c-470e-99ec-ddaef02da3aa-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "7cc2f9f3-d61c-470e-99ec-ddaef02da3aa" (UID: "7cc2f9f3-d61c-470e-99ec-ddaef02da3aa"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 15:49:07.104408 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:49:07.104353    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7cc2f9f3-d61c-470e-99ec-ddaef02da3aa-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "7cc2f9f3-d61c-470e-99ec-ddaef02da3aa" (UID: "7cc2f9f3-d61c-470e-99ec-ddaef02da3aa"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 15:49:07.106070 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:49:07.106047    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7cc2f9f3-d61c-470e-99ec-ddaef02da3aa-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "7cc2f9f3-d61c-470e-99ec-ddaef02da3aa" (UID: "7cc2f9f3-d61c-470e-99ec-ddaef02da3aa"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 15:49:07.106132 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:49:07.106089    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7cc2f9f3-d61c-470e-99ec-ddaef02da3aa-kube-api-access-c46md" (OuterVolumeSpecName: "kube-api-access-c46md") pod "7cc2f9f3-d61c-470e-99ec-ddaef02da3aa" (UID: "7cc2f9f3-d61c-470e-99ec-ddaef02da3aa"). InnerVolumeSpecName "kube-api-access-c46md". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 15:49:07.205617 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:49:07.205575    2568 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7cc2f9f3-d61c-470e-99ec-ddaef02da3aa-cabundle-cert\") on node \"ip-10-0-131-121.ec2.internal\" DevicePath \"\""
Jul 28 15:49:07.205617 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:49:07.205608    2568 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7cc2f9f3-d61c-470e-99ec-ddaef02da3aa-proxy-tls\") on node \"ip-10-0-131-121.ec2.internal\" DevicePath \"\""
Jul 28 15:49:07.205617 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:49:07.205623    2568 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7cc2f9f3-d61c-470e-99ec-ddaef02da3aa-kube-rbac-proxy-sar-config\") on node \"ip-10-0-131-121.ec2.internal\" DevicePath \"\""
Jul 28 15:49:07.205970 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:49:07.205634    2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c46md\" (UniqueName: \"kubernetes.io/projected/7cc2f9f3-d61c-470e-99ec-ddaef02da3aa-kube-api-access-c46md\") on node \"ip-10-0-131-121.ec2.internal\" DevicePath \"\""
Jul 28 15:49:07.205970 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:49:07.205643    2568 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7cc2f9f3-d61c-470e-99ec-ddaef02da3aa-kserve-provision-location\") on node \"ip-10-0-131-121.ec2.internal\" DevicePath \"\""
Jul 28 15:49:07.374893 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:49:07.374801    2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-c70d2-predictor-7c7cb8b49-8qmv2"]
Jul 28 15:49:07.381206 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:49:07.381174    2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-c70d2-predictor-7c7cb8b49-8qmv2"]
Jul 28 15:49:08.294829 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:49:08.294794    2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7cc2f9f3-d61c-470e-99ec-ddaef02da3aa" path="/var/lib/kubelet/pods/7cc2f9f3-d61c-470e-99ec-ddaef02da3aa/volumes"
Jul 28 15:49:09.989372 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:49:09.989333    2568 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/model-chainer-raw-hpa-c70d2-5f8cf6db8b-wgdr9" podUID="ec023ae8-844f-48c3-abf7-15e62e25d89a" containerName="model-chainer-raw-hpa-c70d2" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 15:49:13.055131 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:49:13.055103    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/message-dumper-raw-22083-predictor-7fc855cfcc-kv9rd"
Jul 28 15:49:14.989849 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:49:14.989803    2568 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/model-chainer-raw-hpa-c70d2-5f8cf6db8b-wgdr9" podUID="ec023ae8-844f-48c3-abf7-15e62e25d89a" containerName="model-chainer-raw-hpa-c70d2" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 15:49:14.990235 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:49:14.989918    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/model-chainer-raw-hpa-c70d2-5f8cf6db8b-wgdr9"
Jul 28 15:49:19.990416 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:49:19.990381    2568 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/model-chainer-raw-hpa-c70d2-5f8cf6db8b-wgdr9" podUID="ec023ae8-844f-48c3-abf7-15e62e25d89a" containerName="model-chainer-raw-hpa-c70d2" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 15:49:23.111561 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:49:23.111527    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22nh9_ebf75c0d-1597-4a1c-b668-dc2a041eb35e/konnectivity-agent/0.log" limit=1024
Jul 28 15:49:23.572366 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:49:23.572338    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22nh9_ebf75c0d-1597-4a1c-b668-dc2a041eb35e/konnectivity-agent/0.log" limit=1024
Jul 28 15:49:24.989149 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:49:24.989111    2568 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/model-chainer-raw-hpa-c70d2-5f8cf6db8b-wgdr9" podUID="ec023ae8-844f-48c3-abf7-15e62e25d89a" containerName="model-chainer-raw-hpa-c70d2" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 15:49:29.989752 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:49:29.989707    2568 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/model-chainer-raw-hpa-c70d2-5f8cf6db8b-wgdr9" podUID="ec023ae8-844f-48c3-abf7-15e62e25d89a" containerName="model-chainer-raw-hpa-c70d2" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 15:49:33.040047 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:49:33.040016    2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/model-chainer-raw-hpa-c70d2-5f8cf6db8b-wgdr9"
Jul 28 15:49:33.130479 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:49:33.130440    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ec023ae8-844f-48c3-abf7-15e62e25d89a-openshift-service-ca-bundle\") pod \"ec023ae8-844f-48c3-abf7-15e62e25d89a\" (UID: \"ec023ae8-844f-48c3-abf7-15e62e25d89a\") "
Jul 28 15:49:33.130653 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:49:33.130497    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ec023ae8-844f-48c3-abf7-15e62e25d89a-proxy-tls\") pod \"ec023ae8-844f-48c3-abf7-15e62e25d89a\" (UID: \"ec023ae8-844f-48c3-abf7-15e62e25d89a\") "
Jul 28 15:49:33.130881 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:49:33.130855    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ec023ae8-844f-48c3-abf7-15e62e25d89a-openshift-service-ca-bundle" (OuterVolumeSpecName: "openshift-service-ca-bundle") pod "ec023ae8-844f-48c3-abf7-15e62e25d89a" (UID: "ec023ae8-844f-48c3-abf7-15e62e25d89a"). InnerVolumeSpecName "openshift-service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 15:49:33.132568 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:49:33.132545    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ec023ae8-844f-48c3-abf7-15e62e25d89a-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "ec023ae8-844f-48c3-abf7-15e62e25d89a" (UID: "ec023ae8-844f-48c3-abf7-15e62e25d89a"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 15:49:33.167557 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:49:33.167468    2568 generic.go:356] "Generic (PLEG): container finished" podID="ec023ae8-844f-48c3-abf7-15e62e25d89a" containerID="201526d286d937b505f53e95bc5f194d8cc757350f3c4cf3e5810b61481306f7" exitCode=0
Jul 28 15:49:33.167557 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:49:33.167511    2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/model-chainer-raw-hpa-c70d2-5f8cf6db8b-wgdr9"
Jul 28 15:49:33.167557 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:49:33.167535    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/model-chainer-raw-hpa-c70d2-5f8cf6db8b-wgdr9" event={"ID":"ec023ae8-844f-48c3-abf7-15e62e25d89a","Type":"ContainerDied","Data":"201526d286d937b505f53e95bc5f194d8cc757350f3c4cf3e5810b61481306f7"}
Jul 28 15:49:33.167808 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:49:33.167578    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/model-chainer-raw-hpa-c70d2-5f8cf6db8b-wgdr9" event={"ID":"ec023ae8-844f-48c3-abf7-15e62e25d89a","Type":"ContainerDied","Data":"6e907a8386050a2900ff7f8e7301bed3dc5fadf343659419283e2326dc6a20f6"}
Jul 28 15:49:33.167808 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:49:33.167598    2568 scope.go:117] "RemoveContainer" containerID="201526d286d937b505f53e95bc5f194d8cc757350f3c4cf3e5810b61481306f7"
Jul 28 15:49:33.175929 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:49:33.175909    2568 scope.go:117] "RemoveContainer" containerID="201526d286d937b505f53e95bc5f194d8cc757350f3c4cf3e5810b61481306f7"
Jul 28 15:49:33.176216 ip-10-0-131-121 kubenswrapper[2568]: E0728 15:49:33.176197    2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"201526d286d937b505f53e95bc5f194d8cc757350f3c4cf3e5810b61481306f7\": container with ID starting with 201526d286d937b505f53e95bc5f194d8cc757350f3c4cf3e5810b61481306f7 not found: ID does not exist" containerID="201526d286d937b505f53e95bc5f194d8cc757350f3c4cf3e5810b61481306f7"
Jul 28 15:49:33.176261 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:49:33.176227    2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"201526d286d937b505f53e95bc5f194d8cc757350f3c4cf3e5810b61481306f7"} err="failed to get container status \"201526d286d937b505f53e95bc5f194d8cc757350f3c4cf3e5810b61481306f7\": rpc error: code = NotFound desc = could not find container \"201526d286d937b505f53e95bc5f194d8cc757350f3c4cf3e5810b61481306f7\": container with ID starting with 201526d286d937b505f53e95bc5f194d8cc757350f3c4cf3e5810b61481306f7 not found: ID does not exist"
Jul 28 15:49:33.190286 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:49:33.190251    2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/model-chainer-raw-hpa-c70d2-5f8cf6db8b-wgdr9"]
Jul 28 15:49:33.195604 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:49:33.195574    2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/model-chainer-raw-hpa-c70d2-5f8cf6db8b-wgdr9"]
Jul 28 15:49:33.232158 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:49:33.232111    2568 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ec023ae8-844f-48c3-abf7-15e62e25d89a-proxy-tls\") on node \"ip-10-0-131-121.ec2.internal\" DevicePath \"\""
Jul 28 15:49:33.232158 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:49:33.232174    2568 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ec023ae8-844f-48c3-abf7-15e62e25d89a-openshift-service-ca-bundle\") on node \"ip-10-0-131-121.ec2.internal\" DevicePath \"\""
Jul 28 15:49:34.294689 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:49:34.294654    2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ec023ae8-844f-48c3-abf7-15e62e25d89a" path="/var/lib/kubelet/pods/ec023ae8-844f-48c3-abf7-15e62e25d89a/volumes"
Jul 28 15:50:58.179434 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:50:58.179405    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_message-dumper-raw-22083-predictor-7fc855cfcc-kv9rd_0d8bf4a5-d7b5-4204-9890-7cace9680330/kserve-container/0.log"
Jul 28 15:50:58.432514 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:50:58.432402    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22nh9_ebf75c0d-1597-4a1c-b668-dc2a041eb35e/konnectivity-agent/0.log" limit=1024
Jul 28 15:50:58.525484 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:50:58.525454    2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/message-dumper-raw-22083-predictor-7fc855cfcc-kv9rd"]
Jul 28 15:50:58.525693 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:50:58.525665    2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/message-dumper-raw-22083-predictor-7fc855cfcc-kv9rd" podUID="0d8bf4a5-d7b5-4204-9890-7cace9680330" containerName="kserve-container" containerID="cri-o://1fba9ebdfc404d34824238ab574b9d04efe079d473aab0e932395018b9ae8a3e" gracePeriod=30
Jul 28 15:50:58.525918 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:50:58.525711    2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/message-dumper-raw-22083-predictor-7fc855cfcc-kv9rd" podUID="0d8bf4a5-d7b5-4204-9890-7cace9680330" containerName="kube-rbac-proxy" containerID="cri-o://cfa0c39877be81ff5714a0b33ec63eb317214d0db0787c05c86b7525199d433b" gracePeriod=30
Jul 28 15:50:58.783255 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:50:58.783230    2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/message-dumper-raw-22083-predictor-7fc855cfcc-kv9rd"
Jul 28 15:50:58.927292 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:50:58.927267    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nd6bm\" (UniqueName: \"kubernetes.io/projected/0d8bf4a5-d7b5-4204-9890-7cace9680330-kube-api-access-nd6bm\") pod \"0d8bf4a5-d7b5-4204-9890-7cace9680330\" (UID: \"0d8bf4a5-d7b5-4204-9890-7cace9680330\") "
Jul 28 15:50:58.927433 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:50:58.927346    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0d8bf4a5-d7b5-4204-9890-7cace9680330-proxy-tls\") pod \"0d8bf4a5-d7b5-4204-9890-7cace9680330\" (UID: \"0d8bf4a5-d7b5-4204-9890-7cace9680330\") "
Jul 28 15:50:58.927433 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:50:58.927391    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0d8bf4a5-d7b5-4204-9890-7cace9680330-kube-rbac-proxy-sar-config\") pod \"0d8bf4a5-d7b5-4204-9890-7cace9680330\" (UID: \"0d8bf4a5-d7b5-4204-9890-7cace9680330\") "
Jul 28 15:50:58.927934 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:50:58.927902    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0d8bf4a5-d7b5-4204-9890-7cace9680330-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "0d8bf4a5-d7b5-4204-9890-7cace9680330" (UID: "0d8bf4a5-d7b5-4204-9890-7cace9680330"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 15:50:58.929446 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:50:58.929417    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0d8bf4a5-d7b5-4204-9890-7cace9680330-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "0d8bf4a5-d7b5-4204-9890-7cace9680330" (UID: "0d8bf4a5-d7b5-4204-9890-7cace9680330"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 15:50:58.929548 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:50:58.929462    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0d8bf4a5-d7b5-4204-9890-7cace9680330-kube-api-access-nd6bm" (OuterVolumeSpecName: "kube-api-access-nd6bm") pod "0d8bf4a5-d7b5-4204-9890-7cace9680330" (UID: "0d8bf4a5-d7b5-4204-9890-7cace9680330"). InnerVolumeSpecName "kube-api-access-nd6bm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 15:50:59.028622 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:50:59.028590    2568 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0d8bf4a5-d7b5-4204-9890-7cace9680330-kube-rbac-proxy-sar-config\") on node \"ip-10-0-131-121.ec2.internal\" DevicePath \"\""
Jul 28 15:50:59.028622 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:50:59.028617    2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nd6bm\" (UniqueName: \"kubernetes.io/projected/0d8bf4a5-d7b5-4204-9890-7cace9680330-kube-api-access-nd6bm\") on node \"ip-10-0-131-121.ec2.internal\" DevicePath \"\""
Jul 28 15:50:59.028622 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:50:59.028627    2568 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0d8bf4a5-d7b5-4204-9890-7cace9680330-proxy-tls\") on node \"ip-10-0-131-121.ec2.internal\" DevicePath \"\""
Jul 28 15:50:59.513581 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:50:59.513550    2568 generic.go:356] "Generic (PLEG): container finished" podID="0d8bf4a5-d7b5-4204-9890-7cace9680330" containerID="cfa0c39877be81ff5714a0b33ec63eb317214d0db0787c05c86b7525199d433b" exitCode=2
Jul 28 15:50:59.513581 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:50:59.513572    2568 generic.go:356] "Generic (PLEG): container finished" podID="0d8bf4a5-d7b5-4204-9890-7cace9680330" containerID="1fba9ebdfc404d34824238ab574b9d04efe079d473aab0e932395018b9ae8a3e" exitCode=2
Jul 28 15:50:59.514040 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:50:59.513601    2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/message-dumper-raw-22083-predictor-7fc855cfcc-kv9rd"
Jul 28 15:50:59.514040 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:50:59.513635    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-raw-22083-predictor-7fc855cfcc-kv9rd" event={"ID":"0d8bf4a5-d7b5-4204-9890-7cace9680330","Type":"ContainerDied","Data":"cfa0c39877be81ff5714a0b33ec63eb317214d0db0787c05c86b7525199d433b"}
Jul 28 15:50:59.514040 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:50:59.513707    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-raw-22083-predictor-7fc855cfcc-kv9rd" event={"ID":"0d8bf4a5-d7b5-4204-9890-7cace9680330","Type":"ContainerDied","Data":"1fba9ebdfc404d34824238ab574b9d04efe079d473aab0e932395018b9ae8a3e"}
Jul 28 15:50:59.514040 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:50:59.513741    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-raw-22083-predictor-7fc855cfcc-kv9rd" event={"ID":"0d8bf4a5-d7b5-4204-9890-7cace9680330","Type":"ContainerDied","Data":"79a3b6f61fdeb39ddafad1adf321ed5d04fffcfa30ae14389fbc418375c5446d"}
Jul 28 15:50:59.514040 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:50:59.513772    2568 scope.go:117] "RemoveContainer" containerID="cfa0c39877be81ff5714a0b33ec63eb317214d0db0787c05c86b7525199d433b"
Jul 28 15:50:59.521917 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:50:59.521899    2568 scope.go:117] "RemoveContainer" containerID="1fba9ebdfc404d34824238ab574b9d04efe079d473aab0e932395018b9ae8a3e"
Jul 28 15:50:59.530046 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:50:59.529889    2568 scope.go:117] "RemoveContainer" containerID="cfa0c39877be81ff5714a0b33ec63eb317214d0db0787c05c86b7525199d433b"
Jul 28 15:50:59.530203 ip-10-0-131-121 kubenswrapper[2568]: E0728 15:50:59.530183    2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cfa0c39877be81ff5714a0b33ec63eb317214d0db0787c05c86b7525199d433b\": container with ID starting with cfa0c39877be81ff5714a0b33ec63eb317214d0db0787c05c86b7525199d433b not found: ID does not exist" containerID="cfa0c39877be81ff5714a0b33ec63eb317214d0db0787c05c86b7525199d433b"
Jul 28 15:50:59.530243 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:50:59.530212    2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cfa0c39877be81ff5714a0b33ec63eb317214d0db0787c05c86b7525199d433b"} err="failed to get container status \"cfa0c39877be81ff5714a0b33ec63eb317214d0db0787c05c86b7525199d433b\": rpc error: code = NotFound desc = could not find container \"cfa0c39877be81ff5714a0b33ec63eb317214d0db0787c05c86b7525199d433b\": container with ID starting with cfa0c39877be81ff5714a0b33ec63eb317214d0db0787c05c86b7525199d433b not found: ID does not exist"
Jul 28 15:50:59.530243 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:50:59.530232    2568 scope.go:117] "RemoveContainer" containerID="1fba9ebdfc404d34824238ab574b9d04efe079d473aab0e932395018b9ae8a3e"
Jul 28 15:50:59.530486 ip-10-0-131-121 kubenswrapper[2568]: E0728 15:50:59.530467    2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1fba9ebdfc404d34824238ab574b9d04efe079d473aab0e932395018b9ae8a3e\": container with ID starting with 1fba9ebdfc404d34824238ab574b9d04efe079d473aab0e932395018b9ae8a3e not found: ID does not exist" containerID="1fba9ebdfc404d34824238ab574b9d04efe079d473aab0e932395018b9ae8a3e"
Jul 28 15:50:59.530523 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:50:59.530493    2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1fba9ebdfc404d34824238ab574b9d04efe079d473aab0e932395018b9ae8a3e"} err="failed to get container status \"1fba9ebdfc404d34824238ab574b9d04efe079d473aab0e932395018b9ae8a3e\": rpc error: code = NotFound desc = could not find container \"1fba9ebdfc404d34824238ab574b9d04efe079d473aab0e932395018b9ae8a3e\": container with ID starting with 1fba9ebdfc404d34824238ab574b9d04efe079d473aab0e932395018b9ae8a3e not found: ID does not exist"
Jul 28 15:50:59.530523 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:50:59.530510    2568 scope.go:117] "RemoveContainer" containerID="cfa0c39877be81ff5714a0b33ec63eb317214d0db0787c05c86b7525199d433b"
Jul 28 15:50:59.530749 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:50:59.530733    2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cfa0c39877be81ff5714a0b33ec63eb317214d0db0787c05c86b7525199d433b"} err="failed to get container status \"cfa0c39877be81ff5714a0b33ec63eb317214d0db0787c05c86b7525199d433b\": rpc error: code = NotFound desc = could not find container \"cfa0c39877be81ff5714a0b33ec63eb317214d0db0787c05c86b7525199d433b\": container with ID starting with cfa0c39877be81ff5714a0b33ec63eb317214d0db0787c05c86b7525199d433b not found: ID does not exist"
Jul 28 15:50:59.530801 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:50:59.530750    2568 scope.go:117] "RemoveContainer" containerID="1fba9ebdfc404d34824238ab574b9d04efe079d473aab0e932395018b9ae8a3e"
Jul 28 15:50:59.531031 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:50:59.531009    2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1fba9ebdfc404d34824238ab574b9d04efe079d473aab0e932395018b9ae8a3e"} err="failed to get container status \"1fba9ebdfc404d34824238ab574b9d04efe079d473aab0e932395018b9ae8a3e\": rpc error: code = NotFound desc = could not find container \"1fba9ebdfc404d34824238ab574b9d04efe079d473aab0e932395018b9ae8a3e\": container with ID starting with 1fba9ebdfc404d34824238ab574b9d04efe079d473aab0e932395018b9ae8a3e not found: ID does not exist"
Jul 28 15:50:59.537121 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:50:59.537096    2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/message-dumper-raw-22083-predictor-7fc855cfcc-kv9rd"]
Jul 28 15:50:59.540701 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:50:59.540680    2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/message-dumper-raw-22083-predictor-7fc855cfcc-kv9rd"]
Jul 28 15:50:59.896363 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:50:59.896280    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22nh9_ebf75c0d-1597-4a1c-b668-dc2a041eb35e/konnectivity-agent/0.log" limit=1024
Jul 28 15:51:00.294716 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:51:00.294678    2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0d8bf4a5-d7b5-4204-9890-7cace9680330" path="/var/lib/kubelet/pods/0d8bf4a5-d7b5-4204-9890-7cace9680330/volumes"
Jul 28 15:53:38.705418 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:53:38.705384    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22nh9_ebf75c0d-1597-4a1c-b668-dc2a041eb35e/konnectivity-agent/0.log" limit=1024
Jul 28 15:53:42.602632 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:53:42.602592    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22nh9_ebf75c0d-1597-4a1c-b668-dc2a041eb35e/konnectivity-agent/0.log" limit=1024
Jul 28 15:53:42.701620 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:53:42.701586    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22nh9_ebf75c0d-1597-4a1c-b668-dc2a041eb35e/konnectivity-agent/0.log" limit=1024
Jul 28 15:53:43.794346 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:53:43.794312    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22nh9_ebf75c0d-1597-4a1c-b668-dc2a041eb35e/konnectivity-agent/0.log" limit=1024
Jul 28 15:53:43.851599 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:53:43.851565    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22nh9_ebf75c0d-1597-4a1c-b668-dc2a041eb35e/konnectivity-agent/0.log" limit=1024
Jul 28 15:53:43.989241 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:53:43.989203    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22nh9_ebf75c0d-1597-4a1c-b668-dc2a041eb35e/konnectivity-agent/0.log" limit=1024
Jul 28 15:55:02.698559 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:55:02.698521    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22nh9_ebf75c0d-1597-4a1c-b668-dc2a041eb35e/konnectivity-agent/0.log" limit=1024
Jul 28 15:55:13.970756 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:55:13.970719    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22nh9_ebf75c0d-1597-4a1c-b668-dc2a041eb35e/konnectivity-agent/0.log" limit=1024
Jul 28 15:55:14.071198 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:55:14.071167    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22nh9_ebf75c0d-1597-4a1c-b668-dc2a041eb35e/konnectivity-agent/0.log" limit=1024
Jul 28 15:55:19.046006 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:55:19.045962    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22nh9_ebf75c0d-1597-4a1c-b668-dc2a041eb35e/konnectivity-agent/0.log" limit=1024
Jul 28 15:55:19.121635 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:55:19.121603    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22nh9_ebf75c0d-1597-4a1c-b668-dc2a041eb35e/konnectivity-agent/0.log" limit=1024
Jul 28 15:55:19.262649 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:55:19.262612    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22nh9_ebf75c0d-1597-4a1c-b668-dc2a041eb35e/konnectivity-agent/0.log" limit=1024
Jul 28 15:55:19.689300 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:55:19.689267    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22nh9_ebf75c0d-1597-4a1c-b668-dc2a041eb35e/konnectivity-agent/0.log" limit=1024
Jul 28 15:56:49.256511 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:56:49.256473    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22nh9_ebf75c0d-1597-4a1c-b668-dc2a041eb35e/konnectivity-agent/0.log" limit=1024
Jul 28 15:56:49.298104 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:56:49.298070    2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-w25q8"]
Jul 28 15:56:49.298446 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:56:49.298426    2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7cc2f9f3-d61c-470e-99ec-ddaef02da3aa" containerName="kserve-container"
Jul 28 15:56:49.298446 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:56:49.298442    2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="7cc2f9f3-d61c-470e-99ec-ddaef02da3aa" containerName="kserve-container"
Jul 28 15:56:49.298632 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:56:49.298453    2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ec023ae8-844f-48c3-abf7-15e62e25d89a" containerName="model-chainer-raw-hpa-c70d2"
Jul 28 15:56:49.298632 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:56:49.298461    2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="ec023ae8-844f-48c3-abf7-15e62e25d89a" containerName="model-chainer-raw-hpa-c70d2"
Jul 28 15:56:49.298632 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:56:49.298469    2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0d8bf4a5-d7b5-4204-9890-7cace9680330" containerName="kube-rbac-proxy"
Jul 28 15:56:49.298632 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:56:49.298475    2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="0d8bf4a5-d7b5-4204-9890-7cace9680330" containerName="kube-rbac-proxy"
Jul 28 15:56:49.298632 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:56:49.298483    2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0d8bf4a5-d7b5-4204-9890-7cace9680330" containerName="kserve-container"
Jul 28 15:56:49.298632 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:56:49.298488    2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="0d8bf4a5-d7b5-4204-9890-7cace9680330" containerName="kserve-container"
Jul 28 15:56:49.298632 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:56:49.298495    2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7cc2f9f3-d61c-470e-99ec-ddaef02da3aa" containerName="storage-initializer"
Jul 28 15:56:49.298632 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:56:49.298500    2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="7cc2f9f3-d61c-470e-99ec-ddaef02da3aa" containerName="storage-initializer"
Jul 28 15:56:49.298632 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:56:49.298508    2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7cc2f9f3-d61c-470e-99ec-ddaef02da3aa" containerName="kube-rbac-proxy"
Jul 28 15:56:49.298632 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:56:49.298513    2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="7cc2f9f3-d61c-470e-99ec-ddaef02da3aa" containerName="kube-rbac-proxy"
Jul 28 15:56:49.298632 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:56:49.298576    2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="ec023ae8-844f-48c3-abf7-15e62e25d89a" containerName="model-chainer-raw-hpa-c70d2"
Jul 28 15:56:49.298632 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:56:49.298585    2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="7cc2f9f3-d61c-470e-99ec-ddaef02da3aa" containerName="kube-rbac-proxy"
Jul 28 15:56:49.298632 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:56:49.298593    2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="0d8bf4a5-d7b5-4204-9890-7cace9680330" containerName="kube-rbac-proxy"
Jul 28 15:56:49.298632 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:56:49.298602    2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="0d8bf4a5-d7b5-4204-9890-7cace9680330" containerName="kserve-container"
Jul 28 15:56:49.298632 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:56:49.298610    2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="7cc2f9f3-d61c-470e-99ec-ddaef02da3aa" containerName="kserve-container"
Jul 28 15:56:49.302071 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:56:49.302054    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-w25q8"
Jul 28 15:56:49.304790 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:56:49.304748    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-promote-v2-predictor-serving-cert\""
Jul 28 15:56:49.304919 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:56:49.304812    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 28 15:56:49.304919 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:56:49.304824    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-promote-kube-rbac-proxy-sar-config\""
Jul 28 15:56:49.304919 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:56:49.304903    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 28 15:56:49.305047 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:56:49.304967    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 28 15:56:49.305892 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:56:49.305876    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 28 15:56:49.305946 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:56:49.305895    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-gcwwc\""
Jul 28 15:56:49.313103 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:56:49.313084    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-w25q8"]
Jul 28 15:56:49.431281 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:56:49.431251    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/69f19a07-b0f5-491c-996d-f842c7ca6bba-cabundle-cert\") pod \"isvc-canary-promote-v2-predictor-86766c4958-w25q8\" (UID: \"69f19a07-b0f5-491c-996d-f842c7ca6bba\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-w25q8"
Jul 28 15:56:49.431447 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:56:49.431289    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/69f19a07-b0f5-491c-996d-f842c7ca6bba-proxy-tls\") pod \"isvc-canary-promote-v2-predictor-86766c4958-w25q8\" (UID: \"69f19a07-b0f5-491c-996d-f842c7ca6bba\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-w25q8"
Jul 28 15:56:49.431447 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:56:49.431316    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/69f19a07-b0f5-491c-996d-f842c7ca6bba-kserve-provision-location\") pod \"isvc-canary-promote-v2-predictor-86766c4958-w25q8\" (UID: \"69f19a07-b0f5-491c-996d-f842c7ca6bba\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-w25q8"
Jul 28 15:56:49.431447 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:56:49.431342    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p5mmk\" (UniqueName: \"kubernetes.io/projected/69f19a07-b0f5-491c-996d-f842c7ca6bba-kube-api-access-p5mmk\") pod \"isvc-canary-promote-v2-predictor-86766c4958-w25q8\" (UID: \"69f19a07-b0f5-491c-996d-f842c7ca6bba\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-w25q8"
Jul 28 15:56:49.431447 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:56:49.431427    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/69f19a07-b0f5-491c-996d-f842c7ca6bba-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-v2-predictor-86766c4958-w25q8\" (UID: \"69f19a07-b0f5-491c-996d-f842c7ca6bba\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-w25q8"
Jul 28 15:56:49.532212 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:56:49.532173    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p5mmk\" (UniqueName: \"kubernetes.io/projected/69f19a07-b0f5-491c-996d-f842c7ca6bba-kube-api-access-p5mmk\") pod \"isvc-canary-promote-v2-predictor-86766c4958-w25q8\" (UID: \"69f19a07-b0f5-491c-996d-f842c7ca6bba\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-w25q8"
Jul 28 15:56:49.532468 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:56:49.532225    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/69f19a07-b0f5-491c-996d-f842c7ca6bba-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-v2-predictor-86766c4958-w25q8\" (UID: \"69f19a07-b0f5-491c-996d-f842c7ca6bba\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-w25q8"
Jul 28 15:56:49.532468 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:56:49.532270    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/69f19a07-b0f5-491c-996d-f842c7ca6bba-cabundle-cert\") pod \"isvc-canary-promote-v2-predictor-86766c4958-w25q8\" (UID: \"69f19a07-b0f5-491c-996d-f842c7ca6bba\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-w25q8"
Jul 28 15:56:49.532468 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:56:49.532290    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/69f19a07-b0f5-491c-996d-f842c7ca6bba-proxy-tls\") pod \"isvc-canary-promote-v2-predictor-86766c4958-w25q8\" (UID: \"69f19a07-b0f5-491c-996d-f842c7ca6bba\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-w25q8"
Jul 28 15:56:49.532468 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:56:49.532316    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/69f19a07-b0f5-491c-996d-f842c7ca6bba-kserve-provision-location\") pod \"isvc-canary-promote-v2-predictor-86766c4958-w25q8\" (UID: \"69f19a07-b0f5-491c-996d-f842c7ca6bba\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-w25q8"
Jul 28 15:56:49.532836 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:56:49.532806    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/69f19a07-b0f5-491c-996d-f842c7ca6bba-kserve-provision-location\") pod \"isvc-canary-promote-v2-predictor-86766c4958-w25q8\" (UID: \"69f19a07-b0f5-491c-996d-f842c7ca6bba\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-w25q8"
Jul 28 15:56:49.533010 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:56:49.532993    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/69f19a07-b0f5-491c-996d-f842c7ca6bba-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-v2-predictor-86766c4958-w25q8\" (UID: \"69f19a07-b0f5-491c-996d-f842c7ca6bba\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-w25q8"
Jul 28 15:56:49.533053 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:56:49.533019    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/69f19a07-b0f5-491c-996d-f842c7ca6bba-cabundle-cert\") pod \"isvc-canary-promote-v2-predictor-86766c4958-w25q8\" (UID: \"69f19a07-b0f5-491c-996d-f842c7ca6bba\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-w25q8"
Jul 28 15:56:49.534756 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:56:49.534739    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/69f19a07-b0f5-491c-996d-f842c7ca6bba-proxy-tls\") pod \"isvc-canary-promote-v2-predictor-86766c4958-w25q8\" (UID: \"69f19a07-b0f5-491c-996d-f842c7ca6bba\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-w25q8"
Jul 28 15:56:49.542634 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:56:49.542610    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p5mmk\" (UniqueName: \"kubernetes.io/projected/69f19a07-b0f5-491c-996d-f842c7ca6bba-kube-api-access-p5mmk\") pod \"isvc-canary-promote-v2-predictor-86766c4958-w25q8\" (UID: \"69f19a07-b0f5-491c-996d-f842c7ca6bba\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-w25q8"
Jul 28 15:56:49.612505 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:56:49.612459    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-w25q8"
Jul 28 15:56:49.748427 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:56:49.748286    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-w25q8"]
Jul 28 15:56:49.753984 ip-10-0-131-121 kubenswrapper[2568]: W0728 15:56:49.753948    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod69f19a07_b0f5_491c_996d_f842c7ca6bba.slice/crio-08668c57fdaca5af00a5b3a3cea076d6ad32646adb8f66a27407799eefd051af WatchSource:0}: Error finding container 08668c57fdaca5af00a5b3a3cea076d6ad32646adb8f66a27407799eefd051af: Status 404 returned error can't find the container with id 08668c57fdaca5af00a5b3a3cea076d6ad32646adb8f66a27407799eefd051af
Jul 28 15:56:49.889469 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:56:49.889423    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-w25q8" event={"ID":"69f19a07-b0f5-491c-996d-f842c7ca6bba","Type":"ContainerStarted","Data":"bcda21cdef5aaada5a08c64f1696c7117ac521aa744cd5e45b4aa02dcaa0e7cf"}
Jul 28 15:56:49.889717 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:56:49.889480    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-w25q8" event={"ID":"69f19a07-b0f5-491c-996d-f842c7ca6bba","Type":"ContainerStarted","Data":"08668c57fdaca5af00a5b3a3cea076d6ad32646adb8f66a27407799eefd051af"}
Jul 28 15:56:53.906734 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:56:53.906701    2568 generic.go:356] "Generic (PLEG): container finished" podID="69f19a07-b0f5-491c-996d-f842c7ca6bba" containerID="bcda21cdef5aaada5a08c64f1696c7117ac521aa744cd5e45b4aa02dcaa0e7cf" exitCode=0
Jul 28 15:56:53.907152 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:56:53.906790    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-w25q8" event={"ID":"69f19a07-b0f5-491c-996d-f842c7ca6bba","Type":"ContainerDied","Data":"bcda21cdef5aaada5a08c64f1696c7117ac521aa744cd5e45b4aa02dcaa0e7cf"}
Jul 28 15:56:53.907912 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:56:53.907897    2568 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 28 15:57:07.979657 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:57:07.979622    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-w25q8" event={"ID":"69f19a07-b0f5-491c-996d-f842c7ca6bba","Type":"ContainerStarted","Data":"e1ec3531a7be8d9e838b2e2e801ae3d11106848a5db8bba05de08a9c06293dd6"}
Jul 28 15:57:07.980112 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:57:07.979684    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-w25q8" event={"ID":"69f19a07-b0f5-491c-996d-f842c7ca6bba","Type":"ContainerStarted","Data":"338f28ccbdd346a4093e746b2aff48a67016f2de457105036edbde41552fe96f"}
Jul 28 15:57:07.980112 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:57:07.979977    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-w25q8"
Jul 28 15:57:07.980218 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:57:07.980131    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-w25q8"
Jul 28 15:57:07.981415 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:57:07.981390    2568 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-w25q8" podUID="69f19a07-b0f5-491c-996d-f842c7ca6bba" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused"
Jul 28 15:57:08.006104 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:57:08.006040    2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-w25q8" podStartSLOduration=1.842199409 podStartE2EDuration="19.00601871s" totalImagesPullingTime="13.259803785s" totalInitContainerRuntime="3.904015516s" isStatefulPod=false podCreationTimestamp="2026-07-28 15:56:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 15:57:08.004557147 +0000 UTC m=+1408.189752315" watchObservedRunningTime="2026-07-28 15:57:08.00601871 +0000 UTC m=+1408.191213880"
Jul 28 15:57:08.984611 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:57:08.984570    2568 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-w25q8" podUID="69f19a07-b0f5-491c-996d-f842c7ca6bba" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused"
Jul 28 15:57:13.988953 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:57:13.988926    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-w25q8"
Jul 28 15:57:13.989481 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:57:13.989456    2568 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-w25q8" podUID="69f19a07-b0f5-491c-996d-f842c7ca6bba" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused"
Jul 28 15:57:23.989741 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:57:23.989700    2568 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-w25q8" podUID="69f19a07-b0f5-491c-996d-f842c7ca6bba" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused"
Jul 28 15:57:26.906813 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:57:26.906739    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22nh9_ebf75c0d-1597-4a1c-b668-dc2a041eb35e/konnectivity-agent/0.log" limit=1024
Jul 28 15:57:33.989723 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:57:33.989636    2568 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-w25q8" podUID="69f19a07-b0f5-491c-996d-f842c7ca6bba" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused"
Jul 28 15:57:43.989430 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:57:43.989391    2568 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-w25q8" podUID="69f19a07-b0f5-491c-996d-f842c7ca6bba" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused"
Jul 28 15:57:53.989578 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:57:53.989538    2568 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-w25q8" podUID="69f19a07-b0f5-491c-996d-f842c7ca6bba" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused"
Jul 28 15:58:03.989598 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:58:03.989554    2568 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-w25q8" podUID="69f19a07-b0f5-491c-996d-f842c7ca6bba" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused"
Jul 28 15:58:13.989628 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:58:13.989590    2568 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-w25q8" podUID="69f19a07-b0f5-491c-996d-f842c7ca6bba" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused"
Jul 28 15:58:23.990641 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:58:23.990610    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-w25q8"
Jul 28 15:58:29.483378 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:58:29.483334    2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-w25q8"]
Jul 28 15:58:29.484205 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:58:29.483639    2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-w25q8" podUID="69f19a07-b0f5-491c-996d-f842c7ca6bba" containerName="kserve-container" containerID="cri-o://338f28ccbdd346a4093e746b2aff48a67016f2de457105036edbde41552fe96f" gracePeriod=30
Jul 28 15:58:29.484205 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:58:29.483678    2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-w25q8" podUID="69f19a07-b0f5-491c-996d-f842c7ca6bba" containerName="kube-rbac-proxy" containerID="cri-o://e1ec3531a7be8d9e838b2e2e801ae3d11106848a5db8bba05de08a9c06293dd6" gracePeriod=30
Jul 28 15:58:30.306477 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:58:30.306446    2568 generic.go:356] "Generic (PLEG): container finished" podID="69f19a07-b0f5-491c-996d-f842c7ca6bba" containerID="e1ec3531a7be8d9e838b2e2e801ae3d11106848a5db8bba05de08a9c06293dd6" exitCode=2
Jul 28 15:58:30.306650 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:58:30.306525    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-w25q8" event={"ID":"69f19a07-b0f5-491c-996d-f842c7ca6bba","Type":"ContainerDied","Data":"e1ec3531a7be8d9e838b2e2e801ae3d11106848a5db8bba05de08a9c06293dd6"}
Jul 28 15:58:30.517440 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:58:30.517405    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22nh9_ebf75c0d-1597-4a1c-b668-dc2a041eb35e/konnectivity-agent/0.log" limit=1024
Jul 28 15:58:33.985738 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:58:33.985690    2568 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-w25q8" podUID="69f19a07-b0f5-491c-996d-f842c7ca6bba" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.29:8643/healthz\": dial tcp 10.134.0.29:8643: connect: connection refused"
Jul 28 15:58:33.990047 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:58:33.990017    2568 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-w25q8" podUID="69f19a07-b0f5-491c-996d-f842c7ca6bba" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused"
Jul 28 15:58:34.642796 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:58:34.642754    2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-w25q8"
Jul 28 15:58:34.710186 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:58:34.710091    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/69f19a07-b0f5-491c-996d-f842c7ca6bba-kube-rbac-proxy-sar-config\") pod \"69f19a07-b0f5-491c-996d-f842c7ca6bba\" (UID: \"69f19a07-b0f5-491c-996d-f842c7ca6bba\") "
Jul 28 15:58:34.710186 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:58:34.710140    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/69f19a07-b0f5-491c-996d-f842c7ca6bba-cabundle-cert\") pod \"69f19a07-b0f5-491c-996d-f842c7ca6bba\" (UID: \"69f19a07-b0f5-491c-996d-f842c7ca6bba\") "
Jul 28 15:58:34.710186 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:58:34.710182    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/69f19a07-b0f5-491c-996d-f842c7ca6bba-kserve-provision-location\") pod \"69f19a07-b0f5-491c-996d-f842c7ca6bba\" (UID: \"69f19a07-b0f5-491c-996d-f842c7ca6bba\") "
Jul 28 15:58:34.710551 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:58:34.710272    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/69f19a07-b0f5-491c-996d-f842c7ca6bba-proxy-tls\") pod \"69f19a07-b0f5-491c-996d-f842c7ca6bba\" (UID: \"69f19a07-b0f5-491c-996d-f842c7ca6bba\") "
Jul 28 15:58:34.710551 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:58:34.710335    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-p5mmk\" (UniqueName: \"kubernetes.io/projected/69f19a07-b0f5-491c-996d-f842c7ca6bba-kube-api-access-p5mmk\") pod \"69f19a07-b0f5-491c-996d-f842c7ca6bba\" (UID: \"69f19a07-b0f5-491c-996d-f842c7ca6bba\") "
Jul 28 15:58:34.710551 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:58:34.710480    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/69f19a07-b0f5-491c-996d-f842c7ca6bba-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "69f19a07-b0f5-491c-996d-f842c7ca6bba" (UID: "69f19a07-b0f5-491c-996d-f842c7ca6bba"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 15:58:34.710551 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:58:34.710488    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/69f19a07-b0f5-491c-996d-f842c7ca6bba-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "69f19a07-b0f5-491c-996d-f842c7ca6bba" (UID: "69f19a07-b0f5-491c-996d-f842c7ca6bba"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 15:58:34.710752 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:58:34.710594    2568 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/69f19a07-b0f5-491c-996d-f842c7ca6bba-kube-rbac-proxy-sar-config\") on node \"ip-10-0-131-121.ec2.internal\" DevicePath \"\""
Jul 28 15:58:34.710752 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:58:34.710611    2568 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/69f19a07-b0f5-491c-996d-f842c7ca6bba-cabundle-cert\") on node \"ip-10-0-131-121.ec2.internal\" DevicePath \"\""
Jul 28 15:58:34.710752 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:58:34.710674    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/69f19a07-b0f5-491c-996d-f842c7ca6bba-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "69f19a07-b0f5-491c-996d-f842c7ca6bba" (UID: "69f19a07-b0f5-491c-996d-f842c7ca6bba"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 15:58:34.712594 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:58:34.712567    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/69f19a07-b0f5-491c-996d-f842c7ca6bba-kube-api-access-p5mmk" (OuterVolumeSpecName: "kube-api-access-p5mmk") pod "69f19a07-b0f5-491c-996d-f842c7ca6bba" (UID: "69f19a07-b0f5-491c-996d-f842c7ca6bba"). InnerVolumeSpecName "kube-api-access-p5mmk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 15:58:34.713129 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:58:34.713103    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/69f19a07-b0f5-491c-996d-f842c7ca6bba-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "69f19a07-b0f5-491c-996d-f842c7ca6bba" (UID: "69f19a07-b0f5-491c-996d-f842c7ca6bba"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 15:58:34.751554 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:58:34.751524    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22nh9_ebf75c0d-1597-4a1c-b668-dc2a041eb35e/konnectivity-agent/0.log" limit=1024
Jul 28 15:58:34.800399 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:58:34.800365    2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-qnljc"]
Jul 28 15:58:34.800696 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:58:34.800683    2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="69f19a07-b0f5-491c-996d-f842c7ca6bba" containerName="storage-initializer"
Jul 28 15:58:34.800757 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:58:34.800697    2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="69f19a07-b0f5-491c-996d-f842c7ca6bba" containerName="storage-initializer"
Jul 28 15:58:34.800757 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:58:34.800705    2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="69f19a07-b0f5-491c-996d-f842c7ca6bba" containerName="kserve-container"
Jul 28 15:58:34.800757 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:58:34.800711    2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="69f19a07-b0f5-491c-996d-f842c7ca6bba" containerName="kserve-container"
Jul 28 15:58:34.800757 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:58:34.800725    2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="69f19a07-b0f5-491c-996d-f842c7ca6bba" containerName="kube-rbac-proxy"
Jul 28 15:58:34.800757 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:58:34.800731    2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="69f19a07-b0f5-491c-996d-f842c7ca6bba" containerName="kube-rbac-proxy"
Jul 28 15:58:34.801023 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:58:34.800795    2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="69f19a07-b0f5-491c-996d-f842c7ca6bba" containerName="kube-rbac-proxy"
Jul 28 15:58:34.801023 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:58:34.800808    2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="69f19a07-b0f5-491c-996d-f842c7ca6bba" containerName="kserve-container"
Jul 28 15:58:34.804097 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:58:34.804079    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-qnljc"
Jul 28 15:58:34.807535 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:58:34.807510    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-rollback-kube-rbac-proxy-sar-config\""
Jul 28 15:58:34.807697 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:58:34.807513    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-rollback-predictor-serving-cert\""
Jul 28 15:58:34.811022 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:58:34.810999    2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p5mmk\" (UniqueName: \"kubernetes.io/projected/69f19a07-b0f5-491c-996d-f842c7ca6bba-kube-api-access-p5mmk\") on node \"ip-10-0-131-121.ec2.internal\" DevicePath \"\""
Jul 28 15:58:34.811022 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:58:34.811020    2568 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/69f19a07-b0f5-491c-996d-f842c7ca6bba-kserve-provision-location\") on node \"ip-10-0-131-121.ec2.internal\" DevicePath \"\""
Jul 28 15:58:34.811211 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:58:34.811032    2568 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/69f19a07-b0f5-491c-996d-f842c7ca6bba-proxy-tls\") on node \"ip-10-0-131-121.ec2.internal\" DevicePath \"\""
Jul 28 15:58:34.821079 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:58:34.821024    2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-84f48b7656-h2tj5"]
Jul 28 15:58:34.825158 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:58:34.825124    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-qnljc"]
Jul 28 15:58:34.825309 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:58:34.825239    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-84f48b7656-h2tj5"
Jul 28 15:58:34.827757 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:58:34.827732    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-rollback-v2-predictor-serving-cert\""
Jul 28 15:58:34.844468 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:58:34.844440    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-84f48b7656-h2tj5"]
Jul 28 15:58:34.862277 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:58:34.862248    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22nh9_ebf75c0d-1597-4a1c-b668-dc2a041eb35e/konnectivity-agent/0.log" limit=1024
Jul 28 15:58:34.911784 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:58:34.911730    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ceba507d-2395-4695-b8f8-72816fd44c52-kserve-provision-location\") pod \"isvc-canary-rollback-predictor-5bbb8c8cfc-qnljc\" (UID: \"ceba507d-2395-4695-b8f8-72816fd44c52\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-qnljc"
Jul 28 15:58:34.911984 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:58:34.911816    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x28nh\" (UniqueName: \"kubernetes.io/projected/ceba507d-2395-4695-b8f8-72816fd44c52-kube-api-access-x28nh\") pod \"isvc-canary-rollback-predictor-5bbb8c8cfc-qnljc\" (UID: \"ceba507d-2395-4695-b8f8-72816fd44c52\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-qnljc"
Jul 28 15:58:34.911984 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:58:34.911854    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7lg4x\" (UniqueName: \"kubernetes.io/projected/383b7313-e6b6-4fc4-8b66-b0d9a6eb4a81-kube-api-access-7lg4x\") pod \"isvc-canary-rollback-v2-predictor-84f48b7656-h2tj5\" (UID: \"383b7313-e6b6-4fc4-8b66-b0d9a6eb4a81\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-84f48b7656-h2tj5"
Jul 28 15:58:34.911984 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:58:34.911913    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/383b7313-e6b6-4fc4-8b66-b0d9a6eb4a81-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-v2-predictor-84f48b7656-h2tj5\" (UID: \"383b7313-e6b6-4fc4-8b66-b0d9a6eb4a81\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-84f48b7656-h2tj5"
Jul 28 15:58:34.911984 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:58:34.911946    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ceba507d-2395-4695-b8f8-72816fd44c52-proxy-tls\") pod \"isvc-canary-rollback-predictor-5bbb8c8cfc-qnljc\" (UID: \"ceba507d-2395-4695-b8f8-72816fd44c52\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-qnljc"
Jul 28 15:58:34.911984 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:58:34.911979    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ceba507d-2395-4695-b8f8-72816fd44c52-cabundle-cert\") pod \"isvc-canary-rollback-predictor-5bbb8c8cfc-qnljc\" (UID: \"ceba507d-2395-4695-b8f8-72816fd44c52\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-qnljc"
Jul 28 15:58:34.912206 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:58:34.912017    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/383b7313-e6b6-4fc4-8b66-b0d9a6eb4a81-cabundle-cert\") pod \"isvc-canary-rollback-v2-predictor-84f48b7656-h2tj5\" (UID: \"383b7313-e6b6-4fc4-8b66-b0d9a6eb4a81\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-84f48b7656-h2tj5"
Jul 28 15:58:34.912206 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:58:34.912041    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ceba507d-2395-4695-b8f8-72816fd44c52-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-predictor-5bbb8c8cfc-qnljc\" (UID: \"ceba507d-2395-4695-b8f8-72816fd44c52\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-qnljc"
Jul 28 15:58:34.912206 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:58:34.912073    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/383b7313-e6b6-4fc4-8b66-b0d9a6eb4a81-kserve-provision-location\") pod \"isvc-canary-rollback-v2-predictor-84f48b7656-h2tj5\" (UID: \"383b7313-e6b6-4fc4-8b66-b0d9a6eb4a81\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-84f48b7656-h2tj5"
Jul 28 15:58:34.912206 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:58:34.912137    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/383b7313-e6b6-4fc4-8b66-b0d9a6eb4a81-proxy-tls\") pod \"isvc-canary-rollback-v2-predictor-84f48b7656-h2tj5\" (UID: \"383b7313-e6b6-4fc4-8b66-b0d9a6eb4a81\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-84f48b7656-h2tj5"
Jul 28 15:58:35.013329 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:58:35.013299    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ceba507d-2395-4695-b8f8-72816fd44c52-kserve-provision-location\") pod \"isvc-canary-rollback-predictor-5bbb8c8cfc-qnljc\" (UID: \"ceba507d-2395-4695-b8f8-72816fd44c52\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-qnljc"
Jul 28 15:58:35.013875 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:58:35.013342    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x28nh\" (UniqueName: \"kubernetes.io/projected/ceba507d-2395-4695-b8f8-72816fd44c52-kube-api-access-x28nh\") pod \"isvc-canary-rollback-predictor-5bbb8c8cfc-qnljc\" (UID: \"ceba507d-2395-4695-b8f8-72816fd44c52\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-qnljc"
Jul 28 15:58:35.013875 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:58:35.013366    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7lg4x\" (UniqueName: \"kubernetes.io/projected/383b7313-e6b6-4fc4-8b66-b0d9a6eb4a81-kube-api-access-7lg4x\") pod \"isvc-canary-rollback-v2-predictor-84f48b7656-h2tj5\" (UID: \"383b7313-e6b6-4fc4-8b66-b0d9a6eb4a81\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-84f48b7656-h2tj5"
Jul 28 15:58:35.013875 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:58:35.013393    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/383b7313-e6b6-4fc4-8b66-b0d9a6eb4a81-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-v2-predictor-84f48b7656-h2tj5\" (UID: \"383b7313-e6b6-4fc4-8b66-b0d9a6eb4a81\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-84f48b7656-h2tj5"
Jul 28 15:58:35.013875 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:58:35.013423    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ceba507d-2395-4695-b8f8-72816fd44c52-proxy-tls\") pod \"isvc-canary-rollback-predictor-5bbb8c8cfc-qnljc\" (UID: \"ceba507d-2395-4695-b8f8-72816fd44c52\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-qnljc"
Jul 28 15:58:35.013875 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:58:35.013450    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ceba507d-2395-4695-b8f8-72816fd44c52-cabundle-cert\") pod \"isvc-canary-rollback-predictor-5bbb8c8cfc-qnljc\" (UID: \"ceba507d-2395-4695-b8f8-72816fd44c52\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-qnljc"
Jul 28 15:58:35.013875 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:58:35.013487    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/383b7313-e6b6-4fc4-8b66-b0d9a6eb4a81-cabundle-cert\") pod \"isvc-canary-rollback-v2-predictor-84f48b7656-h2tj5\" (UID: \"383b7313-e6b6-4fc4-8b66-b0d9a6eb4a81\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-84f48b7656-h2tj5"
Jul 28 15:58:35.013875 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:58:35.013509    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ceba507d-2395-4695-b8f8-72816fd44c52-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-predictor-5bbb8c8cfc-qnljc\" (UID: \"ceba507d-2395-4695-b8f8-72816fd44c52\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-qnljc"
Jul 28 15:58:35.013875 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:58:35.013544    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/383b7313-e6b6-4fc4-8b66-b0d9a6eb4a81-kserve-provision-location\") pod \"isvc-canary-rollback-v2-predictor-84f48b7656-h2tj5\" (UID: \"383b7313-e6b6-4fc4-8b66-b0d9a6eb4a81\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-84f48b7656-h2tj5"
Jul 28 15:58:35.013875 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:58:35.013569    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/383b7313-e6b6-4fc4-8b66-b0d9a6eb4a81-proxy-tls\") pod \"isvc-canary-rollback-v2-predictor-84f48b7656-h2tj5\" (UID: \"383b7313-e6b6-4fc4-8b66-b0d9a6eb4a81\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-84f48b7656-h2tj5"
Jul 28 15:58:35.013875 ip-10-0-131-121 kubenswrapper[2568]: E0728 15:58:35.013675    2568 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-serving-cert: secret "isvc-canary-rollback-v2-predictor-serving-cert" not found
Jul 28 15:58:35.013875 ip-10-0-131-121 kubenswrapper[2568]: E0728 15:58:35.013736    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/383b7313-e6b6-4fc4-8b66-b0d9a6eb4a81-proxy-tls podName:383b7313-e6b6-4fc4-8b66-b0d9a6eb4a81 nodeName:}" failed. No retries permitted until 2026-07-28 15:58:35.513715872 +0000 UTC m=+1495.698911026 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/383b7313-e6b6-4fc4-8b66-b0d9a6eb4a81-proxy-tls") pod "isvc-canary-rollback-v2-predictor-84f48b7656-h2tj5" (UID: "383b7313-e6b6-4fc4-8b66-b0d9a6eb4a81") : secret "isvc-canary-rollback-v2-predictor-serving-cert" not found
Jul 28 15:58:35.013875 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:58:35.013820    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ceba507d-2395-4695-b8f8-72816fd44c52-kserve-provision-location\") pod \"isvc-canary-rollback-predictor-5bbb8c8cfc-qnljc\" (UID: \"ceba507d-2395-4695-b8f8-72816fd44c52\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-qnljc"
Jul 28 15:58:35.014505 ip-10-0-131-121 kubenswrapper[2568]: E0728 15:58:35.013952    2568 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-canary-rollback-predictor-serving-cert: secret "isvc-canary-rollback-predictor-serving-cert" not found
Jul 28 15:58:35.014505 ip-10-0-131-121 kubenswrapper[2568]: E0728 15:58:35.014016    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ceba507d-2395-4695-b8f8-72816fd44c52-proxy-tls podName:ceba507d-2395-4695-b8f8-72816fd44c52 nodeName:}" failed. No retries permitted until 2026-07-28 15:58:35.513999752 +0000 UTC m=+1495.699194911 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/ceba507d-2395-4695-b8f8-72816fd44c52-proxy-tls") pod "isvc-canary-rollback-predictor-5bbb8c8cfc-qnljc" (UID: "ceba507d-2395-4695-b8f8-72816fd44c52") : secret "isvc-canary-rollback-predictor-serving-cert" not found
Jul 28 15:58:35.014505 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:58:35.014107    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/383b7313-e6b6-4fc4-8b66-b0d9a6eb4a81-kserve-provision-location\") pod \"isvc-canary-rollback-v2-predictor-84f48b7656-h2tj5\" (UID: \"383b7313-e6b6-4fc4-8b66-b0d9a6eb4a81\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-84f48b7656-h2tj5"
Jul 28 15:58:35.014505 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:58:35.014200    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/383b7313-e6b6-4fc4-8b66-b0d9a6eb4a81-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-v2-predictor-84f48b7656-h2tj5\" (UID: \"383b7313-e6b6-4fc4-8b66-b0d9a6eb4a81\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-84f48b7656-h2tj5"
Jul 28 15:58:35.014505 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:58:35.014227    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/383b7313-e6b6-4fc4-8b66-b0d9a6eb4a81-cabundle-cert\") pod \"isvc-canary-rollback-v2-predictor-84f48b7656-h2tj5\" (UID: \"383b7313-e6b6-4fc4-8b66-b0d9a6eb4a81\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-84f48b7656-h2tj5"
Jul 28 15:58:35.014505 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:58:35.014434    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ceba507d-2395-4695-b8f8-72816fd44c52-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-predictor-5bbb8c8cfc-qnljc\" (UID: \"ceba507d-2395-4695-b8f8-72816fd44c52\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-qnljc"
Jul 28 15:58:35.014505 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:58:35.014471    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ceba507d-2395-4695-b8f8-72816fd44c52-cabundle-cert\") pod \"isvc-canary-rollback-predictor-5bbb8c8cfc-qnljc\" (UID: \"ceba507d-2395-4695-b8f8-72816fd44c52\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-qnljc"
Jul 28 15:58:35.022366 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:58:35.022344    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x28nh\" (UniqueName: \"kubernetes.io/projected/ceba507d-2395-4695-b8f8-72816fd44c52-kube-api-access-x28nh\") pod \"isvc-canary-rollback-predictor-5bbb8c8cfc-qnljc\" (UID: \"ceba507d-2395-4695-b8f8-72816fd44c52\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-qnljc"
Jul 28 15:58:35.022601 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:58:35.022582    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7lg4x\" (UniqueName: \"kubernetes.io/projected/383b7313-e6b6-4fc4-8b66-b0d9a6eb4a81-kube-api-access-7lg4x\") pod \"isvc-canary-rollback-v2-predictor-84f48b7656-h2tj5\" (UID: \"383b7313-e6b6-4fc4-8b66-b0d9a6eb4a81\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-84f48b7656-h2tj5"
Jul 28 15:58:35.177951 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:58:35.177915    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22nh9_ebf75c0d-1597-4a1c-b668-dc2a041eb35e/konnectivity-agent/0.log" limit=1024
Jul 28 15:58:35.307616 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:58:35.307539    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22nh9_ebf75c0d-1597-4a1c-b668-dc2a041eb35e/konnectivity-agent/0.log" limit=1024
Jul 28 15:58:35.327678 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:58:35.327642    2568 generic.go:356] "Generic (PLEG): container finished" podID="69f19a07-b0f5-491c-996d-f842c7ca6bba" containerID="338f28ccbdd346a4093e746b2aff48a67016f2de457105036edbde41552fe96f" exitCode=0
Jul 28 15:58:35.327865 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:58:35.327705    2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-w25q8"
Jul 28 15:58:35.327865 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:58:35.327710    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-w25q8" event={"ID":"69f19a07-b0f5-491c-996d-f842c7ca6bba","Type":"ContainerDied","Data":"338f28ccbdd346a4093e746b2aff48a67016f2de457105036edbde41552fe96f"}
Jul 28 15:58:35.327865 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:58:35.327788    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-w25q8" event={"ID":"69f19a07-b0f5-491c-996d-f842c7ca6bba","Type":"ContainerDied","Data":"08668c57fdaca5af00a5b3a3cea076d6ad32646adb8f66a27407799eefd051af"}
Jul 28 15:58:35.327865 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:58:35.327828    2568 scope.go:117] "RemoveContainer" containerID="e1ec3531a7be8d9e838b2e2e801ae3d11106848a5db8bba05de08a9c06293dd6"
Jul 28 15:58:35.336346 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:58:35.336324    2568 scope.go:117] "RemoveContainer" containerID="338f28ccbdd346a4093e746b2aff48a67016f2de457105036edbde41552fe96f"
Jul 28 15:58:35.344258 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:58:35.344241    2568 scope.go:117] "RemoveContainer" containerID="bcda21cdef5aaada5a08c64f1696c7117ac521aa744cd5e45b4aa02dcaa0e7cf"
Jul 28 15:58:35.352493 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:58:35.352464    2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-w25q8"]
Jul 28 15:58:35.352846 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:58:35.352830    2568 scope.go:117] "RemoveContainer" containerID="e1ec3531a7be8d9e838b2e2e801ae3d11106848a5db8bba05de08a9c06293dd6"
Jul 28 15:58:35.353139 ip-10-0-131-121 kubenswrapper[2568]: E0728 15:58:35.353110    2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e1ec3531a7be8d9e838b2e2e801ae3d11106848a5db8bba05de08a9c06293dd6\": container with ID starting with e1ec3531a7be8d9e838b2e2e801ae3d11106848a5db8bba05de08a9c06293dd6 not found: ID does not exist" containerID="e1ec3531a7be8d9e838b2e2e801ae3d11106848a5db8bba05de08a9c06293dd6"
Jul 28 15:58:35.353213 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:58:35.353149    2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e1ec3531a7be8d9e838b2e2e801ae3d11106848a5db8bba05de08a9c06293dd6"} err="failed to get container status \"e1ec3531a7be8d9e838b2e2e801ae3d11106848a5db8bba05de08a9c06293dd6\": rpc error: code = NotFound desc = could not find container \"e1ec3531a7be8d9e838b2e2e801ae3d11106848a5db8bba05de08a9c06293dd6\": container with ID starting with e1ec3531a7be8d9e838b2e2e801ae3d11106848a5db8bba05de08a9c06293dd6 not found: ID does not exist"
Jul 28 15:58:35.353213 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:58:35.353169    2568 scope.go:117] "RemoveContainer" containerID="338f28ccbdd346a4093e746b2aff48a67016f2de457105036edbde41552fe96f"
Jul 28 15:58:35.353422 ip-10-0-131-121 kubenswrapper[2568]: E0728 15:58:35.353405    2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"338f28ccbdd346a4093e746b2aff48a67016f2de457105036edbde41552fe96f\": container with ID starting with 338f28ccbdd346a4093e746b2aff48a67016f2de457105036edbde41552fe96f not found: ID does not exist" containerID="338f28ccbdd346a4093e746b2aff48a67016f2de457105036edbde41552fe96f"
Jul 28 15:58:35.353465 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:58:35.353428    2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"338f28ccbdd346a4093e746b2aff48a67016f2de457105036edbde41552fe96f"} err="failed to get container status \"338f28ccbdd346a4093e746b2aff48a67016f2de457105036edbde41552fe96f\": rpc error: code = NotFound desc = could not find container \"338f28ccbdd346a4093e746b2aff48a67016f2de457105036edbde41552fe96f\": container with ID starting with 338f28ccbdd346a4093e746b2aff48a67016f2de457105036edbde41552fe96f not found: ID does not exist"
Jul 28 15:58:35.353465 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:58:35.353440    2568 scope.go:117] "RemoveContainer" containerID="bcda21cdef5aaada5a08c64f1696c7117ac521aa744cd5e45b4aa02dcaa0e7cf"
Jul 28 15:58:35.353654 ip-10-0-131-121 kubenswrapper[2568]: E0728 15:58:35.353637    2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bcda21cdef5aaada5a08c64f1696c7117ac521aa744cd5e45b4aa02dcaa0e7cf\": container with ID starting with bcda21cdef5aaada5a08c64f1696c7117ac521aa744cd5e45b4aa02dcaa0e7cf not found: ID does not exist" containerID="bcda21cdef5aaada5a08c64f1696c7117ac521aa744cd5e45b4aa02dcaa0e7cf"
Jul 28 15:58:35.353694 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:58:35.353659    2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bcda21cdef5aaada5a08c64f1696c7117ac521aa744cd5e45b4aa02dcaa0e7cf"} err="failed to get container status \"bcda21cdef5aaada5a08c64f1696c7117ac521aa744cd5e45b4aa02dcaa0e7cf\": rpc error: code = NotFound desc = could not find container \"bcda21cdef5aaada5a08c64f1696c7117ac521aa744cd5e45b4aa02dcaa0e7cf\": container with ID starting with bcda21cdef5aaada5a08c64f1696c7117ac521aa744cd5e45b4aa02dcaa0e7cf not found: ID does not exist"
Jul 28 15:58:35.357586 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:58:35.357564    2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-w25q8"]
Jul 28 15:58:35.519725 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:58:35.519693    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/383b7313-e6b6-4fc4-8b66-b0d9a6eb4a81-proxy-tls\") pod \"isvc-canary-rollback-v2-predictor-84f48b7656-h2tj5\" (UID: \"383b7313-e6b6-4fc4-8b66-b0d9a6eb4a81\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-84f48b7656-h2tj5"
Jul 28 15:58:35.519916 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:58:35.519768    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ceba507d-2395-4695-b8f8-72816fd44c52-proxy-tls\") pod \"isvc-canary-rollback-predictor-5bbb8c8cfc-qnljc\" (UID: \"ceba507d-2395-4695-b8f8-72816fd44c52\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-qnljc"
Jul 28 15:58:35.522235 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:58:35.522214    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ceba507d-2395-4695-b8f8-72816fd44c52-proxy-tls\") pod \"isvc-canary-rollback-predictor-5bbb8c8cfc-qnljc\" (UID: \"ceba507d-2395-4695-b8f8-72816fd44c52\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-qnljc"
Jul 28 15:58:35.522288 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:58:35.522218    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/383b7313-e6b6-4fc4-8b66-b0d9a6eb4a81-proxy-tls\") pod \"isvc-canary-rollback-v2-predictor-84f48b7656-h2tj5\" (UID: \"383b7313-e6b6-4fc4-8b66-b0d9a6eb4a81\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-84f48b7656-h2tj5"
Jul 28 15:58:35.715647 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:58:35.715537    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-qnljc"
Jul 28 15:58:35.736766 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:58:35.736730    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-84f48b7656-h2tj5"
Jul 28 15:58:35.867539 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:58:35.867356    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-qnljc"]
Jul 28 15:58:35.869868 ip-10-0-131-121 kubenswrapper[2568]: W0728 15:58:35.869831    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podceba507d_2395_4695_b8f8_72816fd44c52.slice/crio-86b64db14bb70852d17495313163d25fbd8c70c99499d1d24067d82013bf9e27 WatchSource:0}: Error finding container 86b64db14bb70852d17495313163d25fbd8c70c99499d1d24067d82013bf9e27: Status 404 returned error can't find the container with id 86b64db14bb70852d17495313163d25fbd8c70c99499d1d24067d82013bf9e27
Jul 28 15:58:35.887169 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:58:35.887145    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-84f48b7656-h2tj5"]
Jul 28 15:58:35.890153 ip-10-0-131-121 kubenswrapper[2568]: W0728 15:58:35.890127    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod383b7313_e6b6_4fc4_8b66_b0d9a6eb4a81.slice/crio-edb479c2152f7926334f53eede023dcf34d104e7e52dc80682feab699b0154e7 WatchSource:0}: Error finding container edb479c2152f7926334f53eede023dcf34d104e7e52dc80682feab699b0154e7: Status 404 returned error can't find the container with id edb479c2152f7926334f53eede023dcf34d104e7e52dc80682feab699b0154e7
Jul 28 15:58:36.296881 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:58:36.296841    2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="69f19a07-b0f5-491c-996d-f842c7ca6bba" path="/var/lib/kubelet/pods/69f19a07-b0f5-491c-996d-f842c7ca6bba/volumes"
Jul 28 15:58:36.334966 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:58:36.334925    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-84f48b7656-h2tj5" event={"ID":"383b7313-e6b6-4fc4-8b66-b0d9a6eb4a81","Type":"ContainerStarted","Data":"5811f6d0ccf45eebb1b5e1c24b14f5f6cbf42554bf47a0c4a7568045dda4b048"}
Jul 28 15:58:36.335161 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:58:36.334994    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-84f48b7656-h2tj5" event={"ID":"383b7313-e6b6-4fc4-8b66-b0d9a6eb4a81","Type":"ContainerStarted","Data":"edb479c2152f7926334f53eede023dcf34d104e7e52dc80682feab699b0154e7"}
Jul 28 15:58:36.336642 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:58:36.336618    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-qnljc" event={"ID":"ceba507d-2395-4695-b8f8-72816fd44c52","Type":"ContainerStarted","Data":"81dc2451a9b4d7c7978380a22801d3e75c3f95d812b7bd1ca53deccbb18a250d"}
Jul 28 15:58:36.336790 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:58:36.336673    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-qnljc" event={"ID":"ceba507d-2395-4695-b8f8-72816fd44c52","Type":"ContainerStarted","Data":"86b64db14bb70852d17495313163d25fbd8c70c99499d1d24067d82013bf9e27"}
Jul 28 15:58:40.353560 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:58:40.353527    2568 generic.go:356] "Generic (PLEG): container finished" podID="ceba507d-2395-4695-b8f8-72816fd44c52" containerID="81dc2451a9b4d7c7978380a22801d3e75c3f95d812b7bd1ca53deccbb18a250d" exitCode=0
Jul 28 15:58:40.353976 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:58:40.353602    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-qnljc" event={"ID":"ceba507d-2395-4695-b8f8-72816fd44c52","Type":"ContainerDied","Data":"81dc2451a9b4d7c7978380a22801d3e75c3f95d812b7bd1ca53deccbb18a250d"}
Jul 28 15:58:40.355255 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:58:40.355230    2568 generic.go:356] "Generic (PLEG): container finished" podID="383b7313-e6b6-4fc4-8b66-b0d9a6eb4a81" containerID="5811f6d0ccf45eebb1b5e1c24b14f5f6cbf42554bf47a0c4a7568045dda4b048" exitCode=0
Jul 28 15:58:40.355393 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:58:40.355257    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-84f48b7656-h2tj5" event={"ID":"383b7313-e6b6-4fc4-8b66-b0d9a6eb4a81","Type":"ContainerDied","Data":"5811f6d0ccf45eebb1b5e1c24b14f5f6cbf42554bf47a0c4a7568045dda4b048"}
Jul 28 15:58:41.361765 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:58:41.361727    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-84f48b7656-h2tj5" event={"ID":"383b7313-e6b6-4fc4-8b66-b0d9a6eb4a81","Type":"ContainerStarted","Data":"7711434af530d1d019ede86ad5f190810b2f61a24753ee1cdd87f4d50579f190"}
Jul 28 15:58:41.362175 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:58:41.361857    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-84f48b7656-h2tj5" event={"ID":"383b7313-e6b6-4fc4-8b66-b0d9a6eb4a81","Type":"ContainerStarted","Data":"c827ea4f6f00de697136ed19dbbb66a6dc5084979985b98328292575f7344e91"}
Jul 28 15:58:41.362175 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:58:41.362141    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-84f48b7656-h2tj5"
Jul 28 15:58:41.363789 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:58:41.363756    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-qnljc" event={"ID":"ceba507d-2395-4695-b8f8-72816fd44c52","Type":"ContainerStarted","Data":"c6d3ac0943d0a0dcb4b77e29b43b6ec860ce7929b9f4739d9096a39887e043bf"}
Jul 28 15:58:41.363907 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:58:41.363818    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-qnljc" event={"ID":"ceba507d-2395-4695-b8f8-72816fd44c52","Type":"ContainerStarted","Data":"f873b229a9d8ebdc8be024d319961a53448eb645de750bed09342b0fa1ae6a38"}
Jul 28 15:58:41.364003 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:58:41.363985    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-qnljc"
Jul 28 15:58:41.364053 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:58:41.364043    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-qnljc"
Jul 28 15:58:41.365264 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:58:41.365241    2568 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-qnljc" podUID="ceba507d-2395-4695-b8f8-72816fd44c52" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.30:8080: connect: connection refused"
Jul 28 15:58:41.385761 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:58:41.385707    2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-84f48b7656-h2tj5" podStartSLOduration=3.685293325 podStartE2EDuration="7.385693682s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.700400357s" isStatefulPod=false podCreationTimestamp="2026-07-28 15:58:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 15:58:41.384309582 +0000 UTC m=+1501.569504750" watchObservedRunningTime="2026-07-28 15:58:41.385693682 +0000 UTC m=+1501.570888851"
Jul 28 15:58:41.405581 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:58:41.405533    2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-qnljc" podStartSLOduration=3.717510077 podStartE2EDuration="7.405515758s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.688005681s" isStatefulPod=false podCreationTimestamp="2026-07-28 15:58:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 15:58:41.40406672 +0000 UTC m=+1501.589261904" watchObservedRunningTime="2026-07-28 15:58:41.405515758 +0000 UTC m=+1501.590711003"
Jul 28 15:58:42.370148 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:58:42.370108    2568 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-qnljc" podUID="ceba507d-2395-4695-b8f8-72816fd44c52" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.30:8080: connect: connection refused"
Jul 28 15:58:42.370610 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:58:42.370251    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-84f48b7656-h2tj5"
Jul 28 15:58:42.371283 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:58:42.371260    2568 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-84f48b7656-h2tj5" podUID="383b7313-e6b6-4fc4-8b66-b0d9a6eb4a81" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:8080: connect: connection refused"
Jul 28 15:58:43.375246 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:58:43.375206    2568 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-84f48b7656-h2tj5" podUID="383b7313-e6b6-4fc4-8b66-b0d9a6eb4a81" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:8080: connect: connection refused"
Jul 28 15:58:47.375022 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:58:47.374993    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-qnljc"
Jul 28 15:58:47.375478 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:58:47.375452    2568 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-qnljc" podUID="ceba507d-2395-4695-b8f8-72816fd44c52" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.30:8080: connect: connection refused"
Jul 28 15:58:48.379834 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:58:48.379801    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-84f48b7656-h2tj5"
Jul 28 15:58:48.380327 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:58:48.380296    2568 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-84f48b7656-h2tj5" podUID="383b7313-e6b6-4fc4-8b66-b0d9a6eb4a81" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:8080: connect: connection refused"
Jul 28 15:58:50.614688 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:58:50.614655    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22nh9_ebf75c0d-1597-4a1c-b668-dc2a041eb35e/konnectivity-agent/0.log" limit=1024
Jul 28 15:58:57.376320 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:58:57.376281    2568 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-qnljc" podUID="ceba507d-2395-4695-b8f8-72816fd44c52" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.30:8080: connect: connection refused"
Jul 28 15:58:58.381269 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:58:58.381233    2568 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-84f48b7656-h2tj5" podUID="383b7313-e6b6-4fc4-8b66-b0d9a6eb4a81" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:8080: connect: connection refused"
Jul 28 15:59:01.428531 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:59:01.428487    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22nh9_ebf75c0d-1597-4a1c-b668-dc2a041eb35e/konnectivity-agent/0.log" limit=1024
Jul 28 15:59:07.376057 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:59:07.376015    2568 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-qnljc" podUID="ceba507d-2395-4695-b8f8-72816fd44c52" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.30:8080: connect: connection refused"
Jul 28 15:59:08.381163 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:59:08.381119    2568 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-84f48b7656-h2tj5" podUID="383b7313-e6b6-4fc4-8b66-b0d9a6eb4a81" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:8080: connect: connection refused"
Jul 28 15:59:17.376257 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:59:17.376209    2568 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-qnljc" podUID="ceba507d-2395-4695-b8f8-72816fd44c52" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.30:8080: connect: connection refused"
Jul 28 15:59:18.380869 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:59:18.380823    2568 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-84f48b7656-h2tj5" podUID="383b7313-e6b6-4fc4-8b66-b0d9a6eb4a81" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:8080: connect: connection refused"
Jul 28 15:59:27.376171 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:59:27.376125    2568 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-qnljc" podUID="ceba507d-2395-4695-b8f8-72816fd44c52" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.30:8080: connect: connection refused"
Jul 28 15:59:28.380312 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:59:28.380270    2568 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-84f48b7656-h2tj5" podUID="383b7313-e6b6-4fc4-8b66-b0d9a6eb4a81" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:8080: connect: connection refused"
Jul 28 15:59:37.376225 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:59:37.376185    2568 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-qnljc" podUID="ceba507d-2395-4695-b8f8-72816fd44c52" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.30:8080: connect: connection refused"
Jul 28 15:59:38.380376 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:59:38.380339    2568 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-84f48b7656-h2tj5" podUID="383b7313-e6b6-4fc4-8b66-b0d9a6eb4a81" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:8080: connect: connection refused"
Jul 28 15:59:47.376073 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:59:47.376023    2568 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-qnljc" podUID="ceba507d-2395-4695-b8f8-72816fd44c52" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.30:8080: connect: connection refused"
Jul 28 15:59:48.380812 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:59:48.380751    2568 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-84f48b7656-h2tj5" podUID="383b7313-e6b6-4fc4-8b66-b0d9a6eb4a81" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:8080: connect: connection refused"
Jul 28 15:59:52.289756 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:59:52.289704    2568 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-84f48b7656-h2tj5" podUID="383b7313-e6b6-4fc4-8b66-b0d9a6eb4a81" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:8080: connect: connection refused"
Jul 28 15:59:57.376182 ip-10-0-131-121 kubenswrapper[2568]: I0728 15:59:57.376153    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-qnljc"
Jul 28 16:00:02.295290 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:00:02.295262    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-84f48b7656-h2tj5"
Jul 28 16:00:04.763033 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:00:04.762994    2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-84f48b7656-h2tj5"]
Jul 28 16:00:04.763526 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:00:04.763219    2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-84f48b7656-h2tj5" podUID="383b7313-e6b6-4fc4-8b66-b0d9a6eb4a81" containerName="kserve-container" containerID="cri-o://c827ea4f6f00de697136ed19dbbb66a6dc5084979985b98328292575f7344e91" gracePeriod=30
Jul 28 16:00:04.763526 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:00:04.763302    2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-84f48b7656-h2tj5" podUID="383b7313-e6b6-4fc4-8b66-b0d9a6eb4a81" containerName="kube-rbac-proxy" containerID="cri-o://7711434af530d1d019ede86ad5f190810b2f61a24753ee1cdd87f4d50579f190" gracePeriod=30
Jul 28 16:00:04.785569 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:00:04.785535    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22nh9_ebf75c0d-1597-4a1c-b668-dc2a041eb35e/konnectivity-agent/0.log" limit=1024
Jul 28 16:00:05.746938 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:00:05.746905    2568 generic.go:356] "Generic (PLEG): container finished" podID="383b7313-e6b6-4fc4-8b66-b0d9a6eb4a81" containerID="7711434af530d1d019ede86ad5f190810b2f61a24753ee1cdd87f4d50579f190" exitCode=2
Jul 28 16:00:05.747115 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:00:05.746947    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-84f48b7656-h2tj5" event={"ID":"383b7313-e6b6-4fc4-8b66-b0d9a6eb4a81","Type":"ContainerDied","Data":"7711434af530d1d019ede86ad5f190810b2f61a24753ee1cdd87f4d50579f190"}
Jul 28 16:00:08.375656 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:00:08.375603    2568 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-84f48b7656-h2tj5" podUID="383b7313-e6b6-4fc4-8b66-b0d9a6eb4a81" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.31:8643/healthz\": dial tcp 10.134.0.31:8643: connect: connection refused"
Jul 28 16:00:09.714286 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:00:09.714260    2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-84f48b7656-h2tj5"
Jul 28 16:00:09.745435 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:00:09.745407    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/383b7313-e6b6-4fc4-8b66-b0d9a6eb4a81-kube-rbac-proxy-sar-config\") pod \"383b7313-e6b6-4fc4-8b66-b0d9a6eb4a81\" (UID: \"383b7313-e6b6-4fc4-8b66-b0d9a6eb4a81\") "
Jul 28 16:00:09.745588 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:00:09.745448    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7lg4x\" (UniqueName: \"kubernetes.io/projected/383b7313-e6b6-4fc4-8b66-b0d9a6eb4a81-kube-api-access-7lg4x\") pod \"383b7313-e6b6-4fc4-8b66-b0d9a6eb4a81\" (UID: \"383b7313-e6b6-4fc4-8b66-b0d9a6eb4a81\") "
Jul 28 16:00:09.745650 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:00:09.745627    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/383b7313-e6b6-4fc4-8b66-b0d9a6eb4a81-cabundle-cert\") pod \"383b7313-e6b6-4fc4-8b66-b0d9a6eb4a81\" (UID: \"383b7313-e6b6-4fc4-8b66-b0d9a6eb4a81\") "
Jul 28 16:00:09.745730 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:00:09.745715    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/383b7313-e6b6-4fc4-8b66-b0d9a6eb4a81-proxy-tls\") pod \"383b7313-e6b6-4fc4-8b66-b0d9a6eb4a81\" (UID: \"383b7313-e6b6-4fc4-8b66-b0d9a6eb4a81\") "
Jul 28 16:00:09.745820 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:00:09.745719    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/383b7313-e6b6-4fc4-8b66-b0d9a6eb4a81-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "383b7313-e6b6-4fc4-8b66-b0d9a6eb4a81" (UID: "383b7313-e6b6-4fc4-8b66-b0d9a6eb4a81"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 16:00:09.745820 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:00:09.745766    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/383b7313-e6b6-4fc4-8b66-b0d9a6eb4a81-kserve-provision-location\") pod \"383b7313-e6b6-4fc4-8b66-b0d9a6eb4a81\" (UID: \"383b7313-e6b6-4fc4-8b66-b0d9a6eb4a81\") "
Jul 28 16:00:09.746002 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:00:09.745978    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/383b7313-e6b6-4fc4-8b66-b0d9a6eb4a81-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "383b7313-e6b6-4fc4-8b66-b0d9a6eb4a81" (UID: "383b7313-e6b6-4fc4-8b66-b0d9a6eb4a81"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 16:00:09.746002 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:00:09.745993    2568 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/383b7313-e6b6-4fc4-8b66-b0d9a6eb4a81-kube-rbac-proxy-sar-config\") on node \"ip-10-0-131-121.ec2.internal\" DevicePath \"\""
Jul 28 16:00:09.746163 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:00:09.746000    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/383b7313-e6b6-4fc4-8b66-b0d9a6eb4a81-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "383b7313-e6b6-4fc4-8b66-b0d9a6eb4a81" (UID: "383b7313-e6b6-4fc4-8b66-b0d9a6eb4a81"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 16:00:09.747620 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:00:09.747585    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/383b7313-e6b6-4fc4-8b66-b0d9a6eb4a81-kube-api-access-7lg4x" (OuterVolumeSpecName: "kube-api-access-7lg4x") pod "383b7313-e6b6-4fc4-8b66-b0d9a6eb4a81" (UID: "383b7313-e6b6-4fc4-8b66-b0d9a6eb4a81"). InnerVolumeSpecName "kube-api-access-7lg4x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 16:00:09.747728 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:00:09.747683    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/383b7313-e6b6-4fc4-8b66-b0d9a6eb4a81-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "383b7313-e6b6-4fc4-8b66-b0d9a6eb4a81" (UID: "383b7313-e6b6-4fc4-8b66-b0d9a6eb4a81"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 16:00:09.770048 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:00:09.770018    2568 generic.go:356] "Generic (PLEG): container finished" podID="383b7313-e6b6-4fc4-8b66-b0d9a6eb4a81" containerID="c827ea4f6f00de697136ed19dbbb66a6dc5084979985b98328292575f7344e91" exitCode=0
Jul 28 16:00:09.770228 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:00:09.770076    2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-84f48b7656-h2tj5"
Jul 28 16:00:09.770228 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:00:09.770081    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-84f48b7656-h2tj5" event={"ID":"383b7313-e6b6-4fc4-8b66-b0d9a6eb4a81","Type":"ContainerDied","Data":"c827ea4f6f00de697136ed19dbbb66a6dc5084979985b98328292575f7344e91"}
Jul 28 16:00:09.770228 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:00:09.770142    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-84f48b7656-h2tj5" event={"ID":"383b7313-e6b6-4fc4-8b66-b0d9a6eb4a81","Type":"ContainerDied","Data":"edb479c2152f7926334f53eede023dcf34d104e7e52dc80682feab699b0154e7"}
Jul 28 16:00:09.770228 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:00:09.770170    2568 scope.go:117] "RemoveContainer" containerID="7711434af530d1d019ede86ad5f190810b2f61a24753ee1cdd87f4d50579f190"
Jul 28 16:00:09.785344 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:00:09.785324    2568 scope.go:117] "RemoveContainer" containerID="c827ea4f6f00de697136ed19dbbb66a6dc5084979985b98328292575f7344e91"
Jul 28 16:00:09.793319 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:00:09.793297    2568 scope.go:117] "RemoveContainer" containerID="5811f6d0ccf45eebb1b5e1c24b14f5f6cbf42554bf47a0c4a7568045dda4b048"
Jul 28 16:00:09.801585 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:00:09.801555    2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-84f48b7656-h2tj5"]
Jul 28 16:00:09.801995 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:00:09.801978    2568 scope.go:117] "RemoveContainer" containerID="7711434af530d1d019ede86ad5f190810b2f61a24753ee1cdd87f4d50579f190"
Jul 28 16:00:09.802294 ip-10-0-131-121 kubenswrapper[2568]: E0728 16:00:09.802273    2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7711434af530d1d019ede86ad5f190810b2f61a24753ee1cdd87f4d50579f190\": container with ID starting with 7711434af530d1d019ede86ad5f190810b2f61a24753ee1cdd87f4d50579f190 not found: ID does not exist" containerID="7711434af530d1d019ede86ad5f190810b2f61a24753ee1cdd87f4d50579f190"
Jul 28 16:00:09.802328 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:00:09.802305    2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7711434af530d1d019ede86ad5f190810b2f61a24753ee1cdd87f4d50579f190"} err="failed to get container status \"7711434af530d1d019ede86ad5f190810b2f61a24753ee1cdd87f4d50579f190\": rpc error: code = NotFound desc = could not find container \"7711434af530d1d019ede86ad5f190810b2f61a24753ee1cdd87f4d50579f190\": container with ID starting with 7711434af530d1d019ede86ad5f190810b2f61a24753ee1cdd87f4d50579f190 not found: ID does not exist"
Jul 28 16:00:09.802328 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:00:09.802325    2568 scope.go:117] "RemoveContainer" containerID="c827ea4f6f00de697136ed19dbbb66a6dc5084979985b98328292575f7344e91"
Jul 28 16:00:09.802570 ip-10-0-131-121 kubenswrapper[2568]: E0728 16:00:09.802554    2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c827ea4f6f00de697136ed19dbbb66a6dc5084979985b98328292575f7344e91\": container with ID starting with c827ea4f6f00de697136ed19dbbb66a6dc5084979985b98328292575f7344e91 not found: ID does not exist" containerID="c827ea4f6f00de697136ed19dbbb66a6dc5084979985b98328292575f7344e91"
Jul 28 16:00:09.802608 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:00:09.802578    2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c827ea4f6f00de697136ed19dbbb66a6dc5084979985b98328292575f7344e91"} err="failed to get container status \"c827ea4f6f00de697136ed19dbbb66a6dc5084979985b98328292575f7344e91\": rpc error: code = NotFound desc = could not find container \"c827ea4f6f00de697136ed19dbbb66a6dc5084979985b98328292575f7344e91\": container with ID starting with c827ea4f6f00de697136ed19dbbb66a6dc5084979985b98328292575f7344e91 not found: ID does not exist"
Jul 28 16:00:09.802608 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:00:09.802593    2568 scope.go:117] "RemoveContainer" containerID="5811f6d0ccf45eebb1b5e1c24b14f5f6cbf42554bf47a0c4a7568045dda4b048"
Jul 28 16:00:09.802951 ip-10-0-131-121 kubenswrapper[2568]: E0728 16:00:09.802930    2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5811f6d0ccf45eebb1b5e1c24b14f5f6cbf42554bf47a0c4a7568045dda4b048\": container with ID starting with 5811f6d0ccf45eebb1b5e1c24b14f5f6cbf42554bf47a0c4a7568045dda4b048 not found: ID does not exist" containerID="5811f6d0ccf45eebb1b5e1c24b14f5f6cbf42554bf47a0c4a7568045dda4b048"
Jul 28 16:00:09.803028 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:00:09.802957    2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5811f6d0ccf45eebb1b5e1c24b14f5f6cbf42554bf47a0c4a7568045dda4b048"} err="failed to get container status \"5811f6d0ccf45eebb1b5e1c24b14f5f6cbf42554bf47a0c4a7568045dda4b048\": rpc error: code = NotFound desc = could not find container \"5811f6d0ccf45eebb1b5e1c24b14f5f6cbf42554bf47a0c4a7568045dda4b048\": container with ID starting with 5811f6d0ccf45eebb1b5e1c24b14f5f6cbf42554bf47a0c4a7568045dda4b048 not found: ID does not exist"
Jul 28 16:00:09.808474 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:00:09.808450    2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-84f48b7656-h2tj5"]
Jul 28 16:00:09.847089 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:00:09.847055    2568 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/383b7313-e6b6-4fc4-8b66-b0d9a6eb4a81-cabundle-cert\") on node \"ip-10-0-131-121.ec2.internal\" DevicePath \"\""
Jul 28 16:00:09.847089 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:00:09.847083    2568 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/383b7313-e6b6-4fc4-8b66-b0d9a6eb4a81-proxy-tls\") on node \"ip-10-0-131-121.ec2.internal\" DevicePath \"\""
Jul 28 16:00:09.847089 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:00:09.847094    2568 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/383b7313-e6b6-4fc4-8b66-b0d9a6eb4a81-kserve-provision-location\") on node \"ip-10-0-131-121.ec2.internal\" DevicePath \"\""
Jul 28 16:00:09.847325 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:00:09.847104    2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7lg4x\" (UniqueName: \"kubernetes.io/projected/383b7313-e6b6-4fc4-8b66-b0d9a6eb4a81-kube-api-access-7lg4x\") on node \"ip-10-0-131-121.ec2.internal\" DevicePath \"\""
Jul 28 16:00:10.294793 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:00:10.294758    2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="383b7313-e6b6-4fc4-8b66-b0d9a6eb4a81" path="/var/lib/kubelet/pods/383b7313-e6b6-4fc4-8b66-b0d9a6eb4a81/volumes"
Jul 28 16:00:16.007012 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:00:16.006976    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22nh9_ebf75c0d-1597-4a1c-b668-dc2a041eb35e/konnectivity-agent/0.log" limit=1024
Jul 28 16:00:16.107387 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:00:16.107358    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22nh9_ebf75c0d-1597-4a1c-b668-dc2a041eb35e/konnectivity-agent/0.log" limit=1024
Jul 28 16:01:34.965428 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:01:34.965366    2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-qnljc"]
Jul 28 16:01:34.966016 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:01:34.965698    2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-qnljc" podUID="ceba507d-2395-4695-b8f8-72816fd44c52" containerName="kserve-container" containerID="cri-o://f873b229a9d8ebdc8be024d319961a53448eb645de750bed09342b0fa1ae6a38" gracePeriod=30
Jul 28 16:01:34.966016 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:01:34.965727    2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-qnljc" podUID="ceba507d-2395-4695-b8f8-72816fd44c52" containerName="kube-rbac-proxy" containerID="cri-o://c6d3ac0943d0a0dcb4b77e29b43b6ec860ce7929b9f4739d9096a39887e043bf" gracePeriod=30
Jul 28 16:01:34.991336 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:01:34.991308    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22nh9_ebf75c0d-1597-4a1c-b668-dc2a041eb35e/konnectivity-agent/0.log" limit=1024
Jul 28 16:01:35.122354 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:01:35.122323    2568 generic.go:356] "Generic (PLEG): container finished" podID="ceba507d-2395-4695-b8f8-72816fd44c52" containerID="c6d3ac0943d0a0dcb4b77e29b43b6ec860ce7929b9f4739d9096a39887e043bf" exitCode=2
Jul 28 16:01:35.122519 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:01:35.122383    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-qnljc" event={"ID":"ceba507d-2395-4695-b8f8-72816fd44c52","Type":"ContainerDied","Data":"c6d3ac0943d0a0dcb4b77e29b43b6ec860ce7929b9f4739d9096a39887e043bf"}
Jul 28 16:01:37.370816 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:01:37.370751    2568 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-qnljc" podUID="ceba507d-2395-4695-b8f8-72816fd44c52" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.30:8643/healthz\": dial tcp 10.134.0.30:8643: connect: connection refused"
Jul 28 16:01:37.376126 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:01:37.376096    2568 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-qnljc" podUID="ceba507d-2395-4695-b8f8-72816fd44c52" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.30:8080: connect: connection refused"
Jul 28 16:01:39.920143 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:01:39.920113    2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-qnljc"
Jul 28 16:01:40.010348 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:01:40.010257    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ceba507d-2395-4695-b8f8-72816fd44c52-kube-rbac-proxy-sar-config\") pod \"ceba507d-2395-4695-b8f8-72816fd44c52\" (UID: \"ceba507d-2395-4695-b8f8-72816fd44c52\") "
Jul 28 16:01:40.010348 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:01:40.010319    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ceba507d-2395-4695-b8f8-72816fd44c52-cabundle-cert\") pod \"ceba507d-2395-4695-b8f8-72816fd44c52\" (UID: \"ceba507d-2395-4695-b8f8-72816fd44c52\") "
Jul 28 16:01:40.010563 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:01:40.010370    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ceba507d-2395-4695-b8f8-72816fd44c52-kserve-provision-location\") pod \"ceba507d-2395-4695-b8f8-72816fd44c52\" (UID: \"ceba507d-2395-4695-b8f8-72816fd44c52\") "
Jul 28 16:01:40.010563 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:01:40.010400    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-x28nh\" (UniqueName: \"kubernetes.io/projected/ceba507d-2395-4695-b8f8-72816fd44c52-kube-api-access-x28nh\") pod \"ceba507d-2395-4695-b8f8-72816fd44c52\" (UID: \"ceba507d-2395-4695-b8f8-72816fd44c52\") "
Jul 28 16:01:40.010563 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:01:40.010445    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ceba507d-2395-4695-b8f8-72816fd44c52-proxy-tls\") pod \"ceba507d-2395-4695-b8f8-72816fd44c52\" (UID: \"ceba507d-2395-4695-b8f8-72816fd44c52\") "
Jul 28 16:01:40.010710 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:01:40.010674    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ceba507d-2395-4695-b8f8-72816fd44c52-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "ceba507d-2395-4695-b8f8-72816fd44c52" (UID: "ceba507d-2395-4695-b8f8-72816fd44c52"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 16:01:40.010804 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:01:40.010707    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ceba507d-2395-4695-b8f8-72816fd44c52-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "ceba507d-2395-4695-b8f8-72816fd44c52" (UID: "ceba507d-2395-4695-b8f8-72816fd44c52"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 16:01:40.010804 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:01:40.010686    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ceba507d-2395-4695-b8f8-72816fd44c52-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "ceba507d-2395-4695-b8f8-72816fd44c52" (UID: "ceba507d-2395-4695-b8f8-72816fd44c52"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 16:01:40.012602 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:01:40.012576    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ceba507d-2395-4695-b8f8-72816fd44c52-kube-api-access-x28nh" (OuterVolumeSpecName: "kube-api-access-x28nh") pod "ceba507d-2395-4695-b8f8-72816fd44c52" (UID: "ceba507d-2395-4695-b8f8-72816fd44c52"). InnerVolumeSpecName "kube-api-access-x28nh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 16:01:40.012708 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:01:40.012622    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ceba507d-2395-4695-b8f8-72816fd44c52-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "ceba507d-2395-4695-b8f8-72816fd44c52" (UID: "ceba507d-2395-4695-b8f8-72816fd44c52"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 16:01:40.087484 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:01:40.087449    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22nh9_ebf75c0d-1597-4a1c-b668-dc2a041eb35e/konnectivity-agent/0.log" limit=1024
Jul 28 16:01:40.111316 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:01:40.111279    2568 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ceba507d-2395-4695-b8f8-72816fd44c52-proxy-tls\") on node \"ip-10-0-131-121.ec2.internal\" DevicePath \"\""
Jul 28 16:01:40.111316 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:01:40.111311    2568 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ceba507d-2395-4695-b8f8-72816fd44c52-kube-rbac-proxy-sar-config\") on node \"ip-10-0-131-121.ec2.internal\" DevicePath \"\""
Jul 28 16:01:40.111316 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:01:40.111323    2568 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ceba507d-2395-4695-b8f8-72816fd44c52-cabundle-cert\") on node \"ip-10-0-131-121.ec2.internal\" DevicePath \"\""
Jul 28 16:01:40.111572 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:01:40.111332    2568 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ceba507d-2395-4695-b8f8-72816fd44c52-kserve-provision-location\") on node \"ip-10-0-131-121.ec2.internal\" DevicePath \"\""
Jul 28 16:01:40.111572 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:01:40.111340    2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x28nh\" (UniqueName: \"kubernetes.io/projected/ceba507d-2395-4695-b8f8-72816fd44c52-kube-api-access-x28nh\") on node \"ip-10-0-131-121.ec2.internal\" DevicePath \"\""
Jul 28 16:01:40.145158 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:01:40.145126    2568 generic.go:356] "Generic (PLEG): container finished" podID="ceba507d-2395-4695-b8f8-72816fd44c52" containerID="f873b229a9d8ebdc8be024d319961a53448eb645de750bed09342b0fa1ae6a38" exitCode=0
Jul 28 16:01:40.145338 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:01:40.145209    2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-qnljc"
Jul 28 16:01:40.145338 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:01:40.145216    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-qnljc" event={"ID":"ceba507d-2395-4695-b8f8-72816fd44c52","Type":"ContainerDied","Data":"f873b229a9d8ebdc8be024d319961a53448eb645de750bed09342b0fa1ae6a38"}
Jul 28 16:01:40.145338 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:01:40.145323    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-qnljc" event={"ID":"ceba507d-2395-4695-b8f8-72816fd44c52","Type":"ContainerDied","Data":"86b64db14bb70852d17495313163d25fbd8c70c99499d1d24067d82013bf9e27"}
Jul 28 16:01:40.145510 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:01:40.145363    2568 scope.go:117] "RemoveContainer" containerID="c6d3ac0943d0a0dcb4b77e29b43b6ec860ce7929b9f4739d9096a39887e043bf"
Jul 28 16:01:40.155418 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:01:40.155396    2568 scope.go:117] "RemoveContainer" containerID="f873b229a9d8ebdc8be024d319961a53448eb645de750bed09342b0fa1ae6a38"
Jul 28 16:01:40.164430 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:01:40.164406    2568 scope.go:117] "RemoveContainer" containerID="81dc2451a9b4d7c7978380a22801d3e75c3f95d812b7bd1ca53deccbb18a250d"
Jul 28 16:01:40.166890 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:01:40.166865    2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-86dc9fdb65-snddz"]
Jul 28 16:01:40.167332 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:01:40.167315    2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ceba507d-2395-4695-b8f8-72816fd44c52" containerName="storage-initializer"
Jul 28 16:01:40.167410 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:01:40.167333    2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="ceba507d-2395-4695-b8f8-72816fd44c52" containerName="storage-initializer"
Jul 28 16:01:40.167410 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:01:40.167345    2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ceba507d-2395-4695-b8f8-72816fd44c52" containerName="kube-rbac-proxy"
Jul 28 16:01:40.167410 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:01:40.167354    2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="ceba507d-2395-4695-b8f8-72816fd44c52" containerName="kube-rbac-proxy"
Jul 28 16:01:40.167410 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:01:40.167367    2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="383b7313-e6b6-4fc4-8b66-b0d9a6eb4a81" containerName="storage-initializer"
Jul 28 16:01:40.167410 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:01:40.167376    2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="383b7313-e6b6-4fc4-8b66-b0d9a6eb4a81" containerName="storage-initializer"
Jul 28 16:01:40.167410 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:01:40.167393    2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ceba507d-2395-4695-b8f8-72816fd44c52" containerName="kserve-container"
Jul 28 16:01:40.167410 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:01:40.167402    2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="ceba507d-2395-4695-b8f8-72816fd44c52" containerName="kserve-container"
Jul 28 16:01:40.167410 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:01:40.167410    2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="383b7313-e6b6-4fc4-8b66-b0d9a6eb4a81" containerName="kube-rbac-proxy"
Jul 28 16:01:40.167982 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:01:40.167418    2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="383b7313-e6b6-4fc4-8b66-b0d9a6eb4a81" containerName="kube-rbac-proxy"
Jul 28 16:01:40.167982 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:01:40.167431    2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="383b7313-e6b6-4fc4-8b66-b0d9a6eb4a81" containerName="kserve-container"
Jul 28 16:01:40.167982 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:01:40.167439    2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="383b7313-e6b6-4fc4-8b66-b0d9a6eb4a81" containerName="kserve-container"
Jul 28 16:01:40.167982 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:01:40.167511    2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="383b7313-e6b6-4fc4-8b66-b0d9a6eb4a81" containerName="kserve-container"
Jul 28 16:01:40.167982 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:01:40.167521    2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="ceba507d-2395-4695-b8f8-72816fd44c52" containerName="kserve-container"
Jul 28 16:01:40.167982 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:01:40.167533    2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="383b7313-e6b6-4fc4-8b66-b0d9a6eb4a81" containerName="kube-rbac-proxy"
Jul 28 16:01:40.167982 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:01:40.167541    2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="ceba507d-2395-4695-b8f8-72816fd44c52" containerName="kube-rbac-proxy"
Jul 28 16:01:40.172511 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:01:40.172492    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-86dc9fdb65-snddz"
Jul 28 16:01:40.173455 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:01:40.173438    2568 scope.go:117] "RemoveContainer" containerID="c6d3ac0943d0a0dcb4b77e29b43b6ec860ce7929b9f4739d9096a39887e043bf"
Jul 28 16:01:40.173769 ip-10-0-131-121 kubenswrapper[2568]: E0728 16:01:40.173747    2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c6d3ac0943d0a0dcb4b77e29b43b6ec860ce7929b9f4739d9096a39887e043bf\": container with ID starting with c6d3ac0943d0a0dcb4b77e29b43b6ec860ce7929b9f4739d9096a39887e043bf not found: ID does not exist" containerID="c6d3ac0943d0a0dcb4b77e29b43b6ec860ce7929b9f4739d9096a39887e043bf"
Jul 28 16:01:40.173874 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:01:40.173793    2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c6d3ac0943d0a0dcb4b77e29b43b6ec860ce7929b9f4739d9096a39887e043bf"} err="failed to get container status \"c6d3ac0943d0a0dcb4b77e29b43b6ec860ce7929b9f4739d9096a39887e043bf\": rpc error: code = NotFound desc = could not find container \"c6d3ac0943d0a0dcb4b77e29b43b6ec860ce7929b9f4739d9096a39887e043bf\": container with ID starting with c6d3ac0943d0a0dcb4b77e29b43b6ec860ce7929b9f4739d9096a39887e043bf not found: ID does not exist"
Jul 28 16:01:40.173874 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:01:40.173817    2568 scope.go:117] "RemoveContainer" containerID="f873b229a9d8ebdc8be024d319961a53448eb645de750bed09342b0fa1ae6a38"
Jul 28 16:01:40.174123 ip-10-0-131-121 kubenswrapper[2568]: E0728 16:01:40.174094    2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f873b229a9d8ebdc8be024d319961a53448eb645de750bed09342b0fa1ae6a38\": container with ID starting with f873b229a9d8ebdc8be024d319961a53448eb645de750bed09342b0fa1ae6a38 not found: ID does not exist" containerID="f873b229a9d8ebdc8be024d319961a53448eb645de750bed09342b0fa1ae6a38"
Jul 28 16:01:40.174229 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:01:40.174208    2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f873b229a9d8ebdc8be024d319961a53448eb645de750bed09342b0fa1ae6a38"} err="failed to get container status \"f873b229a9d8ebdc8be024d319961a53448eb645de750bed09342b0fa1ae6a38\": rpc error: code = NotFound desc = could not find container \"f873b229a9d8ebdc8be024d319961a53448eb645de750bed09342b0fa1ae6a38\": container with ID starting with f873b229a9d8ebdc8be024d319961a53448eb645de750bed09342b0fa1ae6a38 not found: ID does not exist"
Jul 28 16:01:40.174298 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:01:40.174232    2568 scope.go:117] "RemoveContainer" containerID="81dc2451a9b4d7c7978380a22801d3e75c3f95d812b7bd1ca53deccbb18a250d"
Jul 28 16:01:40.174501 ip-10-0-131-121 kubenswrapper[2568]: E0728 16:01:40.174480    2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"81dc2451a9b4d7c7978380a22801d3e75c3f95d812b7bd1ca53deccbb18a250d\": container with ID starting with 81dc2451a9b4d7c7978380a22801d3e75c3f95d812b7bd1ca53deccbb18a250d not found: ID does not exist" containerID="81dc2451a9b4d7c7978380a22801d3e75c3f95d812b7bd1ca53deccbb18a250d"
Jul 28 16:01:40.174561 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:01:40.174507    2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"81dc2451a9b4d7c7978380a22801d3e75c3f95d812b7bd1ca53deccbb18a250d"} err="failed to get container status \"81dc2451a9b4d7c7978380a22801d3e75c3f95d812b7bd1ca53deccbb18a250d\": rpc error: code = NotFound desc = could not find container \"81dc2451a9b4d7c7978380a22801d3e75c3f95d812b7bd1ca53deccbb18a250d\": container with ID starting with 81dc2451a9b4d7c7978380a22801d3e75c3f95d812b7bd1ca53deccbb18a250d not found: ID does not exist"
Jul 28 16:01:40.175247 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:01:40.175002    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 28 16:01:40.175247 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:01:40.175015    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 28 16:01:40.175247 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:01:40.175063    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-stop-kube-rbac-proxy-sar-config\""
Jul 28 16:01:40.175247 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:01:40.175137    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 28 16:01:40.175247 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:01:40.175004    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-stop-predictor-serving-cert\""
Jul 28 16:01:40.175247 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:01:40.175186    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 28 16:01:40.175506 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:01:40.175321    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-gcwwc\""
Jul 28 16:01:40.184343 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:01:40.184284    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22nh9_ebf75c0d-1597-4a1c-b668-dc2a041eb35e/konnectivity-agent/0.log" limit=1024
Jul 28 16:01:40.184626 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:01:40.184603    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-86dc9fdb65-snddz"]
Jul 28 16:01:40.186444 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:01:40.186422    2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-77f778d864-vgkzc"]
Jul 28 16:01:40.191497 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:01:40.191476    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-77f778d864-vgkzc"
Jul 28 16:01:40.193792 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:01:40.193760    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-stop-v2-predictor-serving-cert\""
Jul 28 16:01:40.194645 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:01:40.194625    2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-qnljc"]
Jul 28 16:01:40.203562 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:01:40.203537    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-77f778d864-vgkzc"]
Jul 28 16:01:40.205226 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:01:40.205205    2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-qnljc"]
Jul 28 16:01:40.256416 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:01:40.256384    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22nh9_ebf75c0d-1597-4a1c-b668-dc2a041eb35e/konnectivity-agent/0.log" limit=1024
Jul 28 16:01:40.294893 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:01:40.294859    2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ceba507d-2395-4695-b8f8-72816fd44c52" path="/var/lib/kubelet/pods/ceba507d-2395-4695-b8f8-72816fd44c52/volumes"
Jul 28 16:01:40.313643 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:01:40.313612    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fltcs\" (UniqueName: \"kubernetes.io/projected/9bb04ac6-21f5-4467-93a8-1689b729ec93-kube-api-access-fltcs\") pod \"isvc-canary-stop-v2-predictor-77f778d864-vgkzc\" (UID: \"9bb04ac6-21f5-4467-93a8-1689b729ec93\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-77f778d864-vgkzc"
Jul 28 16:01:40.313800 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:01:40.313667    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9155611b-3e3d-4c28-9d34-c5a27ffea78e-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-predictor-86dc9fdb65-snddz\" (UID: \"9155611b-3e3d-4c28-9d34-c5a27ffea78e\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-86dc9fdb65-snddz"
Jul 28 16:01:40.313863 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:01:40.313770    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9155611b-3e3d-4c28-9d34-c5a27ffea78e-proxy-tls\") pod \"isvc-canary-stop-predictor-86dc9fdb65-snddz\" (UID: \"9155611b-3e3d-4c28-9d34-c5a27ffea78e\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-86dc9fdb65-snddz"
Jul 28 16:01:40.313929 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:01:40.313859    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9bb04ac6-21f5-4467-93a8-1689b729ec93-cabundle-cert\") pod \"isvc-canary-stop-v2-predictor-77f778d864-vgkzc\" (UID: \"9bb04ac6-21f5-4467-93a8-1689b729ec93\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-77f778d864-vgkzc"
Jul 28 16:01:40.313929 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:01:40.313915    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sjdc2\" (UniqueName: \"kubernetes.io/projected/9155611b-3e3d-4c28-9d34-c5a27ffea78e-kube-api-access-sjdc2\") pod \"isvc-canary-stop-predictor-86dc9fdb65-snddz\" (UID: \"9155611b-3e3d-4c28-9d34-c5a27ffea78e\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-86dc9fdb65-snddz"
Jul 28 16:01:40.314018 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:01:40.313943    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9155611b-3e3d-4c28-9d34-c5a27ffea78e-kserve-provision-location\") pod \"isvc-canary-stop-predictor-86dc9fdb65-snddz\" (UID: \"9155611b-3e3d-4c28-9d34-c5a27ffea78e\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-86dc9fdb65-snddz"
Jul 28 16:01:40.314018 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:01:40.313969    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9155611b-3e3d-4c28-9d34-c5a27ffea78e-cabundle-cert\") pod \"isvc-canary-stop-predictor-86dc9fdb65-snddz\" (UID: \"9155611b-3e3d-4c28-9d34-c5a27ffea78e\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-86dc9fdb65-snddz"
Jul 28 16:01:40.314085 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:01:40.314015    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9bb04ac6-21f5-4467-93a8-1689b729ec93-kserve-provision-location\") pod \"isvc-canary-stop-v2-predictor-77f778d864-vgkzc\" (UID: \"9bb04ac6-21f5-4467-93a8-1689b729ec93\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-77f778d864-vgkzc"
Jul 28 16:01:40.314085 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:01:40.314043    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9bb04ac6-21f5-4467-93a8-1689b729ec93-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-v2-predictor-77f778d864-vgkzc\" (UID: \"9bb04ac6-21f5-4467-93a8-1689b729ec93\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-77f778d864-vgkzc"
Jul 28 16:01:40.314085 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:01:40.314075    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9bb04ac6-21f5-4467-93a8-1689b729ec93-proxy-tls\") pod \"isvc-canary-stop-v2-predictor-77f778d864-vgkzc\" (UID: \"9bb04ac6-21f5-4467-93a8-1689b729ec93\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-77f778d864-vgkzc"
Jul 28 16:01:40.378514 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:01:40.378483    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22nh9_ebf75c0d-1597-4a1c-b668-dc2a041eb35e/konnectivity-agent/0.log" limit=1024
Jul 28 16:01:40.414902 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:01:40.414867    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sjdc2\" (UniqueName: \"kubernetes.io/projected/9155611b-3e3d-4c28-9d34-c5a27ffea78e-kube-api-access-sjdc2\") pod \"isvc-canary-stop-predictor-86dc9fdb65-snddz\" (UID: \"9155611b-3e3d-4c28-9d34-c5a27ffea78e\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-86dc9fdb65-snddz"
Jul 28 16:01:40.414902 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:01:40.414905    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9155611b-3e3d-4c28-9d34-c5a27ffea78e-kserve-provision-location\") pod \"isvc-canary-stop-predictor-86dc9fdb65-snddz\" (UID: \"9155611b-3e3d-4c28-9d34-c5a27ffea78e\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-86dc9fdb65-snddz"
Jul 28 16:01:40.415157 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:01:40.414935    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9155611b-3e3d-4c28-9d34-c5a27ffea78e-cabundle-cert\") pod \"isvc-canary-stop-predictor-86dc9fdb65-snddz\" (UID: \"9155611b-3e3d-4c28-9d34-c5a27ffea78e\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-86dc9fdb65-snddz"
Jul 28 16:01:40.415157 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:01:40.414958    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9bb04ac6-21f5-4467-93a8-1689b729ec93-kserve-provision-location\") pod \"isvc-canary-stop-v2-predictor-77f778d864-vgkzc\" (UID: \"9bb04ac6-21f5-4467-93a8-1689b729ec93\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-77f778d864-vgkzc"
Jul 28 16:01:40.415157 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:01:40.414985    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9bb04ac6-21f5-4467-93a8-1689b729ec93-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-v2-predictor-77f778d864-vgkzc\" (UID: \"9bb04ac6-21f5-4467-93a8-1689b729ec93\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-77f778d864-vgkzc"
Jul 28 16:01:40.415157 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:01:40.415012    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9bb04ac6-21f5-4467-93a8-1689b729ec93-proxy-tls\") pod \"isvc-canary-stop-v2-predictor-77f778d864-vgkzc\" (UID: \"9bb04ac6-21f5-4467-93a8-1689b729ec93\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-77f778d864-vgkzc"
Jul 28 16:01:40.415157 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:01:40.415038    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fltcs\" (UniqueName: \"kubernetes.io/projected/9bb04ac6-21f5-4467-93a8-1689b729ec93-kube-api-access-fltcs\") pod \"isvc-canary-stop-v2-predictor-77f778d864-vgkzc\" (UID: \"9bb04ac6-21f5-4467-93a8-1689b729ec93\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-77f778d864-vgkzc"
Jul 28 16:01:40.415157 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:01:40.415072    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9155611b-3e3d-4c28-9d34-c5a27ffea78e-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-predictor-86dc9fdb65-snddz\" (UID: \"9155611b-3e3d-4c28-9d34-c5a27ffea78e\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-86dc9fdb65-snddz"
Jul 28 16:01:40.415157 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:01:40.415141    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9155611b-3e3d-4c28-9d34-c5a27ffea78e-proxy-tls\") pod \"isvc-canary-stop-predictor-86dc9fdb65-snddz\" (UID: \"9155611b-3e3d-4c28-9d34-c5a27ffea78e\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-86dc9fdb65-snddz"
Jul 28 16:01:40.415515 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:01:40.415194    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9bb04ac6-21f5-4467-93a8-1689b729ec93-cabundle-cert\") pod \"isvc-canary-stop-v2-predictor-77f778d864-vgkzc\" (UID: \"9bb04ac6-21f5-4467-93a8-1689b729ec93\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-77f778d864-vgkzc"
Jul 28 16:01:40.415772 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:01:40.415730    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9155611b-3e3d-4c28-9d34-c5a27ffea78e-cabundle-cert\") pod \"isvc-canary-stop-predictor-86dc9fdb65-snddz\" (UID: \"9155611b-3e3d-4c28-9d34-c5a27ffea78e\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-86dc9fdb65-snddz"
Jul 28 16:01:40.415772 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:01:40.415730    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9bb04ac6-21f5-4467-93a8-1689b729ec93-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-v2-predictor-77f778d864-vgkzc\" (UID: \"9bb04ac6-21f5-4467-93a8-1689b729ec93\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-77f778d864-vgkzc"
Jul 28 16:01:40.415961 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:01:40.415817    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9bb04ac6-21f5-4467-93a8-1689b729ec93-kserve-provision-location\") pod \"isvc-canary-stop-v2-predictor-77f778d864-vgkzc\" (UID: \"9bb04ac6-21f5-4467-93a8-1689b729ec93\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-77f778d864-vgkzc"
Jul 28 16:01:40.415961 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:01:40.415849    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9155611b-3e3d-4c28-9d34-c5a27ffea78e-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-predictor-86dc9fdb65-snddz\" (UID: \"9155611b-3e3d-4c28-9d34-c5a27ffea78e\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-86dc9fdb65-snddz"
Jul 28 16:01:40.415961 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:01:40.415868    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9155611b-3e3d-4c28-9d34-c5a27ffea78e-kserve-provision-location\") pod \"isvc-canary-stop-predictor-86dc9fdb65-snddz\" (UID: \"9155611b-3e3d-4c28-9d34-c5a27ffea78e\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-86dc9fdb65-snddz"
Jul 28 16:01:40.415961 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:01:40.415884    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9bb04ac6-21f5-4467-93a8-1689b729ec93-cabundle-cert\") pod \"isvc-canary-stop-v2-predictor-77f778d864-vgkzc\" (UID: \"9bb04ac6-21f5-4467-93a8-1689b729ec93\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-77f778d864-vgkzc"
Jul 28 16:01:40.417587 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:01:40.417559    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9bb04ac6-21f5-4467-93a8-1689b729ec93-proxy-tls\") pod \"isvc-canary-stop-v2-predictor-77f778d864-vgkzc\" (UID: \"9bb04ac6-21f5-4467-93a8-1689b729ec93\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-77f778d864-vgkzc"
Jul 28 16:01:40.417678 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:01:40.417648    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9155611b-3e3d-4c28-9d34-c5a27ffea78e-proxy-tls\") pod \"isvc-canary-stop-predictor-86dc9fdb65-snddz\" (UID: \"9155611b-3e3d-4c28-9d34-c5a27ffea78e\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-86dc9fdb65-snddz"
Jul 28 16:01:40.423562 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:01:40.423538    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sjdc2\" (UniqueName: \"kubernetes.io/projected/9155611b-3e3d-4c28-9d34-c5a27ffea78e-kube-api-access-sjdc2\") pod \"isvc-canary-stop-predictor-86dc9fdb65-snddz\" (UID: \"9155611b-3e3d-4c28-9d34-c5a27ffea78e\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-86dc9fdb65-snddz"
Jul 28 16:01:40.423644 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:01:40.423625    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fltcs\" (UniqueName: \"kubernetes.io/projected/9bb04ac6-21f5-4467-93a8-1689b729ec93-kube-api-access-fltcs\") pod \"isvc-canary-stop-v2-predictor-77f778d864-vgkzc\" (UID: \"9bb04ac6-21f5-4467-93a8-1689b729ec93\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-77f778d864-vgkzc"
Jul 28 16:01:40.486152 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:01:40.486117    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-86dc9fdb65-snddz"
Jul 28 16:01:40.504332 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:01:40.504303    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-77f778d864-vgkzc"
Jul 28 16:01:40.642444 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:01:40.642416    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-86dc9fdb65-snddz"]
Jul 28 16:01:40.644296 ip-10-0-131-121 kubenswrapper[2568]: W0728 16:01:40.644262    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9155611b_3e3d_4c28_9d34_c5a27ffea78e.slice/crio-1ac973122269dacc636dc3e82410aa0ef4250389ad0110ff4f91984c92a6977e WatchSource:0}: Error finding container 1ac973122269dacc636dc3e82410aa0ef4250389ad0110ff4f91984c92a6977e: Status 404 returned error can't find the container with id 1ac973122269dacc636dc3e82410aa0ef4250389ad0110ff4f91984c92a6977e
Jul 28 16:01:40.662587 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:01:40.662558    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-77f778d864-vgkzc"]
Jul 28 16:01:40.664938 ip-10-0-131-121 kubenswrapper[2568]: W0728 16:01:40.664911    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9bb04ac6_21f5_4467_93a8_1689b729ec93.slice/crio-a73fb0e0525b6d81e942036116b706c43b2d733f47eb2b0c4750bba642550c1c WatchSource:0}: Error finding container a73fb0e0525b6d81e942036116b706c43b2d733f47eb2b0c4750bba642550c1c: Status 404 returned error can't find the container with id a73fb0e0525b6d81e942036116b706c43b2d733f47eb2b0c4750bba642550c1c
Jul 28 16:01:41.153011 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:01:41.152972    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-86dc9fdb65-snddz" event={"ID":"9155611b-3e3d-4c28-9d34-c5a27ffea78e","Type":"ContainerStarted","Data":"359ab86a06108b74b80ef24594690427a0b6b96a3909d9ce5cd523ad278ef9cd"}
Jul 28 16:01:41.153494 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:01:41.153089    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-86dc9fdb65-snddz" event={"ID":"9155611b-3e3d-4c28-9d34-c5a27ffea78e","Type":"ContainerStarted","Data":"1ac973122269dacc636dc3e82410aa0ef4250389ad0110ff4f91984c92a6977e"}
Jul 28 16:01:41.154611 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:01:41.154582    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-77f778d864-vgkzc" event={"ID":"9bb04ac6-21f5-4467-93a8-1689b729ec93","Type":"ContainerStarted","Data":"f2a3999dab075bb9c0725b4ecd46cef81951cc04105900709f17469aa412a3d2"}
Jul 28 16:01:41.154715 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:01:41.154649    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-77f778d864-vgkzc" event={"ID":"9bb04ac6-21f5-4467-93a8-1689b729ec93","Type":"ContainerStarted","Data":"a73fb0e0525b6d81e942036116b706c43b2d733f47eb2b0c4750bba642550c1c"}
Jul 28 16:01:45.179081 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:01:45.179045    2568 generic.go:356] "Generic (PLEG): container finished" podID="9bb04ac6-21f5-4467-93a8-1689b729ec93" containerID="f2a3999dab075bb9c0725b4ecd46cef81951cc04105900709f17469aa412a3d2" exitCode=0
Jul 28 16:01:45.179590 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:01:45.179123    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-77f778d864-vgkzc" event={"ID":"9bb04ac6-21f5-4467-93a8-1689b729ec93","Type":"ContainerDied","Data":"f2a3999dab075bb9c0725b4ecd46cef81951cc04105900709f17469aa412a3d2"}
Jul 28 16:01:45.180683 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:01:45.180658    2568 generic.go:356] "Generic (PLEG): container finished" podID="9155611b-3e3d-4c28-9d34-c5a27ffea78e" containerID="359ab86a06108b74b80ef24594690427a0b6b96a3909d9ce5cd523ad278ef9cd" exitCode=0
Jul 28 16:01:45.180801 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:01:45.180726    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-86dc9fdb65-snddz" event={"ID":"9155611b-3e3d-4c28-9d34-c5a27ffea78e","Type":"ContainerDied","Data":"359ab86a06108b74b80ef24594690427a0b6b96a3909d9ce5cd523ad278ef9cd"}
Jul 28 16:01:46.187907 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:01:46.187878    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-77f778d864-vgkzc" event={"ID":"9bb04ac6-21f5-4467-93a8-1689b729ec93","Type":"ContainerStarted","Data":"2c1a717258da7125652c0d6572672b10e5801ccccab46bf82df3827c314f8d26"}
Jul 28 16:01:46.188339 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:01:46.187939    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-77f778d864-vgkzc" event={"ID":"9bb04ac6-21f5-4467-93a8-1689b729ec93","Type":"ContainerStarted","Data":"00d727c3056dea139594bf9abb3d5fa728a746d06db638a30f0b8c66b7cd3e77"}
Jul 28 16:01:46.188339 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:01:46.188157    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-77f778d864-vgkzc"
Jul 28 16:01:46.188339 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:01:46.188201    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-77f778d864-vgkzc"
Jul 28 16:01:46.189837 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:01:46.189809    2568 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-77f778d864-vgkzc" podUID="9bb04ac6-21f5-4467-93a8-1689b729ec93" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused"
Jul 28 16:01:46.189994 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:01:46.189973    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-86dc9fdb65-snddz" event={"ID":"9155611b-3e3d-4c28-9d34-c5a27ffea78e","Type":"ContainerStarted","Data":"f3914f388fb32d0130d1555f06d38b3eee542da48dddbcb2ba671e18d3fbf44e"}
Jul 28 16:01:46.190046 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:01:46.190028    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-86dc9fdb65-snddz" event={"ID":"9155611b-3e3d-4c28-9d34-c5a27ffea78e","Type":"ContainerStarted","Data":"8f7fead927220e49f45ee1c099f821a889c40cc5eeac559f1f5d51233e29d5b7"}
Jul 28 16:01:46.190209 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:01:46.190192    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-86dc9fdb65-snddz"
Jul 28 16:01:46.190255 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:01:46.190244    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-86dc9fdb65-snddz"
Jul 28 16:01:46.191291 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:01:46.191265    2568 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-86dc9fdb65-snddz" podUID="9155611b-3e3d-4c28-9d34-c5a27ffea78e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:8080: connect: connection refused"
Jul 28 16:01:46.207648 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:01:46.207601    2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-77f778d864-vgkzc" podStartSLOduration=2.340489731 podStartE2EDuration="6.207584341s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.86709461s" isStatefulPod=false podCreationTimestamp="2026-07-28 16:01:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 16:01:46.206814685 +0000 UTC m=+1686.392009853" watchObservedRunningTime="2026-07-28 16:01:46.207584341 +0000 UTC m=+1686.392779510"
Jul 28 16:01:46.225583 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:01:46.225524    2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-86dc9fdb65-snddz" podStartSLOduration=2.362460114 podStartE2EDuration="6.225508392s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.863048278s" isStatefulPod=false podCreationTimestamp="2026-07-28 16:01:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 16:01:46.223537743 +0000 UTC m=+1686.408732936" watchObservedRunningTime="2026-07-28 16:01:46.225508392 +0000 UTC m=+1686.410703560"
Jul 28 16:01:47.194849 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:01:47.194806    2568 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-86dc9fdb65-snddz" podUID="9155611b-3e3d-4c28-9d34-c5a27ffea78e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:8080: connect: connection refused"
Jul 28 16:01:47.195250 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:01:47.194895    2568 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-77f778d864-vgkzc" podUID="9bb04ac6-21f5-4467-93a8-1689b729ec93" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused"
Jul 28 16:01:52.199180 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:01:52.199146    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-86dc9fdb65-snddz"
Jul 28 16:01:52.199571 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:01:52.199518    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-77f778d864-vgkzc"
Jul 28 16:01:52.199609 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:01:52.199580    2568 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-86dc9fdb65-snddz" podUID="9155611b-3e3d-4c28-9d34-c5a27ffea78e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:8080: connect: connection refused"
Jul 28 16:01:52.200027 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:01:52.200005    2568 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-77f778d864-vgkzc" podUID="9bb04ac6-21f5-4467-93a8-1689b729ec93" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused"
Jul 28 16:02:02.200384 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:02:02.200297    2568 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-77f778d864-vgkzc" podUID="9bb04ac6-21f5-4467-93a8-1689b729ec93" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused"
Jul 28 16:02:02.200384 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:02:02.200297    2568 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-86dc9fdb65-snddz" podUID="9155611b-3e3d-4c28-9d34-c5a27ffea78e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:8080: connect: connection refused"
Jul 28 16:02:12.199814 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:02:12.199756    2568 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-86dc9fdb65-snddz" podUID="9155611b-3e3d-4c28-9d34-c5a27ffea78e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:8080: connect: connection refused"
Jul 28 16:02:12.200385 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:02:12.199924    2568 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-77f778d864-vgkzc" podUID="9bb04ac6-21f5-4467-93a8-1689b729ec93" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused"
Jul 28 16:02:22.199643 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:02:22.199607    2568 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-86dc9fdb65-snddz" podUID="9155611b-3e3d-4c28-9d34-c5a27ffea78e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:8080: connect: connection refused"
Jul 28 16:02:22.200111 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:02:22.199978    2568 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-77f778d864-vgkzc" podUID="9bb04ac6-21f5-4467-93a8-1689b729ec93" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused"
Jul 28 16:02:32.199859 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:02:32.199803    2568 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-86dc9fdb65-snddz" podUID="9155611b-3e3d-4c28-9d34-c5a27ffea78e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:8080: connect: connection refused"
Jul 28 16:02:32.200270 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:02:32.199878    2568 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-77f778d864-vgkzc" podUID="9bb04ac6-21f5-4467-93a8-1689b729ec93" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused"
Jul 28 16:02:42.199727 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:02:42.199688    2568 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-86dc9fdb65-snddz" podUID="9155611b-3e3d-4c28-9d34-c5a27ffea78e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:8080: connect: connection refused"
Jul 28 16:02:42.200137 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:02:42.199946    2568 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-77f778d864-vgkzc" podUID="9bb04ac6-21f5-4467-93a8-1689b729ec93" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused"
Jul 28 16:02:47.139055 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:02:47.139011    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22nh9_ebf75c0d-1597-4a1c-b668-dc2a041eb35e/konnectivity-agent/0.log" limit=1024
Jul 28 16:02:52.200169 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:02:52.200122    2568 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-86dc9fdb65-snddz" podUID="9155611b-3e3d-4c28-9d34-c5a27ffea78e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:8080: connect: connection refused"
Jul 28 16:02:52.200565 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:02:52.200123    2568 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-77f778d864-vgkzc" podUID="9bb04ac6-21f5-4467-93a8-1689b729ec93" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused"
Jul 28 16:03:02.199982 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:03:02.199951    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-86dc9fdb65-snddz"
Jul 28 16:03:02.200712 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:03:02.200693    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-77f778d864-vgkzc"
Jul 28 16:03:10.209235 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:03:10.209195    2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-86dc9fdb65-snddz"]
Jul 28 16:03:10.209711 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:03:10.209452    2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-86dc9fdb65-snddz" podUID="9155611b-3e3d-4c28-9d34-c5a27ffea78e" containerName="kserve-container" containerID="cri-o://8f7fead927220e49f45ee1c099f821a889c40cc5eeac559f1f5d51233e29d5b7" gracePeriod=30
Jul 28 16:03:10.209711 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:03:10.209476    2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-86dc9fdb65-snddz" podUID="9155611b-3e3d-4c28-9d34-c5a27ffea78e" containerName="kube-rbac-proxy" containerID="cri-o://f3914f388fb32d0130d1555f06d38b3eee542da48dddbcb2ba671e18d3fbf44e" gracePeriod=30
Jul 28 16:03:10.216115 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:03:10.215795    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22nh9_ebf75c0d-1597-4a1c-b668-dc2a041eb35e/konnectivity-agent/0.log" limit=1024
Jul 28 16:03:10.251269 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:03:10.251236    2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-77f778d864-vgkzc"]
Jul 28 16:03:10.251499 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:03:10.251477    2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-77f778d864-vgkzc" podUID="9bb04ac6-21f5-4467-93a8-1689b729ec93" containerName="kserve-container" containerID="cri-o://00d727c3056dea139594bf9abb3d5fa728a746d06db638a30f0b8c66b7cd3e77" gracePeriod=30
Jul 28 16:03:10.251553 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:03:10.251501    2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-77f778d864-vgkzc" podUID="9bb04ac6-21f5-4467-93a8-1689b729ec93" containerName="kube-rbac-proxy" containerID="cri-o://2c1a717258da7125652c0d6572672b10e5801ccccab46bf82df3827c314f8d26" gracePeriod=30
Jul 28 16:03:10.293886 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:03:10.293854    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22nh9_ebf75c0d-1597-4a1c-b668-dc2a041eb35e/konnectivity-agent/0.log" limit=1024
Jul 28 16:03:10.556684 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:03:10.556651    2568 generic.go:356] "Generic (PLEG): container finished" podID="9bb04ac6-21f5-4467-93a8-1689b729ec93" containerID="2c1a717258da7125652c0d6572672b10e5801ccccab46bf82df3827c314f8d26" exitCode=2
Jul 28 16:03:10.556901 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:03:10.556731    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-77f778d864-vgkzc" event={"ID":"9bb04ac6-21f5-4467-93a8-1689b729ec93","Type":"ContainerDied","Data":"2c1a717258da7125652c0d6572672b10e5801ccccab46bf82df3827c314f8d26"}
Jul 28 16:03:10.558734 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:03:10.558712    2568 generic.go:356] "Generic (PLEG): container finished" podID="9155611b-3e3d-4c28-9d34-c5a27ffea78e" containerID="f3914f388fb32d0130d1555f06d38b3eee542da48dddbcb2ba671e18d3fbf44e" exitCode=2
Jul 28 16:03:10.558893 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:03:10.558810    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-86dc9fdb65-snddz" event={"ID":"9155611b-3e3d-4c28-9d34-c5a27ffea78e","Type":"ContainerDied","Data":"f3914f388fb32d0130d1555f06d38b3eee542da48dddbcb2ba671e18d3fbf44e"}
Jul 28 16:03:12.195642 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:03:12.195596    2568 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-77f778d864-vgkzc" podUID="9bb04ac6-21f5-4467-93a8-1689b729ec93" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.33:8643/healthz\": dial tcp 10.134.0.33:8643: connect: connection refused"
Jul 28 16:03:12.195642 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:03:12.195627    2568 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-86dc9fdb65-snddz" podUID="9155611b-3e3d-4c28-9d34-c5a27ffea78e" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.32:8643/healthz\": dial tcp 10.134.0.32:8643: connect: connection refused"
Jul 28 16:03:12.200222 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:03:12.200195    2568 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-77f778d864-vgkzc" podUID="9bb04ac6-21f5-4467-93a8-1689b729ec93" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused"
Jul 28 16:03:12.200276 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:03:12.200200    2568 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-86dc9fdb65-snddz" podUID="9155611b-3e3d-4c28-9d34-c5a27ffea78e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:8080: connect: connection refused"
Jul 28 16:03:15.292424 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:03:15.292350    2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-86dc9fdb65-snddz"
Jul 28 16:03:15.337244 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:03:15.337209    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9155611b-3e3d-4c28-9d34-c5a27ffea78e-proxy-tls\") pod \"9155611b-3e3d-4c28-9d34-c5a27ffea78e\" (UID: \"9155611b-3e3d-4c28-9d34-c5a27ffea78e\") "
Jul 28 16:03:15.337435 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:03:15.337253    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9155611b-3e3d-4c28-9d34-c5a27ffea78e-kserve-provision-location\") pod \"9155611b-3e3d-4c28-9d34-c5a27ffea78e\" (UID: \"9155611b-3e3d-4c28-9d34-c5a27ffea78e\") "
Jul 28 16:03:15.337435 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:03:15.337282    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sjdc2\" (UniqueName: \"kubernetes.io/projected/9155611b-3e3d-4c28-9d34-c5a27ffea78e-kube-api-access-sjdc2\") pod \"9155611b-3e3d-4c28-9d34-c5a27ffea78e\" (UID: \"9155611b-3e3d-4c28-9d34-c5a27ffea78e\") "
Jul 28 16:03:15.337435 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:03:15.337349    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9155611b-3e3d-4c28-9d34-c5a27ffea78e-kube-rbac-proxy-sar-config\") pod \"9155611b-3e3d-4c28-9d34-c5a27ffea78e\" (UID: \"9155611b-3e3d-4c28-9d34-c5a27ffea78e\") "
Jul 28 16:03:15.337435 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:03:15.337374    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9155611b-3e3d-4c28-9d34-c5a27ffea78e-cabundle-cert\") pod \"9155611b-3e3d-4c28-9d34-c5a27ffea78e\" (UID: \"9155611b-3e3d-4c28-9d34-c5a27ffea78e\") "
Jul 28 16:03:15.337632 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:03:15.337604    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9155611b-3e3d-4c28-9d34-c5a27ffea78e-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "9155611b-3e3d-4c28-9d34-c5a27ffea78e" (UID: "9155611b-3e3d-4c28-9d34-c5a27ffea78e"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 16:03:15.337911 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:03:15.337879    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9155611b-3e3d-4c28-9d34-c5a27ffea78e-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "9155611b-3e3d-4c28-9d34-c5a27ffea78e" (UID: "9155611b-3e3d-4c28-9d34-c5a27ffea78e"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 16:03:15.338014 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:03:15.337936    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9155611b-3e3d-4c28-9d34-c5a27ffea78e-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "9155611b-3e3d-4c28-9d34-c5a27ffea78e" (UID: "9155611b-3e3d-4c28-9d34-c5a27ffea78e"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 16:03:15.339903 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:03:15.339868    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9155611b-3e3d-4c28-9d34-c5a27ffea78e-kube-api-access-sjdc2" (OuterVolumeSpecName: "kube-api-access-sjdc2") pod "9155611b-3e3d-4c28-9d34-c5a27ffea78e" (UID: "9155611b-3e3d-4c28-9d34-c5a27ffea78e"). InnerVolumeSpecName "kube-api-access-sjdc2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 16:03:15.340209 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:03:15.339913    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9155611b-3e3d-4c28-9d34-c5a27ffea78e-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "9155611b-3e3d-4c28-9d34-c5a27ffea78e" (UID: "9155611b-3e3d-4c28-9d34-c5a27ffea78e"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 16:03:15.401689 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:03:15.401663    2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-77f778d864-vgkzc"
Jul 28 16:03:15.438350 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:03:15.438305    2568 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9155611b-3e3d-4c28-9d34-c5a27ffea78e-proxy-tls\") on node \"ip-10-0-131-121.ec2.internal\" DevicePath \"\""
Jul 28 16:03:15.438350 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:03:15.438335    2568 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9155611b-3e3d-4c28-9d34-c5a27ffea78e-kserve-provision-location\") on node \"ip-10-0-131-121.ec2.internal\" DevicePath \"\""
Jul 28 16:03:15.438350 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:03:15.438344    2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sjdc2\" (UniqueName: \"kubernetes.io/projected/9155611b-3e3d-4c28-9d34-c5a27ffea78e-kube-api-access-sjdc2\") on node \"ip-10-0-131-121.ec2.internal\" DevicePath \"\""
Jul 28 16:03:15.438350 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:03:15.438355    2568 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9155611b-3e3d-4c28-9d34-c5a27ffea78e-kube-rbac-proxy-sar-config\") on node \"ip-10-0-131-121.ec2.internal\" DevicePath \"\""
Jul 28 16:03:15.438350 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:03:15.438365    2568 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9155611b-3e3d-4c28-9d34-c5a27ffea78e-cabundle-cert\") on node \"ip-10-0-131-121.ec2.internal\" DevicePath \"\""
Jul 28 16:03:15.538926 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:03:15.538886    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9bb04ac6-21f5-4467-93a8-1689b729ec93-cabundle-cert\") pod \"9bb04ac6-21f5-4467-93a8-1689b729ec93\" (UID: \"9bb04ac6-21f5-4467-93a8-1689b729ec93\") "
Jul 28 16:03:15.539120 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:03:15.538939    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9bb04ac6-21f5-4467-93a8-1689b729ec93-kube-rbac-proxy-sar-config\") pod \"9bb04ac6-21f5-4467-93a8-1689b729ec93\" (UID: \"9bb04ac6-21f5-4467-93a8-1689b729ec93\") "
Jul 28 16:03:15.539120 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:03:15.538997    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fltcs\" (UniqueName: \"kubernetes.io/projected/9bb04ac6-21f5-4467-93a8-1689b729ec93-kube-api-access-fltcs\") pod \"9bb04ac6-21f5-4467-93a8-1689b729ec93\" (UID: \"9bb04ac6-21f5-4467-93a8-1689b729ec93\") "
Jul 28 16:03:15.539120 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:03:15.539025    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9bb04ac6-21f5-4467-93a8-1689b729ec93-proxy-tls\") pod \"9bb04ac6-21f5-4467-93a8-1689b729ec93\" (UID: \"9bb04ac6-21f5-4467-93a8-1689b729ec93\") "
Jul 28 16:03:15.539120 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:03:15.539062    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9bb04ac6-21f5-4467-93a8-1689b729ec93-kserve-provision-location\") pod \"9bb04ac6-21f5-4467-93a8-1689b729ec93\" (UID: \"9bb04ac6-21f5-4467-93a8-1689b729ec93\") "
Jul 28 16:03:15.539338 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:03:15.539303    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9bb04ac6-21f5-4467-93a8-1689b729ec93-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "9bb04ac6-21f5-4467-93a8-1689b729ec93" (UID: "9bb04ac6-21f5-4467-93a8-1689b729ec93"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 16:03:15.539395 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:03:15.539340    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9bb04ac6-21f5-4467-93a8-1689b729ec93-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "9bb04ac6-21f5-4467-93a8-1689b729ec93" (UID: "9bb04ac6-21f5-4467-93a8-1689b729ec93"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 16:03:15.539446 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:03:15.539393    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9bb04ac6-21f5-4467-93a8-1689b729ec93-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "9bb04ac6-21f5-4467-93a8-1689b729ec93" (UID: "9bb04ac6-21f5-4467-93a8-1689b729ec93"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 16:03:15.541172 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:03:15.541143    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9bb04ac6-21f5-4467-93a8-1689b729ec93-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "9bb04ac6-21f5-4467-93a8-1689b729ec93" (UID: "9bb04ac6-21f5-4467-93a8-1689b729ec93"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 16:03:15.541290 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:03:15.541222    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9bb04ac6-21f5-4467-93a8-1689b729ec93-kube-api-access-fltcs" (OuterVolumeSpecName: "kube-api-access-fltcs") pod "9bb04ac6-21f5-4467-93a8-1689b729ec93" (UID: "9bb04ac6-21f5-4467-93a8-1689b729ec93"). InnerVolumeSpecName "kube-api-access-fltcs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 16:03:15.584600 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:03:15.584515    2568 generic.go:356] "Generic (PLEG): container finished" podID="9bb04ac6-21f5-4467-93a8-1689b729ec93" containerID="00d727c3056dea139594bf9abb3d5fa728a746d06db638a30f0b8c66b7cd3e77" exitCode=0
Jul 28 16:03:15.584600 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:03:15.584578    2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-77f778d864-vgkzc"
Jul 28 16:03:15.584814 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:03:15.584599    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-77f778d864-vgkzc" event={"ID":"9bb04ac6-21f5-4467-93a8-1689b729ec93","Type":"ContainerDied","Data":"00d727c3056dea139594bf9abb3d5fa728a746d06db638a30f0b8c66b7cd3e77"}
Jul 28 16:03:15.584814 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:03:15.584667    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-77f778d864-vgkzc" event={"ID":"9bb04ac6-21f5-4467-93a8-1689b729ec93","Type":"ContainerDied","Data":"a73fb0e0525b6d81e942036116b706c43b2d733f47eb2b0c4750bba642550c1c"}
Jul 28 16:03:15.584814 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:03:15.584691    2568 scope.go:117] "RemoveContainer" containerID="2c1a717258da7125652c0d6572672b10e5801ccccab46bf82df3827c314f8d26"
Jul 28 16:03:15.586714 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:03:15.586693    2568 generic.go:356] "Generic (PLEG): container finished" podID="9155611b-3e3d-4c28-9d34-c5a27ffea78e" containerID="8f7fead927220e49f45ee1c099f821a889c40cc5eeac559f1f5d51233e29d5b7" exitCode=0
Jul 28 16:03:15.586855 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:03:15.586726    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-86dc9fdb65-snddz" event={"ID":"9155611b-3e3d-4c28-9d34-c5a27ffea78e","Type":"ContainerDied","Data":"8f7fead927220e49f45ee1c099f821a889c40cc5eeac559f1f5d51233e29d5b7"}
Jul 28 16:03:15.586855 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:03:15.586746    2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-86dc9fdb65-snddz"
Jul 28 16:03:15.586855 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:03:15.586790    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-86dc9fdb65-snddz" event={"ID":"9155611b-3e3d-4c28-9d34-c5a27ffea78e","Type":"ContainerDied","Data":"1ac973122269dacc636dc3e82410aa0ef4250389ad0110ff4f91984c92a6977e"}
Jul 28 16:03:15.594494 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:03:15.594472    2568 scope.go:117] "RemoveContainer" containerID="00d727c3056dea139594bf9abb3d5fa728a746d06db638a30f0b8c66b7cd3e77"
Jul 28 16:03:15.602605 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:03:15.602583    2568 scope.go:117] "RemoveContainer" containerID="f2a3999dab075bb9c0725b4ecd46cef81951cc04105900709f17469aa412a3d2"
Jul 28 16:03:15.610586 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:03:15.610567    2568 scope.go:117] "RemoveContainer" containerID="2c1a717258da7125652c0d6572672b10e5801ccccab46bf82df3827c314f8d26"
Jul 28 16:03:15.610978 ip-10-0-131-121 kubenswrapper[2568]: E0728 16:03:15.610960    2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2c1a717258da7125652c0d6572672b10e5801ccccab46bf82df3827c314f8d26\": container with ID starting with 2c1a717258da7125652c0d6572672b10e5801ccccab46bf82df3827c314f8d26 not found: ID does not exist" containerID="2c1a717258da7125652c0d6572672b10e5801ccccab46bf82df3827c314f8d26"
Jul 28 16:03:15.611044 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:03:15.610989    2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2c1a717258da7125652c0d6572672b10e5801ccccab46bf82df3827c314f8d26"} err="failed to get container status \"2c1a717258da7125652c0d6572672b10e5801ccccab46bf82df3827c314f8d26\": rpc error: code = NotFound desc = could not find container \"2c1a717258da7125652c0d6572672b10e5801ccccab46bf82df3827c314f8d26\": container with ID starting with 2c1a717258da7125652c0d6572672b10e5801ccccab46bf82df3827c314f8d26 not found: ID does not exist"
Jul 28 16:03:15.611044 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:03:15.611006    2568 scope.go:117] "RemoveContainer" containerID="00d727c3056dea139594bf9abb3d5fa728a746d06db638a30f0b8c66b7cd3e77"
Jul 28 16:03:15.611256 ip-10-0-131-121 kubenswrapper[2568]: E0728 16:03:15.611237    2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"00d727c3056dea139594bf9abb3d5fa728a746d06db638a30f0b8c66b7cd3e77\": container with ID starting with 00d727c3056dea139594bf9abb3d5fa728a746d06db638a30f0b8c66b7cd3e77 not found: ID does not exist" containerID="00d727c3056dea139594bf9abb3d5fa728a746d06db638a30f0b8c66b7cd3e77"
Jul 28 16:03:15.611305 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:03:15.611260    2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"00d727c3056dea139594bf9abb3d5fa728a746d06db638a30f0b8c66b7cd3e77"} err="failed to get container status \"00d727c3056dea139594bf9abb3d5fa728a746d06db638a30f0b8c66b7cd3e77\": rpc error: code = NotFound desc = could not find container \"00d727c3056dea139594bf9abb3d5fa728a746d06db638a30f0b8c66b7cd3e77\": container with ID starting with 00d727c3056dea139594bf9abb3d5fa728a746d06db638a30f0b8c66b7cd3e77 not found: ID does not exist"
Jul 28 16:03:15.611305 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:03:15.611274    2568 scope.go:117] "RemoveContainer" containerID="f2a3999dab075bb9c0725b4ecd46cef81951cc04105900709f17469aa412a3d2"
Jul 28 16:03:15.611488 ip-10-0-131-121 kubenswrapper[2568]: E0728 16:03:15.611472    2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f2a3999dab075bb9c0725b4ecd46cef81951cc04105900709f17469aa412a3d2\": container with ID starting with f2a3999dab075bb9c0725b4ecd46cef81951cc04105900709f17469aa412a3d2 not found: ID does not exist" containerID="f2a3999dab075bb9c0725b4ecd46cef81951cc04105900709f17469aa412a3d2"
Jul 28 16:03:15.611526 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:03:15.611496    2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f2a3999dab075bb9c0725b4ecd46cef81951cc04105900709f17469aa412a3d2"} err="failed to get container status \"f2a3999dab075bb9c0725b4ecd46cef81951cc04105900709f17469aa412a3d2\": rpc error: code = NotFound desc = could not find container \"f2a3999dab075bb9c0725b4ecd46cef81951cc04105900709f17469aa412a3d2\": container with ID starting with f2a3999dab075bb9c0725b4ecd46cef81951cc04105900709f17469aa412a3d2 not found: ID does not exist"
Jul 28 16:03:15.611526 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:03:15.611508    2568 scope.go:117] "RemoveContainer" containerID="f3914f388fb32d0130d1555f06d38b3eee542da48dddbcb2ba671e18d3fbf44e"
Jul 28 16:03:15.614289 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:03:15.614266    2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-77f778d864-vgkzc"]
Jul 28 16:03:15.619118 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:03:15.619096    2568 scope.go:117] "RemoveContainer" containerID="8f7fead927220e49f45ee1c099f821a889c40cc5eeac559f1f5d51233e29d5b7"
Jul 28 16:03:15.621816 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:03:15.621794    2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-77f778d864-vgkzc"]
Jul 28 16:03:15.631075 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:03:15.631056    2568 scope.go:117] "RemoveContainer" containerID="359ab86a06108b74b80ef24594690427a0b6b96a3909d9ce5cd523ad278ef9cd"
Jul 28 16:03:15.638974 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:03:15.638947    2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-86dc9fdb65-snddz"]
Jul 28 16:03:15.639344 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:03:15.639327    2568 scope.go:117] "RemoveContainer" containerID="f3914f388fb32d0130d1555f06d38b3eee542da48dddbcb2ba671e18d3fbf44e"
Jul 28 16:03:15.639653 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:03:15.639634    2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fltcs\" (UniqueName: \"kubernetes.io/projected/9bb04ac6-21f5-4467-93a8-1689b729ec93-kube-api-access-fltcs\") on node \"ip-10-0-131-121.ec2.internal\" DevicePath \"\""
Jul 28 16:03:15.639709 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:03:15.639663    2568 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9bb04ac6-21f5-4467-93a8-1689b729ec93-proxy-tls\") on node \"ip-10-0-131-121.ec2.internal\" DevicePath \"\""
Jul 28 16:03:15.639709 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:03:15.639673    2568 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9bb04ac6-21f5-4467-93a8-1689b729ec93-kserve-provision-location\") on node \"ip-10-0-131-121.ec2.internal\" DevicePath \"\""
Jul 28 16:03:15.639709 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:03:15.639682    2568 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9bb04ac6-21f5-4467-93a8-1689b729ec93-cabundle-cert\") on node \"ip-10-0-131-121.ec2.internal\" DevicePath \"\""
Jul 28 16:03:15.639709 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:03:15.639690    2568 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9bb04ac6-21f5-4467-93a8-1689b729ec93-kube-rbac-proxy-sar-config\") on node \"ip-10-0-131-121.ec2.internal\" DevicePath \"\""
Jul 28 16:03:15.639709 ip-10-0-131-121 kubenswrapper[2568]: E0728 16:03:15.639633    2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f3914f388fb32d0130d1555f06d38b3eee542da48dddbcb2ba671e18d3fbf44e\": container with ID starting with f3914f388fb32d0130d1555f06d38b3eee542da48dddbcb2ba671e18d3fbf44e not found: ID does not exist" containerID="f3914f388fb32d0130d1555f06d38b3eee542da48dddbcb2ba671e18d3fbf44e"
Jul 28 16:03:15.639894 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:03:15.639714    2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f3914f388fb32d0130d1555f06d38b3eee542da48dddbcb2ba671e18d3fbf44e"} err="failed to get container status \"f3914f388fb32d0130d1555f06d38b3eee542da48dddbcb2ba671e18d3fbf44e\": rpc error: code = NotFound desc = could not find container \"f3914f388fb32d0130d1555f06d38b3eee542da48dddbcb2ba671e18d3fbf44e\": container with ID starting with f3914f388fb32d0130d1555f06d38b3eee542da48dddbcb2ba671e18d3fbf44e not found: ID does not exist"
Jul 28 16:03:15.639894 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:03:15.639735    2568 scope.go:117] "RemoveContainer" containerID="8f7fead927220e49f45ee1c099f821a889c40cc5eeac559f1f5d51233e29d5b7"
Jul 28 16:03:15.640058 ip-10-0-131-121 kubenswrapper[2568]: E0728 16:03:15.640039    2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8f7fead927220e49f45ee1c099f821a889c40cc5eeac559f1f5d51233e29d5b7\": container with ID starting with 8f7fead927220e49f45ee1c099f821a889c40cc5eeac559f1f5d51233e29d5b7 not found: ID does not exist" containerID="8f7fead927220e49f45ee1c099f821a889c40cc5eeac559f1f5d51233e29d5b7"
Jul 28 16:03:15.640131 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:03:15.640070    2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8f7fead927220e49f45ee1c099f821a889c40cc5eeac559f1f5d51233e29d5b7"} err="failed to get container status \"8f7fead927220e49f45ee1c099f821a889c40cc5eeac559f1f5d51233e29d5b7\": rpc error: code = NotFound desc = could not find container \"8f7fead927220e49f45ee1c099f821a889c40cc5eeac559f1f5d51233e29d5b7\": container with ID starting with 8f7fead927220e49f45ee1c099f821a889c40cc5eeac559f1f5d51233e29d5b7 not found: ID does not exist"
Jul 28 16:03:15.640131 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:03:15.640094    2568 scope.go:117] "RemoveContainer" containerID="359ab86a06108b74b80ef24594690427a0b6b96a3909d9ce5cd523ad278ef9cd"
Jul 28 16:03:15.640350 ip-10-0-131-121 kubenswrapper[2568]: E0728 16:03:15.640334    2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"359ab86a06108b74b80ef24594690427a0b6b96a3909d9ce5cd523ad278ef9cd\": container with ID starting with 359ab86a06108b74b80ef24594690427a0b6b96a3909d9ce5cd523ad278ef9cd not found: ID does not exist" containerID="359ab86a06108b74b80ef24594690427a0b6b96a3909d9ce5cd523ad278ef9cd"
Jul 28 16:03:15.640390 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:03:15.640355    2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"359ab86a06108b74b80ef24594690427a0b6b96a3909d9ce5cd523ad278ef9cd"} err="failed to get container status \"359ab86a06108b74b80ef24594690427a0b6b96a3909d9ce5cd523ad278ef9cd\": rpc error: code = NotFound desc = could not find container \"359ab86a06108b74b80ef24594690427a0b6b96a3909d9ce5cd523ad278ef9cd\": container with ID starting with 359ab86a06108b74b80ef24594690427a0b6b96a3909d9ce5cd523ad278ef9cd not found: ID does not exist"
Jul 28 16:03:15.644049 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:03:15.644023    2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-86dc9fdb65-snddz"]
Jul 28 16:03:16.294808 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:03:16.294690    2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9155611b-3e3d-4c28-9d34-c5a27ffea78e" path="/var/lib/kubelet/pods/9155611b-3e3d-4c28-9d34-c5a27ffea78e/volumes"
Jul 28 16:03:16.295417 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:03:16.295394    2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9bb04ac6-21f5-4467-93a8-1689b729ec93" path="/var/lib/kubelet/pods/9bb04ac6-21f5-4467-93a8-1689b729ec93/volumes"
Jul 28 16:03:20.428552 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:03:20.428513    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22nh9_ebf75c0d-1597-4a1c-b668-dc2a041eb35e/konnectivity-agent/0.log" limit=1024
Jul 28 16:03:20.478190 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:03:20.478153    2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-primary-9bf0ca-predictor-7969d99c69-qmc4g"]
Jul 28 16:03:20.478472 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:03:20.478460    2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9bb04ac6-21f5-4467-93a8-1689b729ec93" containerName="storage-initializer"
Jul 28 16:03:20.478515 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:03:20.478473    2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="9bb04ac6-21f5-4467-93a8-1689b729ec93" containerName="storage-initializer"
Jul 28 16:03:20.478515 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:03:20.478483    2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9bb04ac6-21f5-4467-93a8-1689b729ec93" containerName="kube-rbac-proxy"
Jul 28 16:03:20.478515 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:03:20.478490    2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="9bb04ac6-21f5-4467-93a8-1689b729ec93" containerName="kube-rbac-proxy"
Jul 28 16:03:20.478515 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:03:20.478497    2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9155611b-3e3d-4c28-9d34-c5a27ffea78e" containerName="storage-initializer"
Jul 28 16:03:20.478515 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:03:20.478502    2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="9155611b-3e3d-4c28-9d34-c5a27ffea78e" containerName="storage-initializer"
Jul 28 16:03:20.478515 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:03:20.478508    2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9bb04ac6-21f5-4467-93a8-1689b729ec93" containerName="kserve-container"
Jul 28 16:03:20.478515 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:03:20.478513    2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="9bb04ac6-21f5-4467-93a8-1689b729ec93" containerName="kserve-container"
Jul 28 16:03:20.478723 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:03:20.478526    2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9155611b-3e3d-4c28-9d34-c5a27ffea78e" containerName="kube-rbac-proxy"
Jul 28 16:03:20.478723 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:03:20.478531    2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="9155611b-3e3d-4c28-9d34-c5a27ffea78e" containerName="kube-rbac-proxy"
Jul 28 16:03:20.478723 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:03:20.478542    2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9155611b-3e3d-4c28-9d34-c5a27ffea78e" containerName="kserve-container"
Jul 28 16:03:20.478723 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:03:20.478547    2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="9155611b-3e3d-4c28-9d34-c5a27ffea78e" containerName="kserve-container"
Jul 28 16:03:20.478723 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:03:20.478594    2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="9bb04ac6-21f5-4467-93a8-1689b729ec93" containerName="kserve-container"
Jul 28 16:03:20.478723 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:03:20.478599    2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="9155611b-3e3d-4c28-9d34-c5a27ffea78e" containerName="kserve-container"
Jul 28 16:03:20.478723 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:03:20.478607    2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="9bb04ac6-21f5-4467-93a8-1689b729ec93" containerName="kube-rbac-proxy"
Jul 28 16:03:20.478723 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:03:20.478614    2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="9155611b-3e3d-4c28-9d34-c5a27ffea78e" containerName="kube-rbac-proxy"
Jul 28 16:03:20.483403 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:03:20.483378    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-primary-9bf0ca-predictor-7969d99c69-qmc4g"
Jul 28 16:03:20.485898 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:03:20.485870    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 28 16:03:20.486037 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:03:20.485921    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-primary-9bf0ca-predictor-serving-cert\""
Jul 28 16:03:20.486037 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:03:20.485929    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 28 16:03:20.486037 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:03:20.485961    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 28 16:03:20.486350 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:03:20.486327    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 28 16:03:20.486903 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:03:20.486888    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-primary-9bf0ca-kube-rbac-proxy-sar-config\""
Jul 28 16:03:20.486983 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:03:20.486890    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-gcwwc\""
Jul 28 16:03:20.490402 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:03:20.490381    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-primary-9bf0ca-predictor-7969d99c69-qmc4g"]
Jul 28 16:03:20.584443 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:03:20.584402    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a535bf88-d6b8-4a74-a6cd-43c846495c56-kserve-provision-location\") pod \"isvc-primary-9bf0ca-predictor-7969d99c69-qmc4g\" (UID: \"a535bf88-d6b8-4a74-a6cd-43c846495c56\") " pod="kserve-ci-e2e-test/isvc-primary-9bf0ca-predictor-7969d99c69-qmc4g"
Jul 28 16:03:20.584629 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:03:20.584460    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a535bf88-d6b8-4a74-a6cd-43c846495c56-kube-rbac-proxy-sar-config\") pod \"isvc-primary-9bf0ca-predictor-7969d99c69-qmc4g\" (UID: \"a535bf88-d6b8-4a74-a6cd-43c846495c56\") " pod="kserve-ci-e2e-test/isvc-primary-9bf0ca-predictor-7969d99c69-qmc4g"
Jul 28 16:03:20.584629 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:03:20.584520    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a535bf88-d6b8-4a74-a6cd-43c846495c56-cabundle-cert\") pod \"isvc-primary-9bf0ca-predictor-7969d99c69-qmc4g\" (UID: \"a535bf88-d6b8-4a74-a6cd-43c846495c56\") " pod="kserve-ci-e2e-test/isvc-primary-9bf0ca-predictor-7969d99c69-qmc4g"
Jul 28 16:03:20.584629 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:03:20.584585    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a535bf88-d6b8-4a74-a6cd-43c846495c56-proxy-tls\") pod \"isvc-primary-9bf0ca-predictor-7969d99c69-qmc4g\" (UID: \"a535bf88-d6b8-4a74-a6cd-43c846495c56\") " pod="kserve-ci-e2e-test/isvc-primary-9bf0ca-predictor-7969d99c69-qmc4g"
Jul 28 16:03:20.584629 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:03:20.584607    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pv44g\" (UniqueName: \"kubernetes.io/projected/a535bf88-d6b8-4a74-a6cd-43c846495c56-kube-api-access-pv44g\") pod \"isvc-primary-9bf0ca-predictor-7969d99c69-qmc4g\" (UID: \"a535bf88-d6b8-4a74-a6cd-43c846495c56\") " pod="kserve-ci-e2e-test/isvc-primary-9bf0ca-predictor-7969d99c69-qmc4g"
Jul 28 16:03:20.642484 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:03:20.642452    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22nh9_ebf75c0d-1597-4a1c-b668-dc2a041eb35e/konnectivity-agent/0.log" limit=1024
Jul 28 16:03:20.685476 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:03:20.685384    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a535bf88-d6b8-4a74-a6cd-43c846495c56-kserve-provision-location\") pod \"isvc-primary-9bf0ca-predictor-7969d99c69-qmc4g\" (UID: \"a535bf88-d6b8-4a74-a6cd-43c846495c56\") " pod="kserve-ci-e2e-test/isvc-primary-9bf0ca-predictor-7969d99c69-qmc4g"
Jul 28 16:03:20.685476 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:03:20.685422    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a535bf88-d6b8-4a74-a6cd-43c846495c56-kube-rbac-proxy-sar-config\") pod \"isvc-primary-9bf0ca-predictor-7969d99c69-qmc4g\" (UID: \"a535bf88-d6b8-4a74-a6cd-43c846495c56\") " pod="kserve-ci-e2e-test/isvc-primary-9bf0ca-predictor-7969d99c69-qmc4g"
Jul 28 16:03:20.685476 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:03:20.685449    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a535bf88-d6b8-4a74-a6cd-43c846495c56-cabundle-cert\") pod \"isvc-primary-9bf0ca-predictor-7969d99c69-qmc4g\" (UID: \"a535bf88-d6b8-4a74-a6cd-43c846495c56\") " pod="kserve-ci-e2e-test/isvc-primary-9bf0ca-predictor-7969d99c69-qmc4g"
Jul 28 16:03:20.685763 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:03:20.685486    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a535bf88-d6b8-4a74-a6cd-43c846495c56-proxy-tls\") pod \"isvc-primary-9bf0ca-predictor-7969d99c69-qmc4g\" (UID: \"a535bf88-d6b8-4a74-a6cd-43c846495c56\") " pod="kserve-ci-e2e-test/isvc-primary-9bf0ca-predictor-7969d99c69-qmc4g"
Jul 28 16:03:20.685763 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:03:20.685509    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pv44g\" (UniqueName: \"kubernetes.io/projected/a535bf88-d6b8-4a74-a6cd-43c846495c56-kube-api-access-pv44g\") pod \"isvc-primary-9bf0ca-predictor-7969d99c69-qmc4g\" (UID: \"a535bf88-d6b8-4a74-a6cd-43c846495c56\") " pod="kserve-ci-e2e-test/isvc-primary-9bf0ca-predictor-7969d99c69-qmc4g"
Jul 28 16:03:20.685919 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:03:20.685817    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a535bf88-d6b8-4a74-a6cd-43c846495c56-kserve-provision-location\") pod \"isvc-primary-9bf0ca-predictor-7969d99c69-qmc4g\" (UID: \"a535bf88-d6b8-4a74-a6cd-43c846495c56\") " pod="kserve-ci-e2e-test/isvc-primary-9bf0ca-predictor-7969d99c69-qmc4g"
Jul 28 16:03:20.686186 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:03:20.686164    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a535bf88-d6b8-4a74-a6cd-43c846495c56-kube-rbac-proxy-sar-config\") pod \"isvc-primary-9bf0ca-predictor-7969d99c69-qmc4g\" (UID: \"a535bf88-d6b8-4a74-a6cd-43c846495c56\") " pod="kserve-ci-e2e-test/isvc-primary-9bf0ca-predictor-7969d99c69-qmc4g"
Jul 28 16:03:20.686269 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:03:20.686215    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a535bf88-d6b8-4a74-a6cd-43c846495c56-cabundle-cert\") pod \"isvc-primary-9bf0ca-predictor-7969d99c69-qmc4g\" (UID: \"a535bf88-d6b8-4a74-a6cd-43c846495c56\") " pod="kserve-ci-e2e-test/isvc-primary-9bf0ca-predictor-7969d99c69-qmc4g"
Jul 28 16:03:20.687971 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:03:20.687945    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a535bf88-d6b8-4a74-a6cd-43c846495c56-proxy-tls\") pod \"isvc-primary-9bf0ca-predictor-7969d99c69-qmc4g\" (UID: \"a535bf88-d6b8-4a74-a6cd-43c846495c56\") " pod="kserve-ci-e2e-test/isvc-primary-9bf0ca-predictor-7969d99c69-qmc4g"
Jul 28 16:03:20.694384 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:03:20.694359    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pv44g\" (UniqueName: \"kubernetes.io/projected/a535bf88-d6b8-4a74-a6cd-43c846495c56-kube-api-access-pv44g\") pod \"isvc-primary-9bf0ca-predictor-7969d99c69-qmc4g\" (UID: \"a535bf88-d6b8-4a74-a6cd-43c846495c56\") " pod="kserve-ci-e2e-test/isvc-primary-9bf0ca-predictor-7969d99c69-qmc4g"
Jul 28 16:03:20.794714 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:03:20.794676    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-primary-9bf0ca-predictor-7969d99c69-qmc4g"
Jul 28 16:03:20.934257 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:03:20.934232    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-primary-9bf0ca-predictor-7969d99c69-qmc4g"]
Jul 28 16:03:20.935906 ip-10-0-131-121 kubenswrapper[2568]: W0728 16:03:20.935835    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda535bf88_d6b8_4a74_a6cd_43c846495c56.slice/crio-0e213231b65ed4827165dcd89460db0e64de486129bf06f7d960666df9d035c8 WatchSource:0}: Error finding container 0e213231b65ed4827165dcd89460db0e64de486129bf06f7d960666df9d035c8: Status 404 returned error can't find the container with id 0e213231b65ed4827165dcd89460db0e64de486129bf06f7d960666df9d035c8
Jul 28 16:03:21.617815 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:03:21.617759    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-9bf0ca-predictor-7969d99c69-qmc4g" event={"ID":"a535bf88-d6b8-4a74-a6cd-43c846495c56","Type":"ContainerStarted","Data":"0055f7f769ee6f8a617fed9335783566883cc3aeee67968c22ee8f9925311343"}
Jul 28 16:03:21.618201 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:03:21.617855    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-9bf0ca-predictor-7969d99c69-qmc4g" event={"ID":"a535bf88-d6b8-4a74-a6cd-43c846495c56","Type":"ContainerStarted","Data":"0e213231b65ed4827165dcd89460db0e64de486129bf06f7d960666df9d035c8"}
Jul 28 16:03:25.637058 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:03:25.637027    2568 generic.go:356] "Generic (PLEG): container finished" podID="a535bf88-d6b8-4a74-a6cd-43c846495c56" containerID="0055f7f769ee6f8a617fed9335783566883cc3aeee67968c22ee8f9925311343" exitCode=0
Jul 28 16:03:25.637452 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:03:25.637106    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-9bf0ca-predictor-7969d99c69-qmc4g" event={"ID":"a535bf88-d6b8-4a74-a6cd-43c846495c56","Type":"ContainerDied","Data":"0055f7f769ee6f8a617fed9335783566883cc3aeee67968c22ee8f9925311343"}
Jul 28 16:03:26.643296 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:03:26.643263    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-9bf0ca-predictor-7969d99c69-qmc4g" event={"ID":"a535bf88-d6b8-4a74-a6cd-43c846495c56","Type":"ContainerStarted","Data":"866d0c4ed43aefc8a76bcbde95a8aa59de49e0d54fc91d3bb622cd81786800c2"}
Jul 28 16:03:26.643685 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:03:26.643323    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-9bf0ca-predictor-7969d99c69-qmc4g" event={"ID":"a535bf88-d6b8-4a74-a6cd-43c846495c56","Type":"ContainerStarted","Data":"44269ad03a4926b34140292798ca76af90784315ef14894cd98dbc876999f03c"}
Jul 28 16:03:26.643685 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:03:26.643577    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-primary-9bf0ca-predictor-7969d99c69-qmc4g"
Jul 28 16:03:26.643757 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:03:26.643718    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-primary-9bf0ca-predictor-7969d99c69-qmc4g"
Jul 28 16:03:26.644995 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:03:26.644973    2568 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-9bf0ca-predictor-7969d99c69-qmc4g" podUID="a535bf88-d6b8-4a74-a6cd-43c846495c56" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused"
Jul 28 16:03:26.662949 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:03:26.662890    2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-primary-9bf0ca-predictor-7969d99c69-qmc4g" podStartSLOduration=3.033327025 podStartE2EDuration="6.662872465s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.62954544s" isStatefulPod=false podCreationTimestamp="2026-07-28 16:03:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 16:03:26.661257427 +0000 UTC m=+1786.846452595" watchObservedRunningTime="2026-07-28 16:03:26.662872465 +0000 UTC m=+1786.848067635"
Jul 28 16:03:27.647857 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:03:27.647807    2568 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-9bf0ca-predictor-7969d99c69-qmc4g" podUID="a535bf88-d6b8-4a74-a6cd-43c846495c56" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused"
Jul 28 16:03:32.652066 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:03:32.651995    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-primary-9bf0ca-predictor-7969d99c69-qmc4g"
Jul 28 16:03:32.652490 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:03:32.652465    2568 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-9bf0ca-predictor-7969d99c69-qmc4g" podUID="a535bf88-d6b8-4a74-a6cd-43c846495c56" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused"
Jul 28 16:03:42.652770 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:03:42.652725    2568 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-9bf0ca-predictor-7969d99c69-qmc4g" podUID="a535bf88-d6b8-4a74-a6cd-43c846495c56" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused"
Jul 28 16:03:52.652757 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:03:52.652713    2568 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-9bf0ca-predictor-7969d99c69-qmc4g" podUID="a535bf88-d6b8-4a74-a6cd-43c846495c56" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused"
Jul 28 16:04:02.652517 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:04:02.652473    2568 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-9bf0ca-predictor-7969d99c69-qmc4g" podUID="a535bf88-d6b8-4a74-a6cd-43c846495c56" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused"
Jul 28 16:04:12.652735 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:04:12.652692    2568 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-9bf0ca-predictor-7969d99c69-qmc4g" podUID="a535bf88-d6b8-4a74-a6cd-43c846495c56" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused"
Jul 28 16:04:22.653330 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:04:22.653292    2568 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-9bf0ca-predictor-7969d99c69-qmc4g" podUID="a535bf88-d6b8-4a74-a6cd-43c846495c56" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused"
Jul 28 16:04:32.652428 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:04:32.652385    2568 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-9bf0ca-predictor-7969d99c69-qmc4g" podUID="a535bf88-d6b8-4a74-a6cd-43c846495c56" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused"
Jul 28 16:04:42.653469 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:04:42.653439    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-primary-9bf0ca-predictor-7969d99c69-qmc4g"
Jul 28 16:05:05.697046 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:05:05.696961    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22nh9_ebf75c0d-1597-4a1c-b668-dc2a041eb35e/konnectivity-agent/0.log" limit=1024
Jul 28 16:05:05.717751 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:05:05.717718    2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-9bf0ca-predictor-7bf7b679b6-9gpwj"]
Jul 28 16:05:05.721260 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:05:05.721237    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-secondary-9bf0ca-predictor-7bf7b679b6-9gpwj"
Jul 28 16:05:05.723614 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:05:05.723585    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"invalid-s3-sa-9bf0ca-dockercfg-gddc6\""
Jul 28 16:05:05.723764 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:05:05.723690    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-secondary-9bf0ca-predictor-serving-cert\""
Jul 28 16:05:05.723764 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:05:05.723712    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-secondary-9bf0ca-kube-rbac-proxy-sar-config\""
Jul 28 16:05:05.723884 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:05:05.723760    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"invalid-s3-secret-9bf0ca\""
Jul 28 16:05:05.730985 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:05:05.730961    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-9bf0ca-predictor-7bf7b679b6-9gpwj"]
Jul 28 16:05:05.750610 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:05:05.750570    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/dbc9e2a9-e2c3-4845-b8dd-9df66bdd96e7-proxy-tls\") pod \"isvc-secondary-9bf0ca-predictor-7bf7b679b6-9gpwj\" (UID: \"dbc9e2a9-e2c3-4845-b8dd-9df66bdd96e7\") " pod="kserve-ci-e2e-test/isvc-secondary-9bf0ca-predictor-7bf7b679b6-9gpwj"
Jul 28 16:05:05.750808 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:05:05.750667    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nb27t\" (UniqueName: \"kubernetes.io/projected/dbc9e2a9-e2c3-4845-b8dd-9df66bdd96e7-kube-api-access-nb27t\") pod \"isvc-secondary-9bf0ca-predictor-7bf7b679b6-9gpwj\" (UID: \"dbc9e2a9-e2c3-4845-b8dd-9df66bdd96e7\") " pod="kserve-ci-e2e-test/isvc-secondary-9bf0ca-predictor-7bf7b679b6-9gpwj"
Jul 28 16:05:05.750808 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:05:05.750693    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/dbc9e2a9-e2c3-4845-b8dd-9df66bdd96e7-cabundle-cert\") pod \"isvc-secondary-9bf0ca-predictor-7bf7b679b6-9gpwj\" (UID: \"dbc9e2a9-e2c3-4845-b8dd-9df66bdd96e7\") " pod="kserve-ci-e2e-test/isvc-secondary-9bf0ca-predictor-7bf7b679b6-9gpwj"
Jul 28 16:05:05.750808 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:05:05.750722    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/dbc9e2a9-e2c3-4845-b8dd-9df66bdd96e7-kserve-provision-location\") pod \"isvc-secondary-9bf0ca-predictor-7bf7b679b6-9gpwj\" (UID: \"dbc9e2a9-e2c3-4845-b8dd-9df66bdd96e7\") " pod="kserve-ci-e2e-test/isvc-secondary-9bf0ca-predictor-7bf7b679b6-9gpwj"
Jul 28 16:05:05.750808 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:05:05.750771    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/dbc9e2a9-e2c3-4845-b8dd-9df66bdd96e7-kube-rbac-proxy-sar-config\") pod \"isvc-secondary-9bf0ca-predictor-7bf7b679b6-9gpwj\" (UID: \"dbc9e2a9-e2c3-4845-b8dd-9df66bdd96e7\") " pod="kserve-ci-e2e-test/isvc-secondary-9bf0ca-predictor-7bf7b679b6-9gpwj"
Jul 28 16:05:05.784439 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:05:05.784411    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22nh9_ebf75c0d-1597-4a1c-b668-dc2a041eb35e/konnectivity-agent/0.log" limit=1024
Jul 28 16:05:05.851246 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:05:05.851205    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/dbc9e2a9-e2c3-4845-b8dd-9df66bdd96e7-proxy-tls\") pod \"isvc-secondary-9bf0ca-predictor-7bf7b679b6-9gpwj\" (UID: \"dbc9e2a9-e2c3-4845-b8dd-9df66bdd96e7\") " pod="kserve-ci-e2e-test/isvc-secondary-9bf0ca-predictor-7bf7b679b6-9gpwj"
Jul 28 16:05:05.851467 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:05:05.851288    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nb27t\" (UniqueName: \"kubernetes.io/projected/dbc9e2a9-e2c3-4845-b8dd-9df66bdd96e7-kube-api-access-nb27t\") pod \"isvc-secondary-9bf0ca-predictor-7bf7b679b6-9gpwj\" (UID: \"dbc9e2a9-e2c3-4845-b8dd-9df66bdd96e7\") " pod="kserve-ci-e2e-test/isvc-secondary-9bf0ca-predictor-7bf7b679b6-9gpwj"
Jul 28 16:05:05.851467 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:05:05.851322    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/dbc9e2a9-e2c3-4845-b8dd-9df66bdd96e7-cabundle-cert\") pod \"isvc-secondary-9bf0ca-predictor-7bf7b679b6-9gpwj\" (UID: \"dbc9e2a9-e2c3-4845-b8dd-9df66bdd96e7\") " pod="kserve-ci-e2e-test/isvc-secondary-9bf0ca-predictor-7bf7b679b6-9gpwj"
Jul 28 16:05:05.851467 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:05:05.851351    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/dbc9e2a9-e2c3-4845-b8dd-9df66bdd96e7-kserve-provision-location\") pod \"isvc-secondary-9bf0ca-predictor-7bf7b679b6-9gpwj\" (UID: \"dbc9e2a9-e2c3-4845-b8dd-9df66bdd96e7\") " pod="kserve-ci-e2e-test/isvc-secondary-9bf0ca-predictor-7bf7b679b6-9gpwj"
Jul 28 16:05:05.851467 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:05:05.851397    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/dbc9e2a9-e2c3-4845-b8dd-9df66bdd96e7-kube-rbac-proxy-sar-config\") pod \"isvc-secondary-9bf0ca-predictor-7bf7b679b6-9gpwj\" (UID: \"dbc9e2a9-e2c3-4845-b8dd-9df66bdd96e7\") " pod="kserve-ci-e2e-test/isvc-secondary-9bf0ca-predictor-7bf7b679b6-9gpwj"
Jul 28 16:05:05.851942 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:05:05.851919    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/dbc9e2a9-e2c3-4845-b8dd-9df66bdd96e7-kserve-provision-location\") pod \"isvc-secondary-9bf0ca-predictor-7bf7b679b6-9gpwj\" (UID: \"dbc9e2a9-e2c3-4845-b8dd-9df66bdd96e7\") " pod="kserve-ci-e2e-test/isvc-secondary-9bf0ca-predictor-7bf7b679b6-9gpwj"
Jul 28 16:05:05.852241 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:05:05.852218    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/dbc9e2a9-e2c3-4845-b8dd-9df66bdd96e7-kube-rbac-proxy-sar-config\") pod \"isvc-secondary-9bf0ca-predictor-7bf7b679b6-9gpwj\" (UID: \"dbc9e2a9-e2c3-4845-b8dd-9df66bdd96e7\") " pod="kserve-ci-e2e-test/isvc-secondary-9bf0ca-predictor-7bf7b679b6-9gpwj"
Jul 28 16:05:05.852289 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:05:05.852218    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/dbc9e2a9-e2c3-4845-b8dd-9df66bdd96e7-cabundle-cert\") pod \"isvc-secondary-9bf0ca-predictor-7bf7b679b6-9gpwj\" (UID: \"dbc9e2a9-e2c3-4845-b8dd-9df66bdd96e7\") " pod="kserve-ci-e2e-test/isvc-secondary-9bf0ca-predictor-7bf7b679b6-9gpwj"
Jul 28 16:05:05.853785 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:05:05.853757    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/dbc9e2a9-e2c3-4845-b8dd-9df66bdd96e7-proxy-tls\") pod \"isvc-secondary-9bf0ca-predictor-7bf7b679b6-9gpwj\" (UID: \"dbc9e2a9-e2c3-4845-b8dd-9df66bdd96e7\") " pod="kserve-ci-e2e-test/isvc-secondary-9bf0ca-predictor-7bf7b679b6-9gpwj"
Jul 28 16:05:05.861614 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:05:05.861591    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nb27t\" (UniqueName: \"kubernetes.io/projected/dbc9e2a9-e2c3-4845-b8dd-9df66bdd96e7-kube-api-access-nb27t\") pod \"isvc-secondary-9bf0ca-predictor-7bf7b679b6-9gpwj\" (UID: \"dbc9e2a9-e2c3-4845-b8dd-9df66bdd96e7\") " pod="kserve-ci-e2e-test/isvc-secondary-9bf0ca-predictor-7bf7b679b6-9gpwj"
Jul 28 16:05:06.031943 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:05:06.031909    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-secondary-9bf0ca-predictor-7bf7b679b6-9gpwj"
Jul 28 16:05:06.167999 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:05:06.167835    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-9bf0ca-predictor-7bf7b679b6-9gpwj"]
Jul 28 16:05:06.170718 ip-10-0-131-121 kubenswrapper[2568]: W0728 16:05:06.170690    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddbc9e2a9_e2c3_4845_b8dd_9df66bdd96e7.slice/crio-af6abd9cdf4bc80e408e333d231c27b0d215eb7fe0e7e0b95d31721d4525b251 WatchSource:0}: Error finding container af6abd9cdf4bc80e408e333d231c27b0d215eb7fe0e7e0b95d31721d4525b251: Status 404 returned error can't find the container with id af6abd9cdf4bc80e408e333d231c27b0d215eb7fe0e7e0b95d31721d4525b251
Jul 28 16:05:07.061217 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:05:07.061179    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-9bf0ca-predictor-7bf7b679b6-9gpwj" event={"ID":"dbc9e2a9-e2c3-4845-b8dd-9df66bdd96e7","Type":"ContainerStarted","Data":"f457b2e2ced265257b6c95db65b6977f699542cfe3ef0e7e799e7b5821c9ffea"}
Jul 28 16:05:07.061676 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:05:07.061275    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-9bf0ca-predictor-7bf7b679b6-9gpwj" event={"ID":"dbc9e2a9-e2c3-4845-b8dd-9df66bdd96e7","Type":"ContainerStarted","Data":"af6abd9cdf4bc80e408e333d231c27b0d215eb7fe0e7e0b95d31721d4525b251"}
Jul 28 16:05:11.079666 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:05:11.079639    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-9bf0ca-predictor-7bf7b679b6-9gpwj_dbc9e2a9-e2c3-4845-b8dd-9df66bdd96e7/storage-initializer/0.log"
Jul 28 16:05:11.080171 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:05:11.079675    2568 generic.go:356] "Generic (PLEG): container finished" podID="dbc9e2a9-e2c3-4845-b8dd-9df66bdd96e7" containerID="f457b2e2ced265257b6c95db65b6977f699542cfe3ef0e7e799e7b5821c9ffea" exitCode=1
Jul 28 16:05:11.080171 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:05:11.079762    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-9bf0ca-predictor-7bf7b679b6-9gpwj" event={"ID":"dbc9e2a9-e2c3-4845-b8dd-9df66bdd96e7","Type":"ContainerDied","Data":"f457b2e2ced265257b6c95db65b6977f699542cfe3ef0e7e799e7b5821c9ffea"}
Jul 28 16:05:12.089174 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:05:12.089146    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-9bf0ca-predictor-7bf7b679b6-9gpwj_dbc9e2a9-e2c3-4845-b8dd-9df66bdd96e7/storage-initializer/0.log"
Jul 28 16:05:12.089569 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:05:12.089259    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-9bf0ca-predictor-7bf7b679b6-9gpwj" event={"ID":"dbc9e2a9-e2c3-4845-b8dd-9df66bdd96e7","Type":"ContainerStarted","Data":"30ac46ab45c1d0395ebd832a87a3c34f220a2c10437c39848b5442a76840f1e3"}
Jul 28 16:05:15.102856 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:05:15.102827    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-9bf0ca-predictor-7bf7b679b6-9gpwj_dbc9e2a9-e2c3-4845-b8dd-9df66bdd96e7/storage-initializer/1.log"
Jul 28 16:05:15.103280 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:05:15.103244    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-9bf0ca-predictor-7bf7b679b6-9gpwj_dbc9e2a9-e2c3-4845-b8dd-9df66bdd96e7/storage-initializer/0.log"
Jul 28 16:05:15.103326 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:05:15.103276    2568 generic.go:356] "Generic (PLEG): container finished" podID="dbc9e2a9-e2c3-4845-b8dd-9df66bdd96e7" containerID="30ac46ab45c1d0395ebd832a87a3c34f220a2c10437c39848b5442a76840f1e3" exitCode=1
Jul 28 16:05:15.103361 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:05:15.103341    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-9bf0ca-predictor-7bf7b679b6-9gpwj" event={"ID":"dbc9e2a9-e2c3-4845-b8dd-9df66bdd96e7","Type":"ContainerDied","Data":"30ac46ab45c1d0395ebd832a87a3c34f220a2c10437c39848b5442a76840f1e3"}
Jul 28 16:05:15.103409 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:05:15.103396    2568 scope.go:117] "RemoveContainer" containerID="f457b2e2ced265257b6c95db65b6977f699542cfe3ef0e7e799e7b5821c9ffea"
Jul 28 16:05:15.103751 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:05:15.103727    2568 scope.go:117] "RemoveContainer" containerID="f457b2e2ced265257b6c95db65b6977f699542cfe3ef0e7e799e7b5821c9ffea"
Jul 28 16:05:15.114307 ip-10-0-131-121 kubenswrapper[2568]: E0728 16:05:15.114280    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"storage-initializer\" with CrashLoopBackOff: \"back-off 10s restarting failed container=storage-initializer pod=isvc-secondary-9bf0ca-predictor-7bf7b679b6-9gpwj_kserve-ci-e2e-test(dbc9e2a9-e2c3-4845-b8dd-9df66bdd96e7)\"" pod="kserve-ci-e2e-test/isvc-secondary-9bf0ca-predictor-7bf7b679b6-9gpwj" podUID="dbc9e2a9-e2c3-4845-b8dd-9df66bdd96e7"
Jul 28 16:05:15.114667 ip-10-0-131-121 kubenswrapper[2568]: E0728 16:05:15.114642    2568 log.go:32] "RemoveContainer from runtime service failed" err="rpc error: code = Unknown desc = failed to delete container k8s_storage-initializer_isvc-secondary-9bf0ca-predictor-7bf7b679b6-9gpwj_kserve-ci-e2e-test_dbc9e2a9-e2c3-4845-b8dd-9df66bdd96e7_0 in pod sandbox af6abd9cdf4bc80e408e333d231c27b0d215eb7fe0e7e0b95d31721d4525b251 from index: no such id: 'f457b2e2ced265257b6c95db65b6977f699542cfe3ef0e7e799e7b5821c9ffea'" containerID="f457b2e2ced265257b6c95db65b6977f699542cfe3ef0e7e799e7b5821c9ffea"
Jul 28 16:05:15.114721 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:05:15.114679    2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f457b2e2ced265257b6c95db65b6977f699542cfe3ef0e7e799e7b5821c9ffea"} err="rpc error: code = Unknown desc = failed to delete container k8s_storage-initializer_isvc-secondary-9bf0ca-predictor-7bf7b679b6-9gpwj_kserve-ci-e2e-test_dbc9e2a9-e2c3-4845-b8dd-9df66bdd96e7_0 in pod sandbox af6abd9cdf4bc80e408e333d231c27b0d215eb7fe0e7e0b95d31721d4525b251 from index: no such id: 'f457b2e2ced265257b6c95db65b6977f699542cfe3ef0e7e799e7b5821c9ffea'"
Jul 28 16:05:16.109604 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:05:16.109573    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-9bf0ca-predictor-7bf7b679b6-9gpwj_dbc9e2a9-e2c3-4845-b8dd-9df66bdd96e7/storage-initializer/1.log"
Jul 28 16:05:21.774609 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:05:21.774575    2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-9bf0ca-predictor-7bf7b679b6-9gpwj"]
Jul 28 16:05:21.799197 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:05:21.799162    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22nh9_ebf75c0d-1597-4a1c-b668-dc2a041eb35e/konnectivity-agent/0.log" limit=1024
Jul 28 16:05:21.834580 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:05:21.834544    2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-primary-9bf0ca-predictor-7969d99c69-qmc4g"]
Jul 28 16:05:21.835215 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:05:21.835175    2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-primary-9bf0ca-predictor-7969d99c69-qmc4g" podUID="a535bf88-d6b8-4a74-a6cd-43c846495c56" containerName="kube-rbac-proxy" containerID="cri-o://866d0c4ed43aefc8a76bcbde95a8aa59de49e0d54fc91d3bb622cd81786800c2" gracePeriod=30
Jul 28 16:05:21.835419 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:05:21.835392    2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-primary-9bf0ca-predictor-7969d99c69-qmc4g" podUID="a535bf88-d6b8-4a74-a6cd-43c846495c56" containerName="kserve-container" containerID="cri-o://44269ad03a4926b34140292798ca76af90784315ef14894cd98dbc876999f03c" gracePeriod=30
Jul 28 16:05:21.862679 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:05:21.862644    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22nh9_ebf75c0d-1597-4a1c-b668-dc2a041eb35e/konnectivity-agent/0.log" limit=1024
Jul 28 16:05:21.956884 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:05:21.956862    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-9bf0ca-predictor-7bf7b679b6-9gpwj_dbc9e2a9-e2c3-4845-b8dd-9df66bdd96e7/storage-initializer/1.log"
Jul 28 16:05:21.958965 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:05:21.958950    2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-secondary-9bf0ca-predictor-7bf7b679b6-9gpwj"
Jul 28 16:05:21.978505 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:05:21.978477    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/dbc9e2a9-e2c3-4845-b8dd-9df66bdd96e7-cabundle-cert\") pod \"dbc9e2a9-e2c3-4845-b8dd-9df66bdd96e7\" (UID: \"dbc9e2a9-e2c3-4845-b8dd-9df66bdd96e7\") "
Jul 28 16:05:21.978505 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:05:21.978509    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/dbc9e2a9-e2c3-4845-b8dd-9df66bdd96e7-kserve-provision-location\") pod \"dbc9e2a9-e2c3-4845-b8dd-9df66bdd96e7\" (UID: \"dbc9e2a9-e2c3-4845-b8dd-9df66bdd96e7\") "
Jul 28 16:05:21.978765 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:05:21.978544    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nb27t\" (UniqueName: \"kubernetes.io/projected/dbc9e2a9-e2c3-4845-b8dd-9df66bdd96e7-kube-api-access-nb27t\") pod \"dbc9e2a9-e2c3-4845-b8dd-9df66bdd96e7\" (UID: \"dbc9e2a9-e2c3-4845-b8dd-9df66bdd96e7\") "
Jul 28 16:05:21.978765 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:05:21.978572    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/dbc9e2a9-e2c3-4845-b8dd-9df66bdd96e7-kube-rbac-proxy-sar-config\") pod \"dbc9e2a9-e2c3-4845-b8dd-9df66bdd96e7\" (UID: \"dbc9e2a9-e2c3-4845-b8dd-9df66bdd96e7\") "
Jul 28 16:05:21.978765 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:05:21.978595    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/dbc9e2a9-e2c3-4845-b8dd-9df66bdd96e7-proxy-tls\") pod \"dbc9e2a9-e2c3-4845-b8dd-9df66bdd96e7\" (UID: \"dbc9e2a9-e2c3-4845-b8dd-9df66bdd96e7\") "
Jul 28 16:05:21.978980 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:05:21.978843    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dbc9e2a9-e2c3-4845-b8dd-9df66bdd96e7-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "dbc9e2a9-e2c3-4845-b8dd-9df66bdd96e7" (UID: "dbc9e2a9-e2c3-4845-b8dd-9df66bdd96e7"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 16:05:21.979029 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:05:21.978969    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dbc9e2a9-e2c3-4845-b8dd-9df66bdd96e7-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "dbc9e2a9-e2c3-4845-b8dd-9df66bdd96e7" (UID: "dbc9e2a9-e2c3-4845-b8dd-9df66bdd96e7"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 16:05:21.979071 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:05:21.979028    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dbc9e2a9-e2c3-4845-b8dd-9df66bdd96e7-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "dbc9e2a9-e2c3-4845-b8dd-9df66bdd96e7" (UID: "dbc9e2a9-e2c3-4845-b8dd-9df66bdd96e7"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 16:05:21.980717 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:05:21.980696    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dbc9e2a9-e2c3-4845-b8dd-9df66bdd96e7-kube-api-access-nb27t" (OuterVolumeSpecName: "kube-api-access-nb27t") pod "dbc9e2a9-e2c3-4845-b8dd-9df66bdd96e7" (UID: "dbc9e2a9-e2c3-4845-b8dd-9df66bdd96e7"). InnerVolumeSpecName "kube-api-access-nb27t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 16:05:21.980880 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:05:21.980764    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dbc9e2a9-e2c3-4845-b8dd-9df66bdd96e7-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "dbc9e2a9-e2c3-4845-b8dd-9df66bdd96e7" (UID: "dbc9e2a9-e2c3-4845-b8dd-9df66bdd96e7"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 16:05:22.079421 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:05:22.079326    2568 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/dbc9e2a9-e2c3-4845-b8dd-9df66bdd96e7-cabundle-cert\") on node \"ip-10-0-131-121.ec2.internal\" DevicePath \"\""
Jul 28 16:05:22.079421 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:05:22.079358    2568 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/dbc9e2a9-e2c3-4845-b8dd-9df66bdd96e7-kserve-provision-location\") on node \"ip-10-0-131-121.ec2.internal\" DevicePath \"\""
Jul 28 16:05:22.079421 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:05:22.079368    2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nb27t\" (UniqueName: \"kubernetes.io/projected/dbc9e2a9-e2c3-4845-b8dd-9df66bdd96e7-kube-api-access-nb27t\") on node \"ip-10-0-131-121.ec2.internal\" DevicePath \"\""
Jul 28 16:05:22.079421 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:05:22.079379    2568 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/dbc9e2a9-e2c3-4845-b8dd-9df66bdd96e7-kube-rbac-proxy-sar-config\") on node \"ip-10-0-131-121.ec2.internal\" DevicePath \"\""
Jul 28 16:05:22.079421 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:05:22.079388    2568 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/dbc9e2a9-e2c3-4845-b8dd-9df66bdd96e7-proxy-tls\") on node \"ip-10-0-131-121.ec2.internal\" DevicePath \"\""
Jul 28 16:05:22.138190 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:05:22.138152    2568 generic.go:356] "Generic (PLEG): container finished" podID="a535bf88-d6b8-4a74-a6cd-43c846495c56" containerID="866d0c4ed43aefc8a76bcbde95a8aa59de49e0d54fc91d3bb622cd81786800c2" exitCode=2
Jul 28 16:05:22.138375 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:05:22.138222    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-9bf0ca-predictor-7969d99c69-qmc4g" event={"ID":"a535bf88-d6b8-4a74-a6cd-43c846495c56","Type":"ContainerDied","Data":"866d0c4ed43aefc8a76bcbde95a8aa59de49e0d54fc91d3bb622cd81786800c2"}
Jul 28 16:05:22.139525 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:05:22.139508    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-9bf0ca-predictor-7bf7b679b6-9gpwj_dbc9e2a9-e2c3-4845-b8dd-9df66bdd96e7/storage-initializer/1.log"
Jul 28 16:05:22.139583 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:05:22.139569    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-9bf0ca-predictor-7bf7b679b6-9gpwj" event={"ID":"dbc9e2a9-e2c3-4845-b8dd-9df66bdd96e7","Type":"ContainerDied","Data":"af6abd9cdf4bc80e408e333d231c27b0d215eb7fe0e7e0b95d31721d4525b251"}
Jul 28 16:05:22.139632 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:05:22.139601    2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-secondary-9bf0ca-predictor-7bf7b679b6-9gpwj"
Jul 28 16:05:22.139632 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:05:22.139614    2568 scope.go:117] "RemoveContainer" containerID="30ac46ab45c1d0395ebd832a87a3c34f220a2c10437c39848b5442a76840f1e3"
Jul 28 16:05:22.178897 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:05:22.178862    2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-9bf0ca-predictor-7bf7b679b6-9gpwj"]
Jul 28 16:05:22.183819 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:05:22.183768    2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-9bf0ca-predictor-7bf7b679b6-9gpwj"]
Jul 28 16:05:22.295157 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:05:22.295126    2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="dbc9e2a9-e2c3-4845-b8dd-9df66bdd96e7" path="/var/lib/kubelet/pods/dbc9e2a9-e2c3-4845-b8dd-9df66bdd96e7/volumes"
Jul 28 16:05:22.648861 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:05:22.648811    2568 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-9bf0ca-predictor-7969d99c69-qmc4g" podUID="a535bf88-d6b8-4a74-a6cd-43c846495c56" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.34:8643/healthz\": dial tcp 10.134.0.34:8643: connect: connection refused"
Jul 28 16:05:22.653052 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:05:22.653021    2568 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-9bf0ca-predictor-7969d99c69-qmc4g" podUID="a535bf88-d6b8-4a74-a6cd-43c846495c56" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused"
Jul 28 16:05:26.974528 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:05:26.974487    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22nh9_ebf75c0d-1597-4a1c-b668-dc2a041eb35e/konnectivity-agent/0.log" limit=1024
Jul 28 16:05:26.999942 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:05:26.999911    2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-primary-9bf0ca-predictor-7969d99c69-qmc4g"
Jul 28 16:05:27.001617 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:05:27.001579    2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-6a365a-predictor-5467b89cf6-8hvpg"]
Jul 28 16:05:27.001961 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:05:27.001947    2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a535bf88-d6b8-4a74-a6cd-43c846495c56" containerName="storage-initializer"
Jul 28 16:05:27.002016 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:05:27.001962    2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="a535bf88-d6b8-4a74-a6cd-43c846495c56" containerName="storage-initializer"
Jul 28 16:05:27.002016 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:05:27.001977    2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a535bf88-d6b8-4a74-a6cd-43c846495c56" containerName="kserve-container"
Jul 28 16:05:27.002016 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:05:27.001983    2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="a535bf88-d6b8-4a74-a6cd-43c846495c56" containerName="kserve-container"
Jul 28 16:05:27.002016 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:05:27.001989    2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dbc9e2a9-e2c3-4845-b8dd-9df66bdd96e7" containerName="storage-initializer"
Jul 28 16:05:27.002016 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:05:27.001995    2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="dbc9e2a9-e2c3-4845-b8dd-9df66bdd96e7" containerName="storage-initializer"
Jul 28 16:05:27.002016 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:05:27.002002    2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a535bf88-d6b8-4a74-a6cd-43c846495c56" containerName="kube-rbac-proxy"
Jul 28 16:05:27.002016 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:05:27.002007    2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="a535bf88-d6b8-4a74-a6cd-43c846495c56" containerName="kube-rbac-proxy"
Jul 28 16:05:27.002225 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:05:27.002063    2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="a535bf88-d6b8-4a74-a6cd-43c846495c56" containerName="kserve-container"
Jul 28 16:05:27.002225 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:05:27.002071    2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="dbc9e2a9-e2c3-4845-b8dd-9df66bdd96e7" containerName="storage-initializer"
Jul 28 16:05:27.002225 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:05:27.002079    2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="dbc9e2a9-e2c3-4845-b8dd-9df66bdd96e7" containerName="storage-initializer"
Jul 28 16:05:27.002225 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:05:27.002085    2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="a535bf88-d6b8-4a74-a6cd-43c846495c56" containerName="kube-rbac-proxy"
Jul 28 16:05:27.002225 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:05:27.002146    2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dbc9e2a9-e2c3-4845-b8dd-9df66bdd96e7" containerName="storage-initializer"
Jul 28 16:05:27.002225 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:05:27.002151    2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="dbc9e2a9-e2c3-4845-b8dd-9df66bdd96e7" containerName="storage-initializer"
Jul 28 16:05:27.005153 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:05:27.005131    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-init-fail-6a365a-predictor-5467b89cf6-8hvpg"
Jul 28 16:05:27.007384 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:05:27.007361    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"fail-s3-secret-6a365a\""
Jul 28 16:05:27.007524 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:05:27.007436    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-init-fail-6a365a-kube-rbac-proxy-sar-config\""
Jul 28 16:05:27.007524 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:05:27.007435    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-init-fail-6a365a-predictor-serving-cert\""
Jul 28 16:05:27.007699 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:05:27.007684    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"fail-s3-sa-6a365a-dockercfg-lj8tt\""
Jul 28 16:05:27.016563 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:05:27.016535    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-6a365a-predictor-5467b89cf6-8hvpg"]
Jul 28 16:05:27.122072 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:05:27.121974    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a535bf88-d6b8-4a74-a6cd-43c846495c56-proxy-tls\") pod \"a535bf88-d6b8-4a74-a6cd-43c846495c56\" (UID: \"a535bf88-d6b8-4a74-a6cd-43c846495c56\") "
Jul 28 16:05:27.122072 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:05:27.122030    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a535bf88-d6b8-4a74-a6cd-43c846495c56-kserve-provision-location\") pod \"a535bf88-d6b8-4a74-a6cd-43c846495c56\" (UID: \"a535bf88-d6b8-4a74-a6cd-43c846495c56\") "
Jul 28 16:05:27.122072 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:05:27.122062    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pv44g\" (UniqueName: \"kubernetes.io/projected/a535bf88-d6b8-4a74-a6cd-43c846495c56-kube-api-access-pv44g\") pod \"a535bf88-d6b8-4a74-a6cd-43c846495c56\" (UID: \"a535bf88-d6b8-4a74-a6cd-43c846495c56\") "
Jul 28 16:05:27.122366 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:05:27.122152    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a535bf88-d6b8-4a74-a6cd-43c846495c56-kube-rbac-proxy-sar-config\") pod \"a535bf88-d6b8-4a74-a6cd-43c846495c56\" (UID: \"a535bf88-d6b8-4a74-a6cd-43c846495c56\") "
Jul 28 16:05:27.122366 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:05:27.122172    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a535bf88-d6b8-4a74-a6cd-43c846495c56-cabundle-cert\") pod \"a535bf88-d6b8-4a74-a6cd-43c846495c56\" (UID: \"a535bf88-d6b8-4a74-a6cd-43c846495c56\") "
Jul 28 16:05:27.122366 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:05:27.122262    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e7997e4a-f9a6-44da-a217-fe19d9c42971-cabundle-cert\") pod \"isvc-init-fail-6a365a-predictor-5467b89cf6-8hvpg\" (UID: \"e7997e4a-f9a6-44da-a217-fe19d9c42971\") " pod="kserve-ci-e2e-test/isvc-init-fail-6a365a-predictor-5467b89cf6-8hvpg"
Jul 28 16:05:27.122366 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:05:27.122316    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e7997e4a-f9a6-44da-a217-fe19d9c42971-kube-rbac-proxy-sar-config\") pod \"isvc-init-fail-6a365a-predictor-5467b89cf6-8hvpg\" (UID: \"e7997e4a-f9a6-44da-a217-fe19d9c42971\") " pod="kserve-ci-e2e-test/isvc-init-fail-6a365a-predictor-5467b89cf6-8hvpg"
Jul 28 16:05:27.122366 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:05:27.122349    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e7997e4a-f9a6-44da-a217-fe19d9c42971-proxy-tls\") pod \"isvc-init-fail-6a365a-predictor-5467b89cf6-8hvpg\" (UID: \"e7997e4a-f9a6-44da-a217-fe19d9c42971\") " pod="kserve-ci-e2e-test/isvc-init-fail-6a365a-predictor-5467b89cf6-8hvpg"
Jul 28 16:05:27.122614 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:05:27.122417    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e7997e4a-f9a6-44da-a217-fe19d9c42971-kserve-provision-location\") pod \"isvc-init-fail-6a365a-predictor-5467b89cf6-8hvpg\" (UID: \"e7997e4a-f9a6-44da-a217-fe19d9c42971\") " pod="kserve-ci-e2e-test/isvc-init-fail-6a365a-predictor-5467b89cf6-8hvpg"
Jul 28 16:05:27.122614 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:05:27.122454    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xwb6n\" (UniqueName: \"kubernetes.io/projected/e7997e4a-f9a6-44da-a217-fe19d9c42971-kube-api-access-xwb6n\") pod \"isvc-init-fail-6a365a-predictor-5467b89cf6-8hvpg\" (UID: \"e7997e4a-f9a6-44da-a217-fe19d9c42971\") " pod="kserve-ci-e2e-test/isvc-init-fail-6a365a-predictor-5467b89cf6-8hvpg"
Jul 28 16:05:27.122614 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:05:27.122484    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a535bf88-d6b8-4a74-a6cd-43c846495c56-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "a535bf88-d6b8-4a74-a6cd-43c846495c56" (UID: "a535bf88-d6b8-4a74-a6cd-43c846495c56"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 16:05:27.122614 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:05:27.122549    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a535bf88-d6b8-4a74-a6cd-43c846495c56-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "a535bf88-d6b8-4a74-a6cd-43c846495c56" (UID: "a535bf88-d6b8-4a74-a6cd-43c846495c56"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 16:05:27.122746 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:05:27.122618    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a535bf88-d6b8-4a74-a6cd-43c846495c56-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "a535bf88-d6b8-4a74-a6cd-43c846495c56" (UID: "a535bf88-d6b8-4a74-a6cd-43c846495c56"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 16:05:27.124340 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:05:27.124316    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a535bf88-d6b8-4a74-a6cd-43c846495c56-kube-api-access-pv44g" (OuterVolumeSpecName: "kube-api-access-pv44g") pod "a535bf88-d6b8-4a74-a6cd-43c846495c56" (UID: "a535bf88-d6b8-4a74-a6cd-43c846495c56"). InnerVolumeSpecName "kube-api-access-pv44g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 16:05:27.124449 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:05:27.124345    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a535bf88-d6b8-4a74-a6cd-43c846495c56-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "a535bf88-d6b8-4a74-a6cd-43c846495c56" (UID: "a535bf88-d6b8-4a74-a6cd-43c846495c56"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 16:05:27.166251 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:05:27.166217    2568 generic.go:356] "Generic (PLEG): container finished" podID="a535bf88-d6b8-4a74-a6cd-43c846495c56" containerID="44269ad03a4926b34140292798ca76af90784315ef14894cd98dbc876999f03c" exitCode=0
Jul 28 16:05:27.166410 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:05:27.166265    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-9bf0ca-predictor-7969d99c69-qmc4g" event={"ID":"a535bf88-d6b8-4a74-a6cd-43c846495c56","Type":"ContainerDied","Data":"44269ad03a4926b34140292798ca76af90784315ef14894cd98dbc876999f03c"}
Jul 28 16:05:27.166410 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:05:27.166292    2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-primary-9bf0ca-predictor-7969d99c69-qmc4g"
Jul 28 16:05:27.166410 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:05:27.166334    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-9bf0ca-predictor-7969d99c69-qmc4g" event={"ID":"a535bf88-d6b8-4a74-a6cd-43c846495c56","Type":"ContainerDied","Data":"0e213231b65ed4827165dcd89460db0e64de486129bf06f7d960666df9d035c8"}
Jul 28 16:05:27.166410 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:05:27.166360    2568 scope.go:117] "RemoveContainer" containerID="866d0c4ed43aefc8a76bcbde95a8aa59de49e0d54fc91d3bb622cd81786800c2"
Jul 28 16:05:27.176737 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:05:27.176541    2568 scope.go:117] "RemoveContainer" containerID="44269ad03a4926b34140292798ca76af90784315ef14894cd98dbc876999f03c"
Jul 28 16:05:27.185732 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:05:27.185710    2568 scope.go:117] "RemoveContainer" containerID="0055f7f769ee6f8a617fed9335783566883cc3aeee67968c22ee8f9925311343"
Jul 28 16:05:27.191862 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:05:27.191832    2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-primary-9bf0ca-predictor-7969d99c69-qmc4g"]
Jul 28 16:05:27.194023 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:05:27.194007    2568 scope.go:117] "RemoveContainer" containerID="866d0c4ed43aefc8a76bcbde95a8aa59de49e0d54fc91d3bb622cd81786800c2"
Jul 28 16:05:27.194342 ip-10-0-131-121 kubenswrapper[2568]: E0728 16:05:27.194323    2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"866d0c4ed43aefc8a76bcbde95a8aa59de49e0d54fc91d3bb622cd81786800c2\": container with ID starting with 866d0c4ed43aefc8a76bcbde95a8aa59de49e0d54fc91d3bb622cd81786800c2 not found: ID does not exist" containerID="866d0c4ed43aefc8a76bcbde95a8aa59de49e0d54fc91d3bb622cd81786800c2"
Jul 28 16:05:27.194391 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:05:27.194353    2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"866d0c4ed43aefc8a76bcbde95a8aa59de49e0d54fc91d3bb622cd81786800c2"} err="failed to get container status \"866d0c4ed43aefc8a76bcbde95a8aa59de49e0d54fc91d3bb622cd81786800c2\": rpc error: code = NotFound desc = could not find container \"866d0c4ed43aefc8a76bcbde95a8aa59de49e0d54fc91d3bb622cd81786800c2\": container with ID starting with 866d0c4ed43aefc8a76bcbde95a8aa59de49e0d54fc91d3bb622cd81786800c2 not found: ID does not exist"
Jul 28 16:05:27.194391 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:05:27.194371    2568 scope.go:117] "RemoveContainer" containerID="44269ad03a4926b34140292798ca76af90784315ef14894cd98dbc876999f03c"
Jul 28 16:05:27.194602 ip-10-0-131-121 kubenswrapper[2568]: E0728 16:05:27.194583    2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"44269ad03a4926b34140292798ca76af90784315ef14894cd98dbc876999f03c\": container with ID starting with 44269ad03a4926b34140292798ca76af90784315ef14894cd98dbc876999f03c not found: ID does not exist" containerID="44269ad03a4926b34140292798ca76af90784315ef14894cd98dbc876999f03c"
Jul 28 16:05:27.194671 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:05:27.194613    2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"44269ad03a4926b34140292798ca76af90784315ef14894cd98dbc876999f03c"} err="failed to get container status \"44269ad03a4926b34140292798ca76af90784315ef14894cd98dbc876999f03c\": rpc error: code = NotFound desc = could not find container \"44269ad03a4926b34140292798ca76af90784315ef14894cd98dbc876999f03c\": container with ID starting with 44269ad03a4926b34140292798ca76af90784315ef14894cd98dbc876999f03c not found: ID does not exist"
Jul 28 16:05:27.194671 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:05:27.194637    2568 scope.go:117] "RemoveContainer" containerID="0055f7f769ee6f8a617fed9335783566883cc3aeee67968c22ee8f9925311343"
Jul 28 16:05:27.194935 ip-10-0-131-121 kubenswrapper[2568]: E0728 16:05:27.194916    2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0055f7f769ee6f8a617fed9335783566883cc3aeee67968c22ee8f9925311343\": container with ID starting with 0055f7f769ee6f8a617fed9335783566883cc3aeee67968c22ee8f9925311343 not found: ID does not exist" containerID="0055f7f769ee6f8a617fed9335783566883cc3aeee67968c22ee8f9925311343"
Jul 28 16:05:27.195001 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:05:27.194941    2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0055f7f769ee6f8a617fed9335783566883cc3aeee67968c22ee8f9925311343"} err="failed to get container status \"0055f7f769ee6f8a617fed9335783566883cc3aeee67968c22ee8f9925311343\": rpc error: code = NotFound desc = could not find container \"0055f7f769ee6f8a617fed9335783566883cc3aeee67968c22ee8f9925311343\": container with ID starting with 0055f7f769ee6f8a617fed9335783566883cc3aeee67968c22ee8f9925311343 not found: ID does not exist"
Jul 28 16:05:27.200843 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:05:27.200818    2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-primary-9bf0ca-predictor-7969d99c69-qmc4g"]
Jul 28 16:05:27.223389 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:05:27.223343    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e7997e4a-f9a6-44da-a217-fe19d9c42971-kube-rbac-proxy-sar-config\") pod \"isvc-init-fail-6a365a-predictor-5467b89cf6-8hvpg\" (UID: \"e7997e4a-f9a6-44da-a217-fe19d9c42971\") " pod="kserve-ci-e2e-test/isvc-init-fail-6a365a-predictor-5467b89cf6-8hvpg"
Jul 28 16:05:27.223389 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:05:27.223389    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e7997e4a-f9a6-44da-a217-fe19d9c42971-proxy-tls\") pod \"isvc-init-fail-6a365a-predictor-5467b89cf6-8hvpg\" (UID: \"e7997e4a-f9a6-44da-a217-fe19d9c42971\") " pod="kserve-ci-e2e-test/isvc-init-fail-6a365a-predictor-5467b89cf6-8hvpg"
Jul 28 16:05:27.223647 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:05:27.223418    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e7997e4a-f9a6-44da-a217-fe19d9c42971-kserve-provision-location\") pod \"isvc-init-fail-6a365a-predictor-5467b89cf6-8hvpg\" (UID: \"e7997e4a-f9a6-44da-a217-fe19d9c42971\") " pod="kserve-ci-e2e-test/isvc-init-fail-6a365a-predictor-5467b89cf6-8hvpg"
Jul 28 16:05:27.223647 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:05:27.223439    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xwb6n\" (UniqueName: \"kubernetes.io/projected/e7997e4a-f9a6-44da-a217-fe19d9c42971-kube-api-access-xwb6n\") pod \"isvc-init-fail-6a365a-predictor-5467b89cf6-8hvpg\" (UID: \"e7997e4a-f9a6-44da-a217-fe19d9c42971\") " pod="kserve-ci-e2e-test/isvc-init-fail-6a365a-predictor-5467b89cf6-8hvpg"
Jul 28 16:05:27.223647 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:05:27.223487    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e7997e4a-f9a6-44da-a217-fe19d9c42971-cabundle-cert\") pod \"isvc-init-fail-6a365a-predictor-5467b89cf6-8hvpg\" (UID: \"e7997e4a-f9a6-44da-a217-fe19d9c42971\") " pod="kserve-ci-e2e-test/isvc-init-fail-6a365a-predictor-5467b89cf6-8hvpg"
Jul 28 16:05:27.223647 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:05:27.223528    2568 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a535bf88-d6b8-4a74-a6cd-43c846495c56-proxy-tls\") on node \"ip-10-0-131-121.ec2.internal\" DevicePath \"\""
Jul 28 16:05:27.223647 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:05:27.223543    2568 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a535bf88-d6b8-4a74-a6cd-43c846495c56-kserve-provision-location\") on node \"ip-10-0-131-121.ec2.internal\" DevicePath \"\""
Jul 28 16:05:27.223647 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:05:27.223559    2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pv44g\" (UniqueName: \"kubernetes.io/projected/a535bf88-d6b8-4a74-a6cd-43c846495c56-kube-api-access-pv44g\") on node \"ip-10-0-131-121.ec2.internal\" DevicePath \"\""
Jul 28 16:05:27.223647 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:05:27.223569    2568 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a535bf88-d6b8-4a74-a6cd-43c846495c56-kube-rbac-proxy-sar-config\") on node \"ip-10-0-131-121.ec2.internal\" DevicePath \"\""
Jul 28 16:05:27.223647 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:05:27.223584    2568 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a535bf88-d6b8-4a74-a6cd-43c846495c56-cabundle-cert\") on node \"ip-10-0-131-121.ec2.internal\" DevicePath \"\""
Jul 28 16:05:27.223949 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:05:27.223928    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e7997e4a-f9a6-44da-a217-fe19d9c42971-kserve-provision-location\") pod \"isvc-init-fail-6a365a-predictor-5467b89cf6-8hvpg\" (UID: \"e7997e4a-f9a6-44da-a217-fe19d9c42971\") " pod="kserve-ci-e2e-test/isvc-init-fail-6a365a-predictor-5467b89cf6-8hvpg"
Jul 28 16:05:27.224088 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:05:27.224070    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e7997e4a-f9a6-44da-a217-fe19d9c42971-kube-rbac-proxy-sar-config\") pod \"isvc-init-fail-6a365a-predictor-5467b89cf6-8hvpg\" (UID: \"e7997e4a-f9a6-44da-a217-fe19d9c42971\") " pod="kserve-ci-e2e-test/isvc-init-fail-6a365a-predictor-5467b89cf6-8hvpg"
Jul 28 16:05:27.224174 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:05:27.224158    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e7997e4a-f9a6-44da-a217-fe19d9c42971-cabundle-cert\") pod \"isvc-init-fail-6a365a-predictor-5467b89cf6-8hvpg\" (UID: \"e7997e4a-f9a6-44da-a217-fe19d9c42971\") " pod="kserve-ci-e2e-test/isvc-init-fail-6a365a-predictor-5467b89cf6-8hvpg"
Jul 28 16:05:27.225938 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:05:27.225922    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e7997e4a-f9a6-44da-a217-fe19d9c42971-proxy-tls\") pod \"isvc-init-fail-6a365a-predictor-5467b89cf6-8hvpg\" (UID: \"e7997e4a-f9a6-44da-a217-fe19d9c42971\") " pod="kserve-ci-e2e-test/isvc-init-fail-6a365a-predictor-5467b89cf6-8hvpg"
Jul 28 16:05:27.233535 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:05:27.233506    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xwb6n\" (UniqueName: \"kubernetes.io/projected/e7997e4a-f9a6-44da-a217-fe19d9c42971-kube-api-access-xwb6n\") pod \"isvc-init-fail-6a365a-predictor-5467b89cf6-8hvpg\" (UID: \"e7997e4a-f9a6-44da-a217-fe19d9c42971\") " pod="kserve-ci-e2e-test/isvc-init-fail-6a365a-predictor-5467b89cf6-8hvpg"
Jul 28 16:05:27.248384 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:05:27.248356    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22nh9_ebf75c0d-1597-4a1c-b668-dc2a041eb35e/konnectivity-agent/0.log" limit=1024
Jul 28 16:05:27.315953 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:05:27.315915    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-init-fail-6a365a-predictor-5467b89cf6-8hvpg"
Jul 28 16:05:27.450675 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:05:27.450641    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-6a365a-predictor-5467b89cf6-8hvpg"]
Jul 28 16:05:27.453886 ip-10-0-131-121 kubenswrapper[2568]: W0728 16:05:27.453855    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode7997e4a_f9a6_44da_a217_fe19d9c42971.slice/crio-c3536e25b233be57f4a2b53f178032326d62d7ea4ca6d27d5f99b4c7d7aefc5b WatchSource:0}: Error finding container c3536e25b233be57f4a2b53f178032326d62d7ea4ca6d27d5f99b4c7d7aefc5b: Status 404 returned error can't find the container with id c3536e25b233be57f4a2b53f178032326d62d7ea4ca6d27d5f99b4c7d7aefc5b
Jul 28 16:05:28.172044 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:05:28.172010    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-6a365a-predictor-5467b89cf6-8hvpg" event={"ID":"e7997e4a-f9a6-44da-a217-fe19d9c42971","Type":"ContainerStarted","Data":"3845fdab4fbd3457da5f75926143af80b7f5cb5ba5099655b4b88b83d4bb05b1"}
Jul 28 16:05:28.172498 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:05:28.172084    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-6a365a-predictor-5467b89cf6-8hvpg" event={"ID":"e7997e4a-f9a6-44da-a217-fe19d9c42971","Type":"ContainerStarted","Data":"c3536e25b233be57f4a2b53f178032326d62d7ea4ca6d27d5f99b4c7d7aefc5b"}
Jul 28 16:05:28.295076 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:05:28.295043    2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a535bf88-d6b8-4a74-a6cd-43c846495c56" path="/var/lib/kubelet/pods/a535bf88-d6b8-4a74-a6cd-43c846495c56/volumes"
Jul 28 16:05:35.204941 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:05:35.204915    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-6a365a-predictor-5467b89cf6-8hvpg_e7997e4a-f9a6-44da-a217-fe19d9c42971/storage-initializer/0.log"
Jul 28 16:05:35.205339 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:05:35.204952    2568 generic.go:356] "Generic (PLEG): container finished" podID="e7997e4a-f9a6-44da-a217-fe19d9c42971" containerID="3845fdab4fbd3457da5f75926143af80b7f5cb5ba5099655b4b88b83d4bb05b1" exitCode=1
Jul 28 16:05:35.205339 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:05:35.204996    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-6a365a-predictor-5467b89cf6-8hvpg" event={"ID":"e7997e4a-f9a6-44da-a217-fe19d9c42971","Type":"ContainerDied","Data":"3845fdab4fbd3457da5f75926143af80b7f5cb5ba5099655b4b88b83d4bb05b1"}
Jul 28 16:05:36.210619 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:05:36.210589    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-6a365a-predictor-5467b89cf6-8hvpg_e7997e4a-f9a6-44da-a217-fe19d9c42971/storage-initializer/0.log"
Jul 28 16:05:36.211011 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:05:36.210687    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-6a365a-predictor-5467b89cf6-8hvpg" event={"ID":"e7997e4a-f9a6-44da-a217-fe19d9c42971","Type":"ContainerStarted","Data":"bd51116aadbb7eb76a26ebf5c37a9ddeb8c4efe201fed6342b7f84fd3b40560d"}
Jul 28 16:05:37.012318 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:05:37.012284    2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-6a365a-predictor-5467b89cf6-8hvpg"]
Jul 28 16:05:37.027058 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:05:37.027020    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22nh9_ebf75c0d-1597-4a1c-b668-dc2a041eb35e/konnectivity-agent/0.log" limit=1024
Jul 28 16:05:37.215052 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:05:37.215012    2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-init-fail-6a365a-predictor-5467b89cf6-8hvpg" podUID="e7997e4a-f9a6-44da-a217-fe19d9c42971" containerName="storage-initializer" containerID="cri-o://bd51116aadbb7eb76a26ebf5c37a9ddeb8c4efe201fed6342b7f84fd3b40560d" gracePeriod=30
Jul 28 16:05:40.228963 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:05:40.228932    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-6a365a-predictor-5467b89cf6-8hvpg_e7997e4a-f9a6-44da-a217-fe19d9c42971/storage-initializer/1.log"
Jul 28 16:05:40.229372 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:05:40.229327    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-6a365a-predictor-5467b89cf6-8hvpg_e7997e4a-f9a6-44da-a217-fe19d9c42971/storage-initializer/0.log"
Jul 28 16:05:40.229372 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:05:40.229359    2568 generic.go:356] "Generic (PLEG): container finished" podID="e7997e4a-f9a6-44da-a217-fe19d9c42971" containerID="bd51116aadbb7eb76a26ebf5c37a9ddeb8c4efe201fed6342b7f84fd3b40560d" exitCode=1
Jul 28 16:05:40.229450 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:05:40.229415    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-6a365a-predictor-5467b89cf6-8hvpg" event={"ID":"e7997e4a-f9a6-44da-a217-fe19d9c42971","Type":"ContainerDied","Data":"bd51116aadbb7eb76a26ebf5c37a9ddeb8c4efe201fed6342b7f84fd3b40560d"}
Jul 28 16:05:40.229485 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:05:40.229472    2568 scope.go:117] "RemoveContainer" containerID="3845fdab4fbd3457da5f75926143af80b7f5cb5ba5099655b4b88b83d4bb05b1"
Jul 28 16:05:40.264738 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:05:40.264718    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-6a365a-predictor-5467b89cf6-8hvpg_e7997e4a-f9a6-44da-a217-fe19d9c42971/storage-initializer/1.log"
Jul 28 16:05:40.268318 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:05:40.268302    2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-init-fail-6a365a-predictor-5467b89cf6-8hvpg"
Jul 28 16:05:40.340840 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:05:40.340810    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e7997e4a-f9a6-44da-a217-fe19d9c42971-kserve-provision-location\") pod \"e7997e4a-f9a6-44da-a217-fe19d9c42971\" (UID: \"e7997e4a-f9a6-44da-a217-fe19d9c42971\") "
Jul 28 16:05:40.341044 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:05:40.340855    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e7997e4a-f9a6-44da-a217-fe19d9c42971-kube-rbac-proxy-sar-config\") pod \"e7997e4a-f9a6-44da-a217-fe19d9c42971\" (UID: \"e7997e4a-f9a6-44da-a217-fe19d9c42971\") "
Jul 28 16:05:40.341044 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:05:40.340886    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e7997e4a-f9a6-44da-a217-fe19d9c42971-proxy-tls\") pod \"e7997e4a-f9a6-44da-a217-fe19d9c42971\" (UID: \"e7997e4a-f9a6-44da-a217-fe19d9c42971\") "
Jul 28 16:05:40.341044 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:05:40.340902    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xwb6n\" (UniqueName: \"kubernetes.io/projected/e7997e4a-f9a6-44da-a217-fe19d9c42971-kube-api-access-xwb6n\") pod \"e7997e4a-f9a6-44da-a217-fe19d9c42971\" (UID: \"e7997e4a-f9a6-44da-a217-fe19d9c42971\") "
Jul 28 16:05:40.341044 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:05:40.340947    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e7997e4a-f9a6-44da-a217-fe19d9c42971-cabundle-cert\") pod \"e7997e4a-f9a6-44da-a217-fe19d9c42971\" (UID: \"e7997e4a-f9a6-44da-a217-fe19d9c42971\") "
Jul 28 16:05:40.341271 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:05:40.341145    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e7997e4a-f9a6-44da-a217-fe19d9c42971-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "e7997e4a-f9a6-44da-a217-fe19d9c42971" (UID: "e7997e4a-f9a6-44da-a217-fe19d9c42971"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 16:05:40.341332 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:05:40.341281    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e7997e4a-f9a6-44da-a217-fe19d9c42971-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "e7997e4a-f9a6-44da-a217-fe19d9c42971" (UID: "e7997e4a-f9a6-44da-a217-fe19d9c42971"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 16:05:40.341381 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:05:40.341358    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e7997e4a-f9a6-44da-a217-fe19d9c42971-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "e7997e4a-f9a6-44da-a217-fe19d9c42971" (UID: "e7997e4a-f9a6-44da-a217-fe19d9c42971"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 16:05:40.342964 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:05:40.342942    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e7997e4a-f9a6-44da-a217-fe19d9c42971-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "e7997e4a-f9a6-44da-a217-fe19d9c42971" (UID: "e7997e4a-f9a6-44da-a217-fe19d9c42971"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 16:05:40.343180 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:05:40.343154    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e7997e4a-f9a6-44da-a217-fe19d9c42971-kube-api-access-xwb6n" (OuterVolumeSpecName: "kube-api-access-xwb6n") pod "e7997e4a-f9a6-44da-a217-fe19d9c42971" (UID: "e7997e4a-f9a6-44da-a217-fe19d9c42971"). InnerVolumeSpecName "kube-api-access-xwb6n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 16:05:40.442392 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:05:40.442354    2568 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e7997e4a-f9a6-44da-a217-fe19d9c42971-proxy-tls\") on node \"ip-10-0-131-121.ec2.internal\" DevicePath \"\""
Jul 28 16:05:40.442392 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:05:40.442384    2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xwb6n\" (UniqueName: \"kubernetes.io/projected/e7997e4a-f9a6-44da-a217-fe19d9c42971-kube-api-access-xwb6n\") on node \"ip-10-0-131-121.ec2.internal\" DevicePath \"\""
Jul 28 16:05:40.442392 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:05:40.442399    2568 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e7997e4a-f9a6-44da-a217-fe19d9c42971-cabundle-cert\") on node \"ip-10-0-131-121.ec2.internal\" DevicePath \"\""
Jul 28 16:05:40.442630 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:05:40.442413    2568 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e7997e4a-f9a6-44da-a217-fe19d9c42971-kserve-provision-location\") on node \"ip-10-0-131-121.ec2.internal\" DevicePath \"\""
Jul 28 16:05:40.442630 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:05:40.442425    2568 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e7997e4a-f9a6-44da-a217-fe19d9c42971-kube-rbac-proxy-sar-config\") on node \"ip-10-0-131-121.ec2.internal\" DevicePath \"\""
Jul 28 16:05:41.234696 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:05:41.234656    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-6a365a-predictor-5467b89cf6-8hvpg" event={"ID":"e7997e4a-f9a6-44da-a217-fe19d9c42971","Type":"ContainerDied","Data":"c3536e25b233be57f4a2b53f178032326d62d7ea4ca6d27d5f99b4c7d7aefc5b"}
Jul 28 16:05:41.235224 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:05:41.234668    2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-init-fail-6a365a-predictor-5467b89cf6-8hvpg"
Jul 28 16:05:41.235224 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:05:41.234768    2568 scope.go:117] "RemoveContainer" containerID="bd51116aadbb7eb76a26ebf5c37a9ddeb8c4efe201fed6342b7f84fd3b40560d"
Jul 28 16:05:41.286113 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:05:41.286079    2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-6a365a-predictor-5467b89cf6-8hvpg"]
Jul 28 16:05:41.290189 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:05:41.290158    2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-6a365a-predictor-5467b89cf6-8hvpg"]
Jul 28 16:05:42.202177 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:05:42.202134    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22nh9_ebf75c0d-1597-4a1c-b668-dc2a041eb35e/konnectivity-agent/0.log" limit=1024
Jul 28 16:05:42.240341 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:05:42.240308    2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-fcfc1-predictor-f6dd84fdd-m492j"]
Jul 28 16:05:42.240842 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:05:42.240703    2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e7997e4a-f9a6-44da-a217-fe19d9c42971" containerName="storage-initializer"
Jul 28 16:05:42.240842 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:05:42.240717    2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="e7997e4a-f9a6-44da-a217-fe19d9c42971" containerName="storage-initializer"
Jul 28 16:05:42.240842 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:05:42.240737    2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e7997e4a-f9a6-44da-a217-fe19d9c42971" containerName="storage-initializer"
Jul 28 16:05:42.240842 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:05:42.240745    2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="e7997e4a-f9a6-44da-a217-fe19d9c42971" containerName="storage-initializer"
Jul 28 16:05:42.240842 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:05:42.240841    2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="e7997e4a-f9a6-44da-a217-fe19d9c42971" containerName="storage-initializer"
Jul 28 16:05:42.241121 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:05:42.240942    2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="e7997e4a-f9a6-44da-a217-fe19d9c42971" containerName="storage-initializer"
Jul 28 16:05:42.245413 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:05:42.245396    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/raw-sklearn-fcfc1-predictor-f6dd84fdd-m492j"
Jul 28 16:05:42.247950 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:05:42.247928    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 28 16:05:42.248091 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:05:42.247953    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"raw-sklearn-fcfc1-predictor-serving-cert\""
Jul 28 16:05:42.248091 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:05:42.247985    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 28 16:05:42.248254 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:05:42.248229    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-gcwwc\""
Jul 28 16:05:42.248254 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:05:42.248233    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 28 16:05:42.249117 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:05:42.249102    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 28 16:05:42.249225 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:05:42.249117    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"raw-sklearn-fcfc1-kube-rbac-proxy-sar-config\""
Jul 28 16:05:42.255703 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:05:42.255677    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-fcfc1-predictor-f6dd84fdd-m492j"]
Jul 28 16:05:42.295531 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:05:42.295498    2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e7997e4a-f9a6-44da-a217-fe19d9c42971" path="/var/lib/kubelet/pods/e7997e4a-f9a6-44da-a217-fe19d9c42971/volumes"
Jul 28 16:05:42.359089 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:05:42.359047    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9a70e00e-b0e5-47e7-9f82-34ad7d24929e-cabundle-cert\") pod \"raw-sklearn-fcfc1-predictor-f6dd84fdd-m492j\" (UID: \"9a70e00e-b0e5-47e7-9f82-34ad7d24929e\") " pod="kserve-ci-e2e-test/raw-sklearn-fcfc1-predictor-f6dd84fdd-m492j"
Jul 28 16:05:42.359275 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:05:42.359109    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9a70e00e-b0e5-47e7-9f82-34ad7d24929e-proxy-tls\") pod \"raw-sklearn-fcfc1-predictor-f6dd84fdd-m492j\" (UID: \"9a70e00e-b0e5-47e7-9f82-34ad7d24929e\") " pod="kserve-ci-e2e-test/raw-sklearn-fcfc1-predictor-f6dd84fdd-m492j"
Jul 28 16:05:42.359275 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:05:42.359153    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9a70e00e-b0e5-47e7-9f82-34ad7d24929e-kube-rbac-proxy-sar-config\") pod \"raw-sklearn-fcfc1-predictor-f6dd84fdd-m492j\" (UID: \"9a70e00e-b0e5-47e7-9f82-34ad7d24929e\") " pod="kserve-ci-e2e-test/raw-sklearn-fcfc1-predictor-f6dd84fdd-m492j"
Jul 28 16:05:42.359275 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:05:42.359185    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7pcmx\" (UniqueName: \"kubernetes.io/projected/9a70e00e-b0e5-47e7-9f82-34ad7d24929e-kube-api-access-7pcmx\") pod \"raw-sklearn-fcfc1-predictor-f6dd84fdd-m492j\" (UID: \"9a70e00e-b0e5-47e7-9f82-34ad7d24929e\") " pod="kserve-ci-e2e-test/raw-sklearn-fcfc1-predictor-f6dd84fdd-m492j"
Jul 28 16:05:42.359275 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:05:42.359255    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9a70e00e-b0e5-47e7-9f82-34ad7d24929e-kserve-provision-location\") pod \"raw-sklearn-fcfc1-predictor-f6dd84fdd-m492j\" (UID: \"9a70e00e-b0e5-47e7-9f82-34ad7d24929e\") " pod="kserve-ci-e2e-test/raw-sklearn-fcfc1-predictor-f6dd84fdd-m492j"
Jul 28 16:05:42.460337 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:05:42.460247    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9a70e00e-b0e5-47e7-9f82-34ad7d24929e-proxy-tls\") pod \"raw-sklearn-fcfc1-predictor-f6dd84fdd-m492j\" (UID: \"9a70e00e-b0e5-47e7-9f82-34ad7d24929e\") " pod="kserve-ci-e2e-test/raw-sklearn-fcfc1-predictor-f6dd84fdd-m492j"
Jul 28 16:05:42.460337 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:05:42.460286    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9a70e00e-b0e5-47e7-9f82-34ad7d24929e-kube-rbac-proxy-sar-config\") pod \"raw-sklearn-fcfc1-predictor-f6dd84fdd-m492j\" (UID: \"9a70e00e-b0e5-47e7-9f82-34ad7d24929e\") " pod="kserve-ci-e2e-test/raw-sklearn-fcfc1-predictor-f6dd84fdd-m492j"
Jul 28 16:05:42.460337 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:05:42.460316    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7pcmx\" (UniqueName: \"kubernetes.io/projected/9a70e00e-b0e5-47e7-9f82-34ad7d24929e-kube-api-access-7pcmx\") pod \"raw-sklearn-fcfc1-predictor-f6dd84fdd-m492j\" (UID: \"9a70e00e-b0e5-47e7-9f82-34ad7d24929e\") " pod="kserve-ci-e2e-test/raw-sklearn-fcfc1-predictor-f6dd84fdd-m492j"
Jul 28 16:05:42.460624 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:05:42.460345    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9a70e00e-b0e5-47e7-9f82-34ad7d24929e-kserve-provision-location\") pod \"raw-sklearn-fcfc1-predictor-f6dd84fdd-m492j\" (UID: \"9a70e00e-b0e5-47e7-9f82-34ad7d24929e\") " pod="kserve-ci-e2e-test/raw-sklearn-fcfc1-predictor-f6dd84fdd-m492j"
Jul 28 16:05:42.460624 ip-10-0-131-121 kubenswrapper[2568]: E0728 16:05:42.460402    2568 secret.go:189] Couldn't get secret kserve-ci-e2e-test/raw-sklearn-fcfc1-predictor-serving-cert: secret "raw-sklearn-fcfc1-predictor-serving-cert" not found
Jul 28 16:05:42.460624 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:05:42.460406    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9a70e00e-b0e5-47e7-9f82-34ad7d24929e-cabundle-cert\") pod \"raw-sklearn-fcfc1-predictor-f6dd84fdd-m492j\" (UID: \"9a70e00e-b0e5-47e7-9f82-34ad7d24929e\") " pod="kserve-ci-e2e-test/raw-sklearn-fcfc1-predictor-f6dd84fdd-m492j"
Jul 28 16:05:42.460624 ip-10-0-131-121 kubenswrapper[2568]: E0728 16:05:42.460488    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9a70e00e-b0e5-47e7-9f82-34ad7d24929e-proxy-tls podName:9a70e00e-b0e5-47e7-9f82-34ad7d24929e nodeName:}" failed. No retries permitted until 2026-07-28 16:05:42.960467669 +0000 UTC m=+1923.145662832 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/9a70e00e-b0e5-47e7-9f82-34ad7d24929e-proxy-tls") pod "raw-sklearn-fcfc1-predictor-f6dd84fdd-m492j" (UID: "9a70e00e-b0e5-47e7-9f82-34ad7d24929e") : secret "raw-sklearn-fcfc1-predictor-serving-cert" not found
Jul 28 16:05:42.460881 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:05:42.460838    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9a70e00e-b0e5-47e7-9f82-34ad7d24929e-kserve-provision-location\") pod \"raw-sklearn-fcfc1-predictor-f6dd84fdd-m492j\" (UID: \"9a70e00e-b0e5-47e7-9f82-34ad7d24929e\") " pod="kserve-ci-e2e-test/raw-sklearn-fcfc1-predictor-f6dd84fdd-m492j"
Jul 28 16:05:42.461093 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:05:42.461073    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9a70e00e-b0e5-47e7-9f82-34ad7d24929e-cabundle-cert\") pod \"raw-sklearn-fcfc1-predictor-f6dd84fdd-m492j\" (UID: \"9a70e00e-b0e5-47e7-9f82-34ad7d24929e\") " pod="kserve-ci-e2e-test/raw-sklearn-fcfc1-predictor-f6dd84fdd-m492j"
Jul 28 16:05:42.461151 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:05:42.461074    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9a70e00e-b0e5-47e7-9f82-34ad7d24929e-kube-rbac-proxy-sar-config\") pod \"raw-sklearn-fcfc1-predictor-f6dd84fdd-m492j\" (UID: \"9a70e00e-b0e5-47e7-9f82-34ad7d24929e\") " pod="kserve-ci-e2e-test/raw-sklearn-fcfc1-predictor-f6dd84fdd-m492j"
Jul 28 16:05:42.470876 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:05:42.470853    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7pcmx\" (UniqueName: \"kubernetes.io/projected/9a70e00e-b0e5-47e7-9f82-34ad7d24929e-kube-api-access-7pcmx\") pod \"raw-sklearn-fcfc1-predictor-f6dd84fdd-m492j\" (UID: \"9a70e00e-b0e5-47e7-9f82-34ad7d24929e\") " pod="kserve-ci-e2e-test/raw-sklearn-fcfc1-predictor-f6dd84fdd-m492j"
Jul 28 16:05:42.734454 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:05:42.734369    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22nh9_ebf75c0d-1597-4a1c-b668-dc2a041eb35e/konnectivity-agent/0.log" limit=1024
Jul 28 16:05:42.964182 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:05:42.964141    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9a70e00e-b0e5-47e7-9f82-34ad7d24929e-proxy-tls\") pod \"raw-sklearn-fcfc1-predictor-f6dd84fdd-m492j\" (UID: \"9a70e00e-b0e5-47e7-9f82-34ad7d24929e\") " pod="kserve-ci-e2e-test/raw-sklearn-fcfc1-predictor-f6dd84fdd-m492j"
Jul 28 16:05:42.966597 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:05:42.966571    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9a70e00e-b0e5-47e7-9f82-34ad7d24929e-proxy-tls\") pod \"raw-sklearn-fcfc1-predictor-f6dd84fdd-m492j\" (UID: \"9a70e00e-b0e5-47e7-9f82-34ad7d24929e\") " pod="kserve-ci-e2e-test/raw-sklearn-fcfc1-predictor-f6dd84fdd-m492j"
Jul 28 16:05:43.158709 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:05:43.158668    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/raw-sklearn-fcfc1-predictor-f6dd84fdd-m492j"
Jul 28 16:05:43.293295 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:05:43.293272    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-fcfc1-predictor-f6dd84fdd-m492j"]
Jul 28 16:05:43.295467 ip-10-0-131-121 kubenswrapper[2568]: W0728 16:05:43.295444    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9a70e00e_b0e5_47e7_9f82_34ad7d24929e.slice/crio-862ec8cff09c6ba1787cd3257dffe1fe1f34426e2960678c2c55e83d3b0eb5d1 WatchSource:0}: Error finding container 862ec8cff09c6ba1787cd3257dffe1fe1f34426e2960678c2c55e83d3b0eb5d1: Status 404 returned error can't find the container with id 862ec8cff09c6ba1787cd3257dffe1fe1f34426e2960678c2c55e83d3b0eb5d1
Jul 28 16:05:44.252250 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:05:44.252211    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-fcfc1-predictor-f6dd84fdd-m492j" event={"ID":"9a70e00e-b0e5-47e7-9f82-34ad7d24929e","Type":"ContainerStarted","Data":"75731ca7c55c6787af5f62a08ef1a5ea871febc890878191dd8f034938d93c2d"}
Jul 28 16:05:44.252424 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:05:44.252267    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-fcfc1-predictor-f6dd84fdd-m492j" event={"ID":"9a70e00e-b0e5-47e7-9f82-34ad7d24929e","Type":"ContainerStarted","Data":"862ec8cff09c6ba1787cd3257dffe1fe1f34426e2960678c2c55e83d3b0eb5d1"}
Jul 28 16:05:47.267616 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:05:47.267579    2568 generic.go:356] "Generic (PLEG): container finished" podID="9a70e00e-b0e5-47e7-9f82-34ad7d24929e" containerID="75731ca7c55c6787af5f62a08ef1a5ea871febc890878191dd8f034938d93c2d" exitCode=0
Jul 28 16:05:47.268030 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:05:47.267651    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-fcfc1-predictor-f6dd84fdd-m492j" event={"ID":"9a70e00e-b0e5-47e7-9f82-34ad7d24929e","Type":"ContainerDied","Data":"75731ca7c55c6787af5f62a08ef1a5ea871febc890878191dd8f034938d93c2d"}
Jul 28 16:05:48.275086 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:05:48.275052    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-fcfc1-predictor-f6dd84fdd-m492j" event={"ID":"9a70e00e-b0e5-47e7-9f82-34ad7d24929e","Type":"ContainerStarted","Data":"ea646a37b215077ac1017890771ac3caac3e3bb08193ab4d849ec98eadf55ff4"}
Jul 28 16:05:48.275480 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:05:48.275144    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-fcfc1-predictor-f6dd84fdd-m492j" event={"ID":"9a70e00e-b0e5-47e7-9f82-34ad7d24929e","Type":"ContainerStarted","Data":"ed8c7f1c86fcfac8ccf24ef75e7a5a98a48d02a76eceeb41f4bf44dbb855bbf1"}
Jul 28 16:05:48.275480 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:05:48.275384    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/raw-sklearn-fcfc1-predictor-f6dd84fdd-m492j"
Jul 28 16:05:48.294187 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:05:48.294141    2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/raw-sklearn-fcfc1-predictor-f6dd84fdd-m492j" podStartSLOduration=2.65765321 podStartE2EDuration="6.294126905s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.636473695s" isStatefulPod=false podCreationTimestamp="2026-07-28 16:05:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 16:05:48.292972063 +0000 UTC m=+1928.478167246" watchObservedRunningTime="2026-07-28 16:05:48.294126905 +0000 UTC m=+1928.479322153"
Jul 28 16:05:49.279654 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:05:49.279626    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/raw-sklearn-fcfc1-predictor-f6dd84fdd-m492j"
Jul 28 16:05:49.280850 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:05:49.280825    2568 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-fcfc1-predictor-f6dd84fdd-m492j" podUID="9a70e00e-b0e5-47e7-9f82-34ad7d24929e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.37:8080: connect: connection refused"
Jul 28 16:05:50.284227 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:05:50.284190    2568 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-fcfc1-predictor-f6dd84fdd-m492j" podUID="9a70e00e-b0e5-47e7-9f82-34ad7d24929e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.37:8080: connect: connection refused"
Jul 28 16:05:55.288648 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:05:55.288619    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/raw-sklearn-fcfc1-predictor-f6dd84fdd-m492j"
Jul 28 16:05:55.289151 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:05:55.289122    2568 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-fcfc1-predictor-f6dd84fdd-m492j" podUID="9a70e00e-b0e5-47e7-9f82-34ad7d24929e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.37:8080: connect: connection refused"
Jul 28 16:06:05.289299 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:06:05.289257    2568 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-fcfc1-predictor-f6dd84fdd-m492j" podUID="9a70e00e-b0e5-47e7-9f82-34ad7d24929e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.37:8080: connect: connection refused"
Jul 28 16:06:15.289662 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:06:15.289621    2568 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-fcfc1-predictor-f6dd84fdd-m492j" podUID="9a70e00e-b0e5-47e7-9f82-34ad7d24929e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.37:8080: connect: connection refused"
Jul 28 16:06:25.289539 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:06:25.289496    2568 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-fcfc1-predictor-f6dd84fdd-m492j" podUID="9a70e00e-b0e5-47e7-9f82-34ad7d24929e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.37:8080: connect: connection refused"
Jul 28 16:06:35.289235 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:06:35.289142    2568 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-fcfc1-predictor-f6dd84fdd-m492j" podUID="9a70e00e-b0e5-47e7-9f82-34ad7d24929e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.37:8080: connect: connection refused"
Jul 28 16:06:45.289745 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:06:45.289705    2568 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-fcfc1-predictor-f6dd84fdd-m492j" podUID="9a70e00e-b0e5-47e7-9f82-34ad7d24929e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.37:8080: connect: connection refused"
Jul 28 16:06:55.289491 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:06:55.289449    2568 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-fcfc1-predictor-f6dd84fdd-m492j" podUID="9a70e00e-b0e5-47e7-9f82-34ad7d24929e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.37:8080: connect: connection refused"
Jul 28 16:07:05.290646 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:07:05.290602    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/raw-sklearn-fcfc1-predictor-f6dd84fdd-m492j"
Jul 28 16:07:07.899578 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:07:07.899543    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22nh9_ebf75c0d-1597-4a1c-b668-dc2a041eb35e/konnectivity-agent/0.log" limit=1024
Jul 28 16:07:12.430112 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:07:12.430077    2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-fcfc1-predictor-f6dd84fdd-m492j"]
Jul 28 16:07:12.430580 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:07:12.430285    2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/raw-sklearn-fcfc1-predictor-f6dd84fdd-m492j" podUID="9a70e00e-b0e5-47e7-9f82-34ad7d24929e" containerName="kserve-container" containerID="cri-o://ed8c7f1c86fcfac8ccf24ef75e7a5a98a48d02a76eceeb41f4bf44dbb855bbf1" gracePeriod=30
Jul 28 16:07:12.430801 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:07:12.430737    2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/raw-sklearn-fcfc1-predictor-f6dd84fdd-m492j" podUID="9a70e00e-b0e5-47e7-9f82-34ad7d24929e" containerName="kube-rbac-proxy" containerID="cri-o://ea646a37b215077ac1017890771ac3caac3e3bb08193ab4d849ec98eadf55ff4" gracePeriod=30
Jul 28 16:07:12.442944 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:07:12.442917    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22nh9_ebf75c0d-1597-4a1c-b668-dc2a041eb35e/konnectivity-agent/0.log" limit=1024
Jul 28 16:07:12.647298 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:07:12.647266    2568 generic.go:356] "Generic (PLEG): container finished" podID="9a70e00e-b0e5-47e7-9f82-34ad7d24929e" containerID="ea646a37b215077ac1017890771ac3caac3e3bb08193ab4d849ec98eadf55ff4" exitCode=2
Jul 28 16:07:12.647453 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:07:12.647322    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-fcfc1-predictor-f6dd84fdd-m492j" event={"ID":"9a70e00e-b0e5-47e7-9f82-34ad7d24929e","Type":"ContainerDied","Data":"ea646a37b215077ac1017890771ac3caac3e3bb08193ab4d849ec98eadf55ff4"}
Jul 28 16:07:15.284424 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:07:15.284376    2568 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-fcfc1-predictor-f6dd84fdd-m492j" podUID="9a70e00e-b0e5-47e7-9f82-34ad7d24929e" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.37:8643/healthz\": dial tcp 10.134.0.37:8643: connect: connection refused"
Jul 28 16:07:15.289700 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:07:15.289668    2568 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-fcfc1-predictor-f6dd84fdd-m492j" podUID="9a70e00e-b0e5-47e7-9f82-34ad7d24929e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.37:8080: connect: connection refused"
Jul 28 16:07:17.285124 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:07:17.285097    2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/raw-sklearn-fcfc1-predictor-f6dd84fdd-m492j"
Jul 28 16:07:17.386673 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:07:17.386573    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9a70e00e-b0e5-47e7-9f82-34ad7d24929e-kserve-provision-location\") pod \"9a70e00e-b0e5-47e7-9f82-34ad7d24929e\" (UID: \"9a70e00e-b0e5-47e7-9f82-34ad7d24929e\") "
Jul 28 16:07:17.386673 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:07:17.386638    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9a70e00e-b0e5-47e7-9f82-34ad7d24929e-proxy-tls\") pod \"9a70e00e-b0e5-47e7-9f82-34ad7d24929e\" (UID: \"9a70e00e-b0e5-47e7-9f82-34ad7d24929e\") "
Jul 28 16:07:17.386673 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:07:17.386671    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9a70e00e-b0e5-47e7-9f82-34ad7d24929e-cabundle-cert\") pod \"9a70e00e-b0e5-47e7-9f82-34ad7d24929e\" (UID: \"9a70e00e-b0e5-47e7-9f82-34ad7d24929e\") "
Jul 28 16:07:17.387019 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:07:17.386708    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9a70e00e-b0e5-47e7-9f82-34ad7d24929e-kube-rbac-proxy-sar-config\") pod \"9a70e00e-b0e5-47e7-9f82-34ad7d24929e\" (UID: \"9a70e00e-b0e5-47e7-9f82-34ad7d24929e\") "
Jul 28 16:07:17.387019 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:07:17.386741    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7pcmx\" (UniqueName: \"kubernetes.io/projected/9a70e00e-b0e5-47e7-9f82-34ad7d24929e-kube-api-access-7pcmx\") pod \"9a70e00e-b0e5-47e7-9f82-34ad7d24929e\" (UID: \"9a70e00e-b0e5-47e7-9f82-34ad7d24929e\") "
Jul 28 16:07:17.387019 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:07:17.387005    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9a70e00e-b0e5-47e7-9f82-34ad7d24929e-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "9a70e00e-b0e5-47e7-9f82-34ad7d24929e" (UID: "9a70e00e-b0e5-47e7-9f82-34ad7d24929e"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 16:07:17.387177 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:07:17.387126    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9a70e00e-b0e5-47e7-9f82-34ad7d24929e-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "9a70e00e-b0e5-47e7-9f82-34ad7d24929e" (UID: "9a70e00e-b0e5-47e7-9f82-34ad7d24929e"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 16:07:17.387224 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:07:17.387138    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9a70e00e-b0e5-47e7-9f82-34ad7d24929e-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "9a70e00e-b0e5-47e7-9f82-34ad7d24929e" (UID: "9a70e00e-b0e5-47e7-9f82-34ad7d24929e"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 16:07:17.389021 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:07:17.388986    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9a70e00e-b0e5-47e7-9f82-34ad7d24929e-kube-api-access-7pcmx" (OuterVolumeSpecName: "kube-api-access-7pcmx") pod "9a70e00e-b0e5-47e7-9f82-34ad7d24929e" (UID: "9a70e00e-b0e5-47e7-9f82-34ad7d24929e"). InnerVolumeSpecName "kube-api-access-7pcmx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 16:07:17.389139 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:07:17.389033    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9a70e00e-b0e5-47e7-9f82-34ad7d24929e-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "9a70e00e-b0e5-47e7-9f82-34ad7d24929e" (UID: "9a70e00e-b0e5-47e7-9f82-34ad7d24929e"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 16:07:17.487745 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:07:17.487707    2568 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9a70e00e-b0e5-47e7-9f82-34ad7d24929e-kserve-provision-location\") on node \"ip-10-0-131-121.ec2.internal\" DevicePath \"\""
Jul 28 16:07:17.487745 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:07:17.487745    2568 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9a70e00e-b0e5-47e7-9f82-34ad7d24929e-proxy-tls\") on node \"ip-10-0-131-121.ec2.internal\" DevicePath \"\""
Jul 28 16:07:17.487968 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:07:17.487759    2568 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9a70e00e-b0e5-47e7-9f82-34ad7d24929e-cabundle-cert\") on node \"ip-10-0-131-121.ec2.internal\" DevicePath \"\""
Jul 28 16:07:17.487968 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:07:17.487771    2568 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9a70e00e-b0e5-47e7-9f82-34ad7d24929e-kube-rbac-proxy-sar-config\") on node \"ip-10-0-131-121.ec2.internal\" DevicePath \"\""
Jul 28 16:07:17.487968 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:07:17.487799    2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7pcmx\" (UniqueName: \"kubernetes.io/projected/9a70e00e-b0e5-47e7-9f82-34ad7d24929e-kube-api-access-7pcmx\") on node \"ip-10-0-131-121.ec2.internal\" DevicePath \"\""
Jul 28 16:07:17.506748 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:07:17.506705    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22nh9_ebf75c0d-1597-4a1c-b668-dc2a041eb35e/konnectivity-agent/0.log" limit=1024
Jul 28 16:07:17.562155 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:07:17.562119    2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-runtime-8771a-predictor-85dbf9b964-sjwqv"]
Jul 28 16:07:17.562452 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:07:17.562438    2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9a70e00e-b0e5-47e7-9f82-34ad7d24929e" containerName="kserve-container"
Jul 28 16:07:17.562452 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:07:17.562452    2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="9a70e00e-b0e5-47e7-9f82-34ad7d24929e" containerName="kserve-container"
Jul 28 16:07:17.562564 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:07:17.562463    2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9a70e00e-b0e5-47e7-9f82-34ad7d24929e" containerName="storage-initializer"
Jul 28 16:07:17.562564 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:07:17.562469    2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="9a70e00e-b0e5-47e7-9f82-34ad7d24929e" containerName="storage-initializer"
Jul 28 16:07:17.562564 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:07:17.562475    2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9a70e00e-b0e5-47e7-9f82-34ad7d24929e" containerName="kube-rbac-proxy"
Jul 28 16:07:17.562564 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:07:17.562481    2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="9a70e00e-b0e5-47e7-9f82-34ad7d24929e" containerName="kube-rbac-proxy"
Jul 28 16:07:17.562564 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:07:17.562531    2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="9a70e00e-b0e5-47e7-9f82-34ad7d24929e" containerName="kube-rbac-proxy"
Jul 28 16:07:17.562564 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:07:17.562540    2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="9a70e00e-b0e5-47e7-9f82-34ad7d24929e" containerName="kserve-container"
Jul 28 16:07:17.565865 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:07:17.565848    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/raw-sklearn-runtime-8771a-predictor-85dbf9b964-sjwqv"
Jul 28 16:07:17.568180 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:07:17.568160    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"raw-sklearn-runtime-8771a-predictor-serving-cert\""
Jul 28 16:07:17.568455 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:07:17.568438    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"raw-sklearn-runtime-8771a-kube-rbac-proxy-sar-config\""
Jul 28 16:07:17.576853 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:07:17.576827    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-runtime-8771a-predictor-85dbf9b964-sjwqv"]
Jul 28 16:07:17.668805 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:07:17.668698    2568 generic.go:356] "Generic (PLEG): container finished" podID="9a70e00e-b0e5-47e7-9f82-34ad7d24929e" containerID="ed8c7f1c86fcfac8ccf24ef75e7a5a98a48d02a76eceeb41f4bf44dbb855bbf1" exitCode=0
Jul 28 16:07:17.668805 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:07:17.668759    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-fcfc1-predictor-f6dd84fdd-m492j" event={"ID":"9a70e00e-b0e5-47e7-9f82-34ad7d24929e","Type":"ContainerDied","Data":"ed8c7f1c86fcfac8ccf24ef75e7a5a98a48d02a76eceeb41f4bf44dbb855bbf1"}
Jul 28 16:07:17.668805 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:07:17.668800    2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/raw-sklearn-fcfc1-predictor-f6dd84fdd-m492j"
Jul 28 16:07:17.669094 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:07:17.668842    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-fcfc1-predictor-f6dd84fdd-m492j" event={"ID":"9a70e00e-b0e5-47e7-9f82-34ad7d24929e","Type":"ContainerDied","Data":"862ec8cff09c6ba1787cd3257dffe1fe1f34426e2960678c2c55e83d3b0eb5d1"}
Jul 28 16:07:17.669094 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:07:17.668879    2568 scope.go:117] "RemoveContainer" containerID="ea646a37b215077ac1017890771ac3caac3e3bb08193ab4d849ec98eadf55ff4"
Jul 28 16:07:17.676833 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:07:17.676764    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22nh9_ebf75c0d-1597-4a1c-b668-dc2a041eb35e/konnectivity-agent/0.log" limit=1024
Jul 28 16:07:17.684071 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:07:17.684053    2568 scope.go:117] "RemoveContainer" containerID="ed8c7f1c86fcfac8ccf24ef75e7a5a98a48d02a76eceeb41f4bf44dbb855bbf1"
Jul 28 16:07:17.689457 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:07:17.689432    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6af91f59-da8e-4f6c-9115-424e438a75d1-kube-rbac-proxy-sar-config\") pod \"raw-sklearn-runtime-8771a-predictor-85dbf9b964-sjwqv\" (UID: \"6af91f59-da8e-4f6c-9115-424e438a75d1\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-8771a-predictor-85dbf9b964-sjwqv"
Jul 28 16:07:17.689577 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:07:17.689492    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6af91f59-da8e-4f6c-9115-424e438a75d1-proxy-tls\") pod \"raw-sklearn-runtime-8771a-predictor-85dbf9b964-sjwqv\" (UID: \"6af91f59-da8e-4f6c-9115-424e438a75d1\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-8771a-predictor-85dbf9b964-sjwqv"
Jul 28 16:07:17.689577 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:07:17.689546    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hbbq9\" (UniqueName: \"kubernetes.io/projected/6af91f59-da8e-4f6c-9115-424e438a75d1-kube-api-access-hbbq9\") pod \"raw-sklearn-runtime-8771a-predictor-85dbf9b964-sjwqv\" (UID: \"6af91f59-da8e-4f6c-9115-424e438a75d1\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-8771a-predictor-85dbf9b964-sjwqv"
Jul 28 16:07:17.689577 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:07:17.689568    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6af91f59-da8e-4f6c-9115-424e438a75d1-kserve-provision-location\") pod \"raw-sklearn-runtime-8771a-predictor-85dbf9b964-sjwqv\" (UID: \"6af91f59-da8e-4f6c-9115-424e438a75d1\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-8771a-predictor-85dbf9b964-sjwqv"
Jul 28 16:07:17.689685 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:07:17.689586    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6af91f59-da8e-4f6c-9115-424e438a75d1-cabundle-cert\") pod \"raw-sklearn-runtime-8771a-predictor-85dbf9b964-sjwqv\" (UID: \"6af91f59-da8e-4f6c-9115-424e438a75d1\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-8771a-predictor-85dbf9b964-sjwqv"
Jul 28 16:07:17.691741 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:07:17.691722    2568 scope.go:117] "RemoveContainer" containerID="75731ca7c55c6787af5f62a08ef1a5ea871febc890878191dd8f034938d93c2d"
Jul 28 16:07:17.694365 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:07:17.694343    2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-fcfc1-predictor-f6dd84fdd-m492j"]
Jul 28 16:07:17.699107 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:07:17.699083    2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-fcfc1-predictor-f6dd84fdd-m492j"]
Jul 28 16:07:17.700362 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:07:17.700346    2568 scope.go:117] "RemoveContainer" containerID="ea646a37b215077ac1017890771ac3caac3e3bb08193ab4d849ec98eadf55ff4"
Jul 28 16:07:17.700644 ip-10-0-131-121 kubenswrapper[2568]: E0728 16:07:17.700610    2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ea646a37b215077ac1017890771ac3caac3e3bb08193ab4d849ec98eadf55ff4\": container with ID starting with ea646a37b215077ac1017890771ac3caac3e3bb08193ab4d849ec98eadf55ff4 not found: ID does not exist" containerID="ea646a37b215077ac1017890771ac3caac3e3bb08193ab4d849ec98eadf55ff4"
Jul 28 16:07:17.700688 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:07:17.700651    2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ea646a37b215077ac1017890771ac3caac3e3bb08193ab4d849ec98eadf55ff4"} err="failed to get container status \"ea646a37b215077ac1017890771ac3caac3e3bb08193ab4d849ec98eadf55ff4\": rpc error: code = NotFound desc = could not find container \"ea646a37b215077ac1017890771ac3caac3e3bb08193ab4d849ec98eadf55ff4\": container with ID starting with ea646a37b215077ac1017890771ac3caac3e3bb08193ab4d849ec98eadf55ff4 not found: ID does not exist"
Jul 28 16:07:17.700688 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:07:17.700672    2568 scope.go:117] "RemoveContainer" containerID="ed8c7f1c86fcfac8ccf24ef75e7a5a98a48d02a76eceeb41f4bf44dbb855bbf1"
Jul 28 16:07:17.700924 ip-10-0-131-121 kubenswrapper[2568]: E0728 16:07:17.700907    2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ed8c7f1c86fcfac8ccf24ef75e7a5a98a48d02a76eceeb41f4bf44dbb855bbf1\": container with ID starting with ed8c7f1c86fcfac8ccf24ef75e7a5a98a48d02a76eceeb41f4bf44dbb855bbf1 not found: ID does not exist" containerID="ed8c7f1c86fcfac8ccf24ef75e7a5a98a48d02a76eceeb41f4bf44dbb855bbf1"
Jul 28 16:07:17.700975 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:07:17.700927    2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ed8c7f1c86fcfac8ccf24ef75e7a5a98a48d02a76eceeb41f4bf44dbb855bbf1"} err="failed to get container status \"ed8c7f1c86fcfac8ccf24ef75e7a5a98a48d02a76eceeb41f4bf44dbb855bbf1\": rpc error: code = NotFound desc = could not find container \"ed8c7f1c86fcfac8ccf24ef75e7a5a98a48d02a76eceeb41f4bf44dbb855bbf1\": container with ID starting with ed8c7f1c86fcfac8ccf24ef75e7a5a98a48d02a76eceeb41f4bf44dbb855bbf1 not found: ID does not exist"
Jul 28 16:07:17.700975 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:07:17.700942    2568 scope.go:117] "RemoveContainer" containerID="75731ca7c55c6787af5f62a08ef1a5ea871febc890878191dd8f034938d93c2d"
Jul 28 16:07:17.701169 ip-10-0-131-121 kubenswrapper[2568]: E0728 16:07:17.701149    2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"75731ca7c55c6787af5f62a08ef1a5ea871febc890878191dd8f034938d93c2d\": container with ID starting with 75731ca7c55c6787af5f62a08ef1a5ea871febc890878191dd8f034938d93c2d not found: ID does not exist" containerID="75731ca7c55c6787af5f62a08ef1a5ea871febc890878191dd8f034938d93c2d"
Jul 28 16:07:17.701214 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:07:17.701174    2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"75731ca7c55c6787af5f62a08ef1a5ea871febc890878191dd8f034938d93c2d"} err="failed to get container status \"75731ca7c55c6787af5f62a08ef1a5ea871febc890878191dd8f034938d93c2d\": rpc error: code = NotFound desc = could not find container \"75731ca7c55c6787af5f62a08ef1a5ea871febc890878191dd8f034938d93c2d\": container with ID starting with 75731ca7c55c6787af5f62a08ef1a5ea871febc890878191dd8f034938d93c2d not found: ID does not exist"
Jul 28 16:07:17.790865 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:07:17.790823    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6af91f59-da8e-4f6c-9115-424e438a75d1-proxy-tls\") pod \"raw-sklearn-runtime-8771a-predictor-85dbf9b964-sjwqv\" (UID: \"6af91f59-da8e-4f6c-9115-424e438a75d1\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-8771a-predictor-85dbf9b964-sjwqv"
Jul 28 16:07:17.791069 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:07:17.790900    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hbbq9\" (UniqueName: \"kubernetes.io/projected/6af91f59-da8e-4f6c-9115-424e438a75d1-kube-api-access-hbbq9\") pod \"raw-sklearn-runtime-8771a-predictor-85dbf9b964-sjwqv\" (UID: \"6af91f59-da8e-4f6c-9115-424e438a75d1\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-8771a-predictor-85dbf9b964-sjwqv"
Jul 28 16:07:17.791069 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:07:17.790926    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6af91f59-da8e-4f6c-9115-424e438a75d1-kserve-provision-location\") pod \"raw-sklearn-runtime-8771a-predictor-85dbf9b964-sjwqv\" (UID: \"6af91f59-da8e-4f6c-9115-424e438a75d1\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-8771a-predictor-85dbf9b964-sjwqv"
Jul 28 16:07:17.791069 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:07:17.790951    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6af91f59-da8e-4f6c-9115-424e438a75d1-cabundle-cert\") pod \"raw-sklearn-runtime-8771a-predictor-85dbf9b964-sjwqv\" (UID: \"6af91f59-da8e-4f6c-9115-424e438a75d1\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-8771a-predictor-85dbf9b964-sjwqv"
Jul 28 16:07:17.791069 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:07:17.791009    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6af91f59-da8e-4f6c-9115-424e438a75d1-kube-rbac-proxy-sar-config\") pod \"raw-sklearn-runtime-8771a-predictor-85dbf9b964-sjwqv\" (UID: \"6af91f59-da8e-4f6c-9115-424e438a75d1\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-8771a-predictor-85dbf9b964-sjwqv"
Jul 28 16:07:17.791425 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:07:17.791399    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6af91f59-da8e-4f6c-9115-424e438a75d1-kserve-provision-location\") pod \"raw-sklearn-runtime-8771a-predictor-85dbf9b964-sjwqv\" (UID: \"6af91f59-da8e-4f6c-9115-424e438a75d1\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-8771a-predictor-85dbf9b964-sjwqv"
Jul 28 16:07:17.791703 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:07:17.791684    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6af91f59-da8e-4f6c-9115-424e438a75d1-cabundle-cert\") pod \"raw-sklearn-runtime-8771a-predictor-85dbf9b964-sjwqv\" (UID: \"6af91f59-da8e-4f6c-9115-424e438a75d1\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-8771a-predictor-85dbf9b964-sjwqv"
Jul 28 16:07:17.791760 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:07:17.791690    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6af91f59-da8e-4f6c-9115-424e438a75d1-kube-rbac-proxy-sar-config\") pod \"raw-sklearn-runtime-8771a-predictor-85dbf9b964-sjwqv\" (UID: \"6af91f59-da8e-4f6c-9115-424e438a75d1\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-8771a-predictor-85dbf9b964-sjwqv"
Jul 28 16:07:17.793338 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:07:17.793320    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6af91f59-da8e-4f6c-9115-424e438a75d1-proxy-tls\") pod \"raw-sklearn-runtime-8771a-predictor-85dbf9b964-sjwqv\" (UID: \"6af91f59-da8e-4f6c-9115-424e438a75d1\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-8771a-predictor-85dbf9b964-sjwqv"
Jul 28 16:07:17.799431 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:07:17.799410    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hbbq9\" (UniqueName: \"kubernetes.io/projected/6af91f59-da8e-4f6c-9115-424e438a75d1-kube-api-access-hbbq9\") pod \"raw-sklearn-runtime-8771a-predictor-85dbf9b964-sjwqv\" (UID: \"6af91f59-da8e-4f6c-9115-424e438a75d1\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-8771a-predictor-85dbf9b964-sjwqv"
Jul 28 16:07:17.877056 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:07:17.877011    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/raw-sklearn-runtime-8771a-predictor-85dbf9b964-sjwqv"
Jul 28 16:07:18.013525 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:07:18.013467    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-runtime-8771a-predictor-85dbf9b964-sjwqv"]
Jul 28 16:07:18.015804 ip-10-0-131-121 kubenswrapper[2568]: W0728 16:07:18.015759    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6af91f59_da8e_4f6c_9115_424e438a75d1.slice/crio-6aaf99705937a91fe253f90e7cd5245901fa8d6eb8646a8811d81d234f4da70f WatchSource:0}: Error finding container 6aaf99705937a91fe253f90e7cd5245901fa8d6eb8646a8811d81d234f4da70f: Status 404 returned error can't find the container with id 6aaf99705937a91fe253f90e7cd5245901fa8d6eb8646a8811d81d234f4da70f
Jul 28 16:07:18.295244 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:07:18.295210    2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9a70e00e-b0e5-47e7-9f82-34ad7d24929e" path="/var/lib/kubelet/pods/9a70e00e-b0e5-47e7-9f82-34ad7d24929e/volumes"
Jul 28 16:07:18.674992 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:07:18.674890    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-runtime-8771a-predictor-85dbf9b964-sjwqv" event={"ID":"6af91f59-da8e-4f6c-9115-424e438a75d1","Type":"ContainerStarted","Data":"bcdff7e9d399b7307116b9dd3da1411f4f3c91a11e3864bd0e1d740c9ee760bd"}
Jul 28 16:07:18.674992 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:07:18.674961    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-runtime-8771a-predictor-85dbf9b964-sjwqv" event={"ID":"6af91f59-da8e-4f6c-9115-424e438a75d1","Type":"ContainerStarted","Data":"6aaf99705937a91fe253f90e7cd5245901fa8d6eb8646a8811d81d234f4da70f"}
Jul 28 16:07:19.754466 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:07:19.754427    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22nh9_ebf75c0d-1597-4a1c-b668-dc2a041eb35e/konnectivity-agent/0.log" limit=1024
Jul 28 16:07:19.853287 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:07:19.853257    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22nh9_ebf75c0d-1597-4a1c-b668-dc2a041eb35e/konnectivity-agent/0.log" limit=1024
Jul 28 16:07:22.694361 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:07:22.694327    2568 generic.go:356] "Generic (PLEG): container finished" podID="6af91f59-da8e-4f6c-9115-424e438a75d1" containerID="bcdff7e9d399b7307116b9dd3da1411f4f3c91a11e3864bd0e1d740c9ee760bd" exitCode=0
Jul 28 16:07:22.694847 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:07:22.694396    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-runtime-8771a-predictor-85dbf9b964-sjwqv" event={"ID":"6af91f59-da8e-4f6c-9115-424e438a75d1","Type":"ContainerDied","Data":"bcdff7e9d399b7307116b9dd3da1411f4f3c91a11e3864bd0e1d740c9ee760bd"}
Jul 28 16:07:23.700383 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:07:23.700344    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-runtime-8771a-predictor-85dbf9b964-sjwqv" event={"ID":"6af91f59-da8e-4f6c-9115-424e438a75d1","Type":"ContainerStarted","Data":"06718b6f46a4cd747a184e57ad50b32eabf8a4eb4d325e7a039af90c38104741"}
Jul 28 16:07:23.700811 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:07:23.700410    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-runtime-8771a-predictor-85dbf9b964-sjwqv" event={"ID":"6af91f59-da8e-4f6c-9115-424e438a75d1","Type":"ContainerStarted","Data":"731834751d56903fc201c307bc48ff82c465d3824144ee1b74d497fdbf84ae2c"}
Jul 28 16:07:23.700811 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:07:23.700609    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/raw-sklearn-runtime-8771a-predictor-85dbf9b964-sjwqv"
Jul 28 16:07:23.720309 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:07:23.720255    2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/raw-sklearn-runtime-8771a-predictor-85dbf9b964-sjwqv" podStartSLOduration=2.83915687 podStartE2EDuration="6.720238491s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.881081621s" isStatefulPod=false podCreationTimestamp="2026-07-28 16:07:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 16:07:23.718673474 +0000 UTC m=+2023.903868644" watchObservedRunningTime="2026-07-28 16:07:23.720238491 +0000 UTC m=+2023.905433660"
Jul 28 16:07:24.705212 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:07:24.705182    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/raw-sklearn-runtime-8771a-predictor-85dbf9b964-sjwqv"
Jul 28 16:07:24.706746 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:07:24.706716    2568 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-8771a-predictor-85dbf9b964-sjwqv" podUID="6af91f59-da8e-4f6c-9115-424e438a75d1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.38:8080: connect: connection refused"
Jul 28 16:07:25.708947 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:07:25.708911    2568 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-8771a-predictor-85dbf9b964-sjwqv" podUID="6af91f59-da8e-4f6c-9115-424e438a75d1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.38:8080: connect: connection refused"
Jul 28 16:07:30.713299 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:07:30.713269    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/raw-sklearn-runtime-8771a-predictor-85dbf9b964-sjwqv"
Jul 28 16:07:30.713931 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:07:30.713902    2568 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-8771a-predictor-85dbf9b964-sjwqv" podUID="6af91f59-da8e-4f6c-9115-424e438a75d1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.38:8080: connect: connection refused"
Jul 28 16:07:40.714566 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:07:40.714479    2568 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-8771a-predictor-85dbf9b964-sjwqv" podUID="6af91f59-da8e-4f6c-9115-424e438a75d1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.38:8080: connect: connection refused"
Jul 28 16:07:50.714002 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:07:50.713961    2568 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-8771a-predictor-85dbf9b964-sjwqv" podUID="6af91f59-da8e-4f6c-9115-424e438a75d1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.38:8080: connect: connection refused"
Jul 28 16:08:00.713908 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:08:00.713868    2568 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-8771a-predictor-85dbf9b964-sjwqv" podUID="6af91f59-da8e-4f6c-9115-424e438a75d1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.38:8080: connect: connection refused"
Jul 28 16:08:02.147269 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:08:02.147170    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22nh9_ebf75c0d-1597-4a1c-b668-dc2a041eb35e/konnectivity-agent/0.log" limit=1024
Jul 28 16:08:10.714430 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:08:10.714384    2568 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-8771a-predictor-85dbf9b964-sjwqv" podUID="6af91f59-da8e-4f6c-9115-424e438a75d1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.38:8080: connect: connection refused"
Jul 28 16:08:20.713853 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:08:20.713814    2568 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-8771a-predictor-85dbf9b964-sjwqv" podUID="6af91f59-da8e-4f6c-9115-424e438a75d1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.38:8080: connect: connection refused"
Jul 28 16:08:30.713845 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:08:30.713807    2568 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-8771a-predictor-85dbf9b964-sjwqv" podUID="6af91f59-da8e-4f6c-9115-424e438a75d1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.38:8080: connect: connection refused"
Jul 28 16:08:40.714985 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:08:40.714957    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/raw-sklearn-runtime-8771a-predictor-85dbf9b964-sjwqv"
Jul 28 16:08:47.683501 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:08:47.683461    2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-runtime-8771a-predictor-85dbf9b964-sjwqv"]
Jul 28 16:08:47.684064 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:08:47.683710    2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/raw-sklearn-runtime-8771a-predictor-85dbf9b964-sjwqv" podUID="6af91f59-da8e-4f6c-9115-424e438a75d1" containerName="kserve-container" containerID="cri-o://731834751d56903fc201c307bc48ff82c465d3824144ee1b74d497fdbf84ae2c" gracePeriod=30
Jul 28 16:08:47.684064 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:08:47.683834    2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/raw-sklearn-runtime-8771a-predictor-85dbf9b964-sjwqv" podUID="6af91f59-da8e-4f6c-9115-424e438a75d1" containerName="kube-rbac-proxy" containerID="cri-o://06718b6f46a4cd747a184e57ad50b32eabf8a4eb4d325e7a039af90c38104741" gracePeriod=30
Jul 28 16:08:47.702385 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:08:47.702357    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22nh9_ebf75c0d-1597-4a1c-b668-dc2a041eb35e/konnectivity-agent/0.log" limit=1024
Jul 28 16:08:48.057877 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:08:48.057827    2568 generic.go:356] "Generic (PLEG): container finished" podID="6af91f59-da8e-4f6c-9115-424e438a75d1" containerID="06718b6f46a4cd747a184e57ad50b32eabf8a4eb4d325e7a039af90c38104741" exitCode=2
Jul 28 16:08:48.058049 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:08:48.057885    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-runtime-8771a-predictor-85dbf9b964-sjwqv" event={"ID":"6af91f59-da8e-4f6c-9115-424e438a75d1","Type":"ContainerDied","Data":"06718b6f46a4cd747a184e57ad50b32eabf8a4eb4d325e7a039af90c38104741"}
Jul 28 16:08:50.709876 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:08:50.709827    2568 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-8771a-predictor-85dbf9b964-sjwqv" podUID="6af91f59-da8e-4f6c-9115-424e438a75d1" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.38:8643/healthz\": dial tcp 10.134.0.38:8643: connect: connection refused"
Jul 28 16:08:50.714192 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:08:50.714162    2568 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-8771a-predictor-85dbf9b964-sjwqv" podUID="6af91f59-da8e-4f6c-9115-424e438a75d1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.38:8080: connect: connection refused"
Jul 28 16:08:52.635059 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:08:52.635031    2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/raw-sklearn-runtime-8771a-predictor-85dbf9b964-sjwqv"
Jul 28 16:08:52.700642 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:08:52.700545    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6af91f59-da8e-4f6c-9115-424e438a75d1-kserve-provision-location\") pod \"6af91f59-da8e-4f6c-9115-424e438a75d1\" (UID: \"6af91f59-da8e-4f6c-9115-424e438a75d1\") "
Jul 28 16:08:52.700642 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:08:52.700591    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6af91f59-da8e-4f6c-9115-424e438a75d1-proxy-tls\") pod \"6af91f59-da8e-4f6c-9115-424e438a75d1\" (UID: \"6af91f59-da8e-4f6c-9115-424e438a75d1\") "
Jul 28 16:08:52.700642 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:08:52.700624    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6af91f59-da8e-4f6c-9115-424e438a75d1-kube-rbac-proxy-sar-config\") pod \"6af91f59-da8e-4f6c-9115-424e438a75d1\" (UID: \"6af91f59-da8e-4f6c-9115-424e438a75d1\") "
Jul 28 16:08:52.700642 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:08:52.700646    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hbbq9\" (UniqueName: \"kubernetes.io/projected/6af91f59-da8e-4f6c-9115-424e438a75d1-kube-api-access-hbbq9\") pod \"6af91f59-da8e-4f6c-9115-424e438a75d1\" (UID: \"6af91f59-da8e-4f6c-9115-424e438a75d1\") "
Jul 28 16:08:52.701028 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:08:52.700691    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6af91f59-da8e-4f6c-9115-424e438a75d1-cabundle-cert\") pod \"6af91f59-da8e-4f6c-9115-424e438a75d1\" (UID: \"6af91f59-da8e-4f6c-9115-424e438a75d1\") "
Jul 28 16:08:52.701028 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:08:52.700933    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6af91f59-da8e-4f6c-9115-424e438a75d1-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "6af91f59-da8e-4f6c-9115-424e438a75d1" (UID: "6af91f59-da8e-4f6c-9115-424e438a75d1"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 16:08:52.701146 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:08:52.701076    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6af91f59-da8e-4f6c-9115-424e438a75d1-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "6af91f59-da8e-4f6c-9115-424e438a75d1" (UID: "6af91f59-da8e-4f6c-9115-424e438a75d1"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 16:08:52.701203 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:08:52.701174    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6af91f59-da8e-4f6c-9115-424e438a75d1-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "6af91f59-da8e-4f6c-9115-424e438a75d1" (UID: "6af91f59-da8e-4f6c-9115-424e438a75d1"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 16:08:52.702818 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:08:52.702796    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6af91f59-da8e-4f6c-9115-424e438a75d1-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "6af91f59-da8e-4f6c-9115-424e438a75d1" (UID: "6af91f59-da8e-4f6c-9115-424e438a75d1"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 16:08:52.702886 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:08:52.702840    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6af91f59-da8e-4f6c-9115-424e438a75d1-kube-api-access-hbbq9" (OuterVolumeSpecName: "kube-api-access-hbbq9") pod "6af91f59-da8e-4f6c-9115-424e438a75d1" (UID: "6af91f59-da8e-4f6c-9115-424e438a75d1"). InnerVolumeSpecName "kube-api-access-hbbq9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 16:08:52.801919 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:08:52.801880    2568 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6af91f59-da8e-4f6c-9115-424e438a75d1-kserve-provision-location\") on node \"ip-10-0-131-121.ec2.internal\" DevicePath \"\""
Jul 28 16:08:52.801919 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:08:52.801910    2568 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6af91f59-da8e-4f6c-9115-424e438a75d1-proxy-tls\") on node \"ip-10-0-131-121.ec2.internal\" DevicePath \"\""
Jul 28 16:08:52.801919 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:08:52.801920    2568 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6af91f59-da8e-4f6c-9115-424e438a75d1-kube-rbac-proxy-sar-config\") on node \"ip-10-0-131-121.ec2.internal\" DevicePath \"\""
Jul 28 16:08:52.801919 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:08:52.801929    2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hbbq9\" (UniqueName: \"kubernetes.io/projected/6af91f59-da8e-4f6c-9115-424e438a75d1-kube-api-access-hbbq9\") on node \"ip-10-0-131-121.ec2.internal\" DevicePath \"\""
Jul 28 16:08:52.802186 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:08:52.801938    2568 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6af91f59-da8e-4f6c-9115-424e438a75d1-cabundle-cert\") on node \"ip-10-0-131-121.ec2.internal\" DevicePath \"\""
Jul 28 16:08:53.088141 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:08:53.088108    2568 generic.go:356] "Generic (PLEG): container finished" podID="6af91f59-da8e-4f6c-9115-424e438a75d1" containerID="731834751d56903fc201c307bc48ff82c465d3824144ee1b74d497fdbf84ae2c" exitCode=0
Jul 28 16:08:53.088344 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:08:53.088192    2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/raw-sklearn-runtime-8771a-predictor-85dbf9b964-sjwqv"
Jul 28 16:08:53.088344 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:08:53.088191    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-runtime-8771a-predictor-85dbf9b964-sjwqv" event={"ID":"6af91f59-da8e-4f6c-9115-424e438a75d1","Type":"ContainerDied","Data":"731834751d56903fc201c307bc48ff82c465d3824144ee1b74d497fdbf84ae2c"}
Jul 28 16:08:53.088470 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:08:53.088394    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-runtime-8771a-predictor-85dbf9b964-sjwqv" event={"ID":"6af91f59-da8e-4f6c-9115-424e438a75d1","Type":"ContainerDied","Data":"6aaf99705937a91fe253f90e7cd5245901fa8d6eb8646a8811d81d234f4da70f"}
Jul 28 16:08:53.088470 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:08:53.088436    2568 scope.go:117] "RemoveContainer" containerID="06718b6f46a4cd747a184e57ad50b32eabf8a4eb4d325e7a039af90c38104741"
Jul 28 16:08:53.097444 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:08:53.097420    2568 scope.go:117] "RemoveContainer" containerID="731834751d56903fc201c307bc48ff82c465d3824144ee1b74d497fdbf84ae2c"
Jul 28 16:08:53.105538 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:08:53.105517    2568 scope.go:117] "RemoveContainer" containerID="bcdff7e9d399b7307116b9dd3da1411f4f3c91a11e3864bd0e1d740c9ee760bd"
Jul 28 16:08:53.112002 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:08:53.111977    2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-runtime-8771a-predictor-85dbf9b964-sjwqv"]
Jul 28 16:08:53.113493 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:08:53.113479    2568 scope.go:117] "RemoveContainer" containerID="06718b6f46a4cd747a184e57ad50b32eabf8a4eb4d325e7a039af90c38104741"
Jul 28 16:08:53.113763 ip-10-0-131-121 kubenswrapper[2568]: E0728 16:08:53.113739    2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"06718b6f46a4cd747a184e57ad50b32eabf8a4eb4d325e7a039af90c38104741\": container with ID starting with 06718b6f46a4cd747a184e57ad50b32eabf8a4eb4d325e7a039af90c38104741 not found: ID does not exist" containerID="06718b6f46a4cd747a184e57ad50b32eabf8a4eb4d325e7a039af90c38104741"
Jul 28 16:08:53.113863 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:08:53.113771    2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"06718b6f46a4cd747a184e57ad50b32eabf8a4eb4d325e7a039af90c38104741"} err="failed to get container status \"06718b6f46a4cd747a184e57ad50b32eabf8a4eb4d325e7a039af90c38104741\": rpc error: code = NotFound desc = could not find container \"06718b6f46a4cd747a184e57ad50b32eabf8a4eb4d325e7a039af90c38104741\": container with ID starting with 06718b6f46a4cd747a184e57ad50b32eabf8a4eb4d325e7a039af90c38104741 not found: ID does not exist"
Jul 28 16:08:53.113863 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:08:53.113807    2568 scope.go:117] "RemoveContainer" containerID="731834751d56903fc201c307bc48ff82c465d3824144ee1b74d497fdbf84ae2c"
Jul 28 16:08:53.114101 ip-10-0-131-121 kubenswrapper[2568]: E0728 16:08:53.114062    2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"731834751d56903fc201c307bc48ff82c465d3824144ee1b74d497fdbf84ae2c\": container with ID starting with 731834751d56903fc201c307bc48ff82c465d3824144ee1b74d497fdbf84ae2c not found: ID does not exist" containerID="731834751d56903fc201c307bc48ff82c465d3824144ee1b74d497fdbf84ae2c"
Jul 28 16:08:53.114101 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:08:53.114093    2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"731834751d56903fc201c307bc48ff82c465d3824144ee1b74d497fdbf84ae2c"} err="failed to get container status \"731834751d56903fc201c307bc48ff82c465d3824144ee1b74d497fdbf84ae2c\": rpc error: code = NotFound desc = could not find container \"731834751d56903fc201c307bc48ff82c465d3824144ee1b74d497fdbf84ae2c\": container with ID starting with 731834751d56903fc201c307bc48ff82c465d3824144ee1b74d497fdbf84ae2c not found: ID does not exist"
Jul 28 16:08:53.114269 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:08:53.114114    2568 scope.go:117] "RemoveContainer" containerID="bcdff7e9d399b7307116b9dd3da1411f4f3c91a11e3864bd0e1d740c9ee760bd"
Jul 28 16:08:53.114501 ip-10-0-131-121 kubenswrapper[2568]: E0728 16:08:53.114478    2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bcdff7e9d399b7307116b9dd3da1411f4f3c91a11e3864bd0e1d740c9ee760bd\": container with ID starting with bcdff7e9d399b7307116b9dd3da1411f4f3c91a11e3864bd0e1d740c9ee760bd not found: ID does not exist" containerID="bcdff7e9d399b7307116b9dd3da1411f4f3c91a11e3864bd0e1d740c9ee760bd"
Jul 28 16:08:53.114659 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:08:53.114509    2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bcdff7e9d399b7307116b9dd3da1411f4f3c91a11e3864bd0e1d740c9ee760bd"} err="failed to get container status \"bcdff7e9d399b7307116b9dd3da1411f4f3c91a11e3864bd0e1d740c9ee760bd\": rpc error: code = NotFound desc = could not find container \"bcdff7e9d399b7307116b9dd3da1411f4f3c91a11e3864bd0e1d740c9ee760bd\": container with ID starting with bcdff7e9d399b7307116b9dd3da1411f4f3c91a11e3864bd0e1d740c9ee760bd not found: ID does not exist"
Jul 28 16:08:53.116205 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:08:53.116185    2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-runtime-8771a-predictor-85dbf9b964-sjwqv"]
Jul 28 16:08:54.295042 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:08:54.295001    2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6af91f59-da8e-4f6c-9115-424e438a75d1" path="/var/lib/kubelet/pods/6af91f59-da8e-4f6c-9115-424e438a75d1/volumes"
Jul 28 16:08:54.873768 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:08:54.873732    2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-7k6rg/must-gather-wxx4c"]
Jul 28 16:08:54.874067 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:08:54.874054    2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6af91f59-da8e-4f6c-9115-424e438a75d1" containerName="storage-initializer"
Jul 28 16:08:54.874119 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:08:54.874068    2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="6af91f59-da8e-4f6c-9115-424e438a75d1" containerName="storage-initializer"
Jul 28 16:08:54.874119 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:08:54.874079    2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6af91f59-da8e-4f6c-9115-424e438a75d1" containerName="kube-rbac-proxy"
Jul 28 16:08:54.874119 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:08:54.874085    2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="6af91f59-da8e-4f6c-9115-424e438a75d1" containerName="kube-rbac-proxy"
Jul 28 16:08:54.874119 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:08:54.874098    2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6af91f59-da8e-4f6c-9115-424e438a75d1" containerName="kserve-container"
Jul 28 16:08:54.874119 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:08:54.874103    2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="6af91f59-da8e-4f6c-9115-424e438a75d1" containerName="kserve-container"
Jul 28 16:08:54.874302 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:08:54.874154    2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="6af91f59-da8e-4f6c-9115-424e438a75d1" containerName="kserve-container"
Jul 28 16:08:54.874302 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:08:54.874161    2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="6af91f59-da8e-4f6c-9115-424e438a75d1" containerName="kube-rbac-proxy"
Jul 28 16:08:54.878679 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:08:54.878663    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-7k6rg/must-gather-wxx4c"
Jul 28 16:08:54.881118 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:08:54.881096    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-7k6rg\"/\"kube-root-ca.crt\""
Jul 28 16:08:54.882151 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:08:54.882134    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-7k6rg\"/\"default-dockercfg-sn946\""
Jul 28 16:08:54.882200 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:08:54.882134    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-7k6rg\"/\"openshift-service-ca.crt\""
Jul 28 16:08:54.886030 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:08:54.886007    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-7k6rg/must-gather-wxx4c"]
Jul 28 16:08:54.919963 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:08:54.919925    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/9c058ebd-07f8-4696-8f06-840b07af0f3c-must-gather-output\") pod \"must-gather-wxx4c\" (UID: \"9c058ebd-07f8-4696-8f06-840b07af0f3c\") " pod="openshift-must-gather-7k6rg/must-gather-wxx4c"
Jul 28 16:08:54.920159 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:08:54.919977    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8j5mb\" (UniqueName: \"kubernetes.io/projected/9c058ebd-07f8-4696-8f06-840b07af0f3c-kube-api-access-8j5mb\") pod \"must-gather-wxx4c\" (UID: \"9c058ebd-07f8-4696-8f06-840b07af0f3c\") " pod="openshift-must-gather-7k6rg/must-gather-wxx4c"
Jul 28 16:08:54.935881 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:08:54.935853    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22nh9_ebf75c0d-1597-4a1c-b668-dc2a041eb35e/konnectivity-agent/0.log" limit=1024
Jul 28 16:08:54.997370 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:08:54.997337    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22nh9_ebf75c0d-1597-4a1c-b668-dc2a041eb35e/konnectivity-agent/0.log" limit=1024
Jul 28 16:08:55.020628 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:08:55.020577    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/9c058ebd-07f8-4696-8f06-840b07af0f3c-must-gather-output\") pod \"must-gather-wxx4c\" (UID: \"9c058ebd-07f8-4696-8f06-840b07af0f3c\") " pod="openshift-must-gather-7k6rg/must-gather-wxx4c"
Jul 28 16:08:55.020821 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:08:55.020646    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8j5mb\" (UniqueName: \"kubernetes.io/projected/9c058ebd-07f8-4696-8f06-840b07af0f3c-kube-api-access-8j5mb\") pod \"must-gather-wxx4c\" (UID: \"9c058ebd-07f8-4696-8f06-840b07af0f3c\") " pod="openshift-must-gather-7k6rg/must-gather-wxx4c"
Jul 28 16:08:55.020978 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:08:55.020958    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/9c058ebd-07f8-4696-8f06-840b07af0f3c-must-gather-output\") pod \"must-gather-wxx4c\" (UID: \"9c058ebd-07f8-4696-8f06-840b07af0f3c\") " pod="openshift-must-gather-7k6rg/must-gather-wxx4c"
Jul 28 16:08:55.029013 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:08:55.028993    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8j5mb\" (UniqueName: \"kubernetes.io/projected/9c058ebd-07f8-4696-8f06-840b07af0f3c-kube-api-access-8j5mb\") pod \"must-gather-wxx4c\" (UID: \"9c058ebd-07f8-4696-8f06-840b07af0f3c\") " pod="openshift-must-gather-7k6rg/must-gather-wxx4c"
Jul 28 16:08:55.188399 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:08:55.188312    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-7k6rg/must-gather-wxx4c"
Jul 28 16:08:55.316295 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:08:55.316250    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-7k6rg/must-gather-wxx4c"]
Jul 28 16:08:55.318770 ip-10-0-131-121 kubenswrapper[2568]: W0728 16:08:55.318742    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9c058ebd_07f8_4696_8f06_840b07af0f3c.slice/crio-a4d5959bec38565820b8602123fc554289ebfb487c9d198abf9d57531fb39f7d WatchSource:0}: Error finding container a4d5959bec38565820b8602123fc554289ebfb487c9d198abf9d57531fb39f7d: Status 404 returned error can't find the container with id a4d5959bec38565820b8602123fc554289ebfb487c9d198abf9d57531fb39f7d
Jul 28 16:08:55.321034 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:08:55.321018    2568 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 28 16:08:56.102545 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:08:56.102513    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-7k6rg/must-gather-wxx4c" event={"ID":"9c058ebd-07f8-4696-8f06-840b07af0f3c","Type":"ContainerStarted","Data":"a4d5959bec38565820b8602123fc554289ebfb487c9d198abf9d57531fb39f7d"}
Jul 28 16:09:02.134147 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:09:02.134105    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-7k6rg/must-gather-wxx4c" event={"ID":"9c058ebd-07f8-4696-8f06-840b07af0f3c","Type":"ContainerStarted","Data":"548b41edc71d53ff74098536b9acead87350abfcf678b11d138b03a61ad76822"}
Jul 28 16:09:02.134147 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:09:02.134150    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-7k6rg/must-gather-wxx4c" event={"ID":"9c058ebd-07f8-4696-8f06-840b07af0f3c","Type":"ContainerStarted","Data":"46dea0443a90ad4fdd384b6a7aa316d288b999172634ec2f80ad14b2e5323048"}
Jul 28 16:09:02.151056 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:09:02.151004    2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-7k6rg/must-gather-wxx4c" podStartSLOduration=1.8147312740000001 podStartE2EDuration="8.150986354s" totalImagesPullingTime="6.33625508s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 16:08:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 16:09:02.149414173 +0000 UTC m=+2122.334609405" watchObservedRunningTime="2026-07-28 16:09:02.150986354 +0000 UTC m=+2122.336181523"
Jul 28 16:09:23.231068 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:09:23.231034    2568 generic.go:356] "Generic (PLEG): container finished" podID="9c058ebd-07f8-4696-8f06-840b07af0f3c" containerID="46dea0443a90ad4fdd384b6a7aa316d288b999172634ec2f80ad14b2e5323048" exitCode=0
Jul 28 16:09:23.231506 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:09:23.231088    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-7k6rg/must-gather-wxx4c" event={"ID":"9c058ebd-07f8-4696-8f06-840b07af0f3c","Type":"ContainerDied","Data":"46dea0443a90ad4fdd384b6a7aa316d288b999172634ec2f80ad14b2e5323048"}
Jul 28 16:09:23.231506 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:09:23.231385    2568 scope.go:117] "RemoveContainer" containerID="46dea0443a90ad4fdd384b6a7aa316d288b999172634ec2f80ad14b2e5323048"
Jul 28 16:09:23.525473 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:09:23.525441    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-7k6rg_must-gather-wxx4c_9c058ebd-07f8-4696-8f06-840b07af0f3c/gather/0.log"
Jul 28 16:09:24.047645 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:09:24.047614    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22nh9_ebf75c0d-1597-4a1c-b668-dc2a041eb35e/konnectivity-agent/0.log" limit=1024
Jul 28 16:09:24.246677 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:09:24.246628    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22nh9_ebf75c0d-1597-4a1c-b668-dc2a041eb35e/konnectivity-agent/0.log" limit=1024
Jul 28 16:09:24.345455 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:09:24.345311    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22nh9_ebf75c0d-1597-4a1c-b668-dc2a041eb35e/konnectivity-agent/0.log" limit=1024
Jul 28 16:09:27.268449 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:09:27.268420    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-tr647_3ba72ab7-7ab4-4e7f-ace4-8a6b7bbff25c/global-pull-secret-syncer/0.log"
Jul 28 16:09:27.320541 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:09:27.320498    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-22nh9_ebf75c0d-1597-4a1c-b668-dc2a041eb35e/konnectivity-agent/0.log"
Jul 28 16:09:27.416063 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:09:27.416029    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-131-121.ec2.internal_0c5d5894bca76db08e7d990578b15cf0/haproxy/0.log"
Jul 28 16:09:29.026398 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:09:29.026364    2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-must-gather-7k6rg/must-gather-wxx4c"]
Jul 28 16:09:29.026830 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:09:29.026563    2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-must-gather-7k6rg/must-gather-wxx4c" podUID="9c058ebd-07f8-4696-8f06-840b07af0f3c" containerName="copy" containerID="cri-o://548b41edc71d53ff74098536b9acead87350abfcf678b11d138b03a61ad76822" gracePeriod=2
Jul 28 16:09:29.030679 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:09:29.030202    2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-must-gather-7k6rg/must-gather-wxx4c"]
Jul 28 16:09:29.256596 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:09:29.256571    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-7k6rg_must-gather-wxx4c_9c058ebd-07f8-4696-8f06-840b07af0f3c/copy/0.log"
Jul 28 16:09:29.256958 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:09:29.256938    2568 generic.go:356] "Generic (PLEG): container finished" podID="9c058ebd-07f8-4696-8f06-840b07af0f3c" containerID="548b41edc71d53ff74098536b9acead87350abfcf678b11d138b03a61ad76822" exitCode=143
Jul 28 16:09:29.257020 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:09:29.256989    2568 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a4d5959bec38565820b8602123fc554289ebfb487c9d198abf9d57531fb39f7d"
Jul 28 16:09:29.259570 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:09:29.259555    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-7k6rg_must-gather-wxx4c_9c058ebd-07f8-4696-8f06-840b07af0f3c/copy/0.log"
Jul 28 16:09:29.261036 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:09:29.261021    2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-7k6rg/must-gather-wxx4c"
Jul 28 16:09:29.263111 ip-10-0-131-121 kubenswrapper[2568]: E0728 16:09:29.263091    2568 status_manager.go:1018] "Failed to get status for pod" err="pods \"must-gather-wxx4c\" is forbidden: User \"system:node:ip-10-0-131-121.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-must-gather-7k6rg\": no relationship found between node 'ip-10-0-131-121.ec2.internal' and this object" podUID="9c058ebd-07f8-4696-8f06-840b07af0f3c" pod="openshift-must-gather-7k6rg/must-gather-wxx4c"
Jul 28 16:09:29.421547 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:09:29.421467    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/9c058ebd-07f8-4696-8f06-840b07af0f3c-must-gather-output\") pod \"9c058ebd-07f8-4696-8f06-840b07af0f3c\" (UID: \"9c058ebd-07f8-4696-8f06-840b07af0f3c\") "
Jul 28 16:09:29.421547 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:09:29.421509    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8j5mb\" (UniqueName: \"kubernetes.io/projected/9c058ebd-07f8-4696-8f06-840b07af0f3c-kube-api-access-8j5mb\") pod \"9c058ebd-07f8-4696-8f06-840b07af0f3c\" (UID: \"9c058ebd-07f8-4696-8f06-840b07af0f3c\") "
Jul 28 16:09:29.422968 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:09:29.422938    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9c058ebd-07f8-4696-8f06-840b07af0f3c-must-gather-output" (OuterVolumeSpecName: "must-gather-output") pod "9c058ebd-07f8-4696-8f06-840b07af0f3c" (UID: "9c058ebd-07f8-4696-8f06-840b07af0f3c"). InnerVolumeSpecName "must-gather-output". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 16:09:29.423676 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:09:29.423651    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9c058ebd-07f8-4696-8f06-840b07af0f3c-kube-api-access-8j5mb" (OuterVolumeSpecName: "kube-api-access-8j5mb") pod "9c058ebd-07f8-4696-8f06-840b07af0f3c" (UID: "9c058ebd-07f8-4696-8f06-840b07af0f3c"). InnerVolumeSpecName "kube-api-access-8j5mb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 16:09:29.522535 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:09:29.522497    2568 reconciler_common.go:299] "Volume detached for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/9c058ebd-07f8-4696-8f06-840b07af0f3c-must-gather-output\") on node \"ip-10-0-131-121.ec2.internal\" DevicePath \"\""
Jul 28 16:09:29.522535 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:09:29.522529    2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8j5mb\" (UniqueName: \"kubernetes.io/projected/9c058ebd-07f8-4696-8f06-840b07af0f3c-kube-api-access-8j5mb\") on node \"ip-10-0-131-121.ec2.internal\" DevicePath \"\""
Jul 28 16:09:29.835428 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:09:29.835393    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22nh9_ebf75c0d-1597-4a1c-b668-dc2a041eb35e/konnectivity-agent/0.log" limit=1024
Jul 28 16:09:30.261451 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:09:30.261373    2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-7k6rg/must-gather-wxx4c"
Jul 28 16:09:30.263820 ip-10-0-131-121 kubenswrapper[2568]: E0728 16:09:30.263793    2568 status_manager.go:1018] "Failed to get status for pod" err="pods \"must-gather-wxx4c\" is forbidden: User \"system:node:ip-10-0-131-121.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-must-gather-7k6rg\": no relationship found between node 'ip-10-0-131-121.ec2.internal' and this object" podUID="9c058ebd-07f8-4696-8f06-840b07af0f3c" pod="openshift-must-gather-7k6rg/must-gather-wxx4c"
Jul 28 16:09:30.271192 ip-10-0-131-121 kubenswrapper[2568]: E0728 16:09:30.271164    2568 status_manager.go:1018] "Failed to get status for pod" err="pods \"must-gather-wxx4c\" is forbidden: User \"system:node:ip-10-0-131-121.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-must-gather-7k6rg\": no relationship found between node 'ip-10-0-131-121.ec2.internal' and this object" podUID="9c058ebd-07f8-4696-8f06-840b07af0f3c" pod="openshift-must-gather-7k6rg/must-gather-wxx4c"
Jul 28 16:09:30.296820 ip-10-0-131-121 kubenswrapper[2568]: E0728 16:09:30.296702    2568 status_manager.go:1018] "Failed to get status for pod" err="pods \"must-gather-wxx4c\" is forbidden: User \"system:node:ip-10-0-131-121.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-must-gather-7k6rg\": no relationship found between node 'ip-10-0-131-121.ec2.internal' and this object" podUID="9c058ebd-07f8-4696-8f06-840b07af0f3c" pod="openshift-must-gather-7k6rg/must-gather-wxx4c"
Jul 28 16:09:30.300126 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:09:30.300100    2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9c058ebd-07f8-4696-8f06-840b07af0f3c" path="/var/lib/kubelet/pods/9c058ebd-07f8-4696-8f06-840b07af0f3c/volumes"
Jul 28 16:09:30.757841 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:09:30.757805    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-22nh9_ebf75c0d-1597-4a1c-b668-dc2a041eb35e/konnectivity-agent/0.log" limit=1024
Jul 28 16:09:31.404820 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:09:31.404792    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-6f489fff9f-cxft6_150c669a-1db3-4dbe-90b0-cf0f9fb4ab15/metrics-server/0.log"
Jul 28 16:09:31.564976 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:09:31.564946    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-sbtm9_eb97dc56-9614-4c55-bfad-38869c6bbbad/node-exporter/0.log"
Jul 28 16:09:31.592140 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:09:31.592114    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-sbtm9_eb97dc56-9614-4c55-bfad-38869c6bbbad/kube-rbac-proxy/0.log"
Jul 28 16:09:31.624101 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:09:31.624028    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-sbtm9_eb97dc56-9614-4c55-bfad-38869c6bbbad/init-textfile/0.log"
Jul 28 16:09:32.139514 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:09:32.139482    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-84d69f794d-662rc_e8dfa62b-ad30-49fa-a0ed-08aeed3407f8/prometheus-operator/0.log"
Jul 28 16:09:32.172492 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:09:32.172461    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-84d69f794d-662rc_e8dfa62b-ad30-49fa-a0ed-08aeed3407f8/kube-rbac-proxy/0.log"
Jul 28 16:09:32.198833 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:09:32.198798    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-7f8d78f54b-cbpwg_d918112c-b663-4576-bf8b-3225bc38edb4/prometheus-operator-admission-webhook/0.log"
Jul 28 16:09:32.232574 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:09:32.232547    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-7c86964b5b-jrqrl_45c39b24-8978-4302-910a-5a590a422a55/telemeter-client/0.log"
Jul 28 16:09:32.256829 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:09:32.256801    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-7c86964b5b-jrqrl_45c39b24-8978-4302-910a-5a590a422a55/reload/0.log"
Jul 28 16:09:32.282429 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:09:32.282389    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-7c86964b5b-jrqrl_45c39b24-8978-4302-910a-5a590a422a55/kube-rbac-proxy/0.log"
Jul 28 16:09:32.319123 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:09:32.319094    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-867fc7464c-c4h8h_29c85cac-34d3-4dae-8d46-6090c978a936/thanos-query/0.log"
Jul 28 16:09:32.341125 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:09:32.341094    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-867fc7464c-c4h8h_29c85cac-34d3-4dae-8d46-6090c978a936/kube-rbac-proxy-web/0.log"
Jul 28 16:09:32.370131 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:09:32.370101    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-867fc7464c-c4h8h_29c85cac-34d3-4dae-8d46-6090c978a936/kube-rbac-proxy/0.log"
Jul 28 16:09:32.394664 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:09:32.394579    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-867fc7464c-c4h8h_29c85cac-34d3-4dae-8d46-6090c978a936/prom-label-proxy/0.log"
Jul 28 16:09:32.419372 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:09:32.419345    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-867fc7464c-c4h8h_29c85cac-34d3-4dae-8d46-6090c978a936/kube-rbac-proxy-rules/0.log"
Jul 28 16:09:32.445649 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:09:32.445622    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-867fc7464c-c4h8h_29c85cac-34d3-4dae-8d46-6090c978a936/kube-rbac-proxy-metrics/0.log"
Jul 28 16:09:33.735709 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:09:33.735678    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-console_networking-console-plugin-74c9f884b7-cd6sf_39b0e13f-c4ec-48bd-9c67-99b1d4506463/networking-console-plugin/0.log"
Jul 28 16:09:34.708567 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:09:34.708538    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-77cd488857-w8w89_e4c44c7d-70ae-4890-81c7-939ec40e8b51/console/0.log"
Jul 28 16:09:34.746383 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:09:34.746356    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_downloads-7c97dc4555-x5gtq_1e75cd2a-7fb1-4d78-aef3-523a82c5dde6/download-server/0.log"
Jul 28 16:09:35.628038 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:09:35.628001    2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-9zm2w/perf-node-gather-daemonset-qbxv5"]
Jul 28 16:09:35.628350 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:09:35.628338    2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9c058ebd-07f8-4696-8f06-840b07af0f3c" containerName="copy"
Jul 28 16:09:35.628396 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:09:35.628351    2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="9c058ebd-07f8-4696-8f06-840b07af0f3c" containerName="copy"
Jul 28 16:09:35.628396 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:09:35.628364    2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9c058ebd-07f8-4696-8f06-840b07af0f3c" containerName="gather"
Jul 28 16:09:35.628396 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:09:35.628370    2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="9c058ebd-07f8-4696-8f06-840b07af0f3c" containerName="gather"
Jul 28 16:09:35.628491 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:09:35.628417    2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="9c058ebd-07f8-4696-8f06-840b07af0f3c" containerName="gather"
Jul 28 16:09:35.628491 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:09:35.628425    2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="9c058ebd-07f8-4696-8f06-840b07af0f3c" containerName="copy"
Jul 28 16:09:35.631310 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:09:35.631289    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-9zm2w/perf-node-gather-daemonset-qbxv5"
Jul 28 16:09:35.633716 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:09:35.633695    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-9zm2w\"/\"openshift-service-ca.crt\""
Jul 28 16:09:35.633716 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:09:35.633715    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-9zm2w\"/\"default-dockercfg-l8dtj\""
Jul 28 16:09:35.634848 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:09:35.634818    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-9zm2w\"/\"kube-root-ca.crt\""
Jul 28 16:09:35.641761 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:09:35.641735    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-9zm2w/perf-node-gather-daemonset-qbxv5"]
Jul 28 16:09:35.772030 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:09:35.771972    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b8469312-09f6-486a-ad0e-bf4f2055c865-sys\") pod \"perf-node-gather-daemonset-qbxv5\" (UID: \"b8469312-09f6-486a-ad0e-bf4f2055c865\") " pod="openshift-must-gather-9zm2w/perf-node-gather-daemonset-qbxv5"
Jul 28 16:09:35.772439 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:09:35.772074    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nmtvr\" (UniqueName: \"kubernetes.io/projected/b8469312-09f6-486a-ad0e-bf4f2055c865-kube-api-access-nmtvr\") pod \"perf-node-gather-daemonset-qbxv5\" (UID: \"b8469312-09f6-486a-ad0e-bf4f2055c865\") " pod="openshift-must-gather-9zm2w/perf-node-gather-daemonset-qbxv5"
Jul 28 16:09:35.772439 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:09:35.772147    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/b8469312-09f6-486a-ad0e-bf4f2055c865-podres\") pod \"perf-node-gather-daemonset-qbxv5\" (UID: \"b8469312-09f6-486a-ad0e-bf4f2055c865\") " pod="openshift-must-gather-9zm2w/perf-node-gather-daemonset-qbxv5"
Jul 28 16:09:35.772439 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:09:35.772170    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/b8469312-09f6-486a-ad0e-bf4f2055c865-proc\") pod \"perf-node-gather-daemonset-qbxv5\" (UID: \"b8469312-09f6-486a-ad0e-bf4f2055c865\") " pod="openshift-must-gather-9zm2w/perf-node-gather-daemonset-qbxv5"
Jul 28 16:09:35.772439 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:09:35.772201    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/b8469312-09f6-486a-ad0e-bf4f2055c865-lib-modules\") pod \"perf-node-gather-daemonset-qbxv5\" (UID: \"b8469312-09f6-486a-ad0e-bf4f2055c865\") " pod="openshift-must-gather-9zm2w/perf-node-gather-daemonset-qbxv5"
Jul 28 16:09:35.872700 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:09:35.872659    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nmtvr\" (UniqueName: \"kubernetes.io/projected/b8469312-09f6-486a-ad0e-bf4f2055c865-kube-api-access-nmtvr\") pod \"perf-node-gather-daemonset-qbxv5\" (UID: \"b8469312-09f6-486a-ad0e-bf4f2055c865\") " pod="openshift-must-gather-9zm2w/perf-node-gather-daemonset-qbxv5"
Jul 28 16:09:35.872935 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:09:35.872737    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/b8469312-09f6-486a-ad0e-bf4f2055c865-podres\") pod \"perf-node-gather-daemonset-qbxv5\" (UID: \"b8469312-09f6-486a-ad0e-bf4f2055c865\") " pod="openshift-must-gather-9zm2w/perf-node-gather-daemonset-qbxv5"
Jul 28 16:09:35.872935 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:09:35.872764    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/b8469312-09f6-486a-ad0e-bf4f2055c865-proc\") pod \"perf-node-gather-daemonset-qbxv5\" (UID: \"b8469312-09f6-486a-ad0e-bf4f2055c865\") " pod="openshift-must-gather-9zm2w/perf-node-gather-daemonset-qbxv5"
Jul 28 16:09:35.872935 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:09:35.872826    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/b8469312-09f6-486a-ad0e-bf4f2055c865-lib-modules\") pod \"perf-node-gather-daemonset-qbxv5\" (UID: \"b8469312-09f6-486a-ad0e-bf4f2055c865\") " pod="openshift-must-gather-9zm2w/perf-node-gather-daemonset-qbxv5"
Jul 28 16:09:35.872935 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:09:35.872877    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b8469312-09f6-486a-ad0e-bf4f2055c865-sys\") pod \"perf-node-gather-daemonset-qbxv5\" (UID: \"b8469312-09f6-486a-ad0e-bf4f2055c865\") " pod="openshift-must-gather-9zm2w/perf-node-gather-daemonset-qbxv5"
Jul 28 16:09:35.873124 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:09:35.872935    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/b8469312-09f6-486a-ad0e-bf4f2055c865-podres\") pod \"perf-node-gather-daemonset-qbxv5\" (UID: \"b8469312-09f6-486a-ad0e-bf4f2055c865\") " pod="openshift-must-gather-9zm2w/perf-node-gather-daemonset-qbxv5"
Jul 28 16:09:35.873124 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:09:35.872948    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b8469312-09f6-486a-ad0e-bf4f2055c865-sys\") pod \"perf-node-gather-daemonset-qbxv5\" (UID: \"b8469312-09f6-486a-ad0e-bf4f2055c865\") " pod="openshift-must-gather-9zm2w/perf-node-gather-daemonset-qbxv5"
Jul 28 16:09:35.873124 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:09:35.872939    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/b8469312-09f6-486a-ad0e-bf4f2055c865-proc\") pod \"perf-node-gather-daemonset-qbxv5\" (UID: \"b8469312-09f6-486a-ad0e-bf4f2055c865\") " pod="openshift-must-gather-9zm2w/perf-node-gather-daemonset-qbxv5"
Jul 28 16:09:35.873124 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:09:35.873026    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/b8469312-09f6-486a-ad0e-bf4f2055c865-lib-modules\") pod \"perf-node-gather-daemonset-qbxv5\" (UID: \"b8469312-09f6-486a-ad0e-bf4f2055c865\") " pod="openshift-must-gather-9zm2w/perf-node-gather-daemonset-qbxv5"
Jul 28 16:09:35.881853 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:09:35.881771    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nmtvr\" (UniqueName: \"kubernetes.io/projected/b8469312-09f6-486a-ad0e-bf4f2055c865-kube-api-access-nmtvr\") pod \"perf-node-gather-daemonset-qbxv5\" (UID: \"b8469312-09f6-486a-ad0e-bf4f2055c865\") " pod="openshift-must-gather-9zm2w/perf-node-gather-daemonset-qbxv5"
Jul 28 16:09:35.941978 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:09:35.941939    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-9zm2w/perf-node-gather-daemonset-qbxv5"
Jul 28 16:09:36.085893 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:09:36.085855    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-9zm2w/perf-node-gather-daemonset-qbxv5"]
Jul 28 16:09:36.088720 ip-10-0-131-121 kubenswrapper[2568]: W0728 16:09:36.088682    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podb8469312_09f6_486a_ad0e_bf4f2055c865.slice/crio-ba74e8654c8dfae59194e95331d16cd00aa46ea205097c3402c99e4151b45390 WatchSource:0}: Error finding container ba74e8654c8dfae59194e95331d16cd00aa46ea205097c3402c99e4151b45390: Status 404 returned error can't find the container with id ba74e8654c8dfae59194e95331d16cd00aa46ea205097c3402c99e4151b45390
Jul 28 16:09:36.248399 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:09:36.248370    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-2m6ls_11c0aa38-e5f8-4e7d-9003-755ac2ba3a6c/dns/0.log"
Jul 28 16:09:36.282166 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:09:36.282129    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-2m6ls_11c0aa38-e5f8-4e7d-9003-755ac2ba3a6c/kube-rbac-proxy/0.log"
Jul 28 16:09:36.287486 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:09:36.287457    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-9zm2w/perf-node-gather-daemonset-qbxv5" event={"ID":"b8469312-09f6-486a-ad0e-bf4f2055c865","Type":"ContainerStarted","Data":"e8b721169dc03b4a53d98b253f3fdfb890df6b57ff26a3ef05edfa999078ad3d"}
Jul 28 16:09:36.287626 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:09:36.287517    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-9zm2w/perf-node-gather-daemonset-qbxv5" event={"ID":"b8469312-09f6-486a-ad0e-bf4f2055c865","Type":"ContainerStarted","Data":"ba74e8654c8dfae59194e95331d16cd00aa46ea205097c3402c99e4151b45390"}
Jul 28 16:09:36.287689 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:09:36.287628    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-9zm2w/perf-node-gather-daemonset-qbxv5"
Jul 28 16:09:36.306212 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:09:36.306158    2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-9zm2w/perf-node-gather-daemonset-qbxv5" podStartSLOduration=1.306140153 podStartE2EDuration="1.306140153s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 16:09:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 16:09:36.304933223 +0000 UTC m=+2156.490128393" watchObservedRunningTime="2026-07-28 16:09:36.306140153 +0000 UTC m=+2156.491335327"
Jul 28 16:09:36.475304 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:09:36.475273    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-whdfr_fab92b7b-ed62-4783-86be-ed4bd56a3128/dns-node-resolver/0.log"
Jul 28 16:09:37.025396 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:09:37.025362    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-65444944d6-t5hmt_8a2e179f-a54d-480c-84f2-d7278c611f86/registry/0.log"
Jul 28 16:09:37.097576 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:09:37.097550    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-9hznr_53a1cbf6-fc45-47f6-a8b1-5f95ec295587/node-ca/0.log"
Jul 28 16:09:38.173500 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:09:38.173468    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-8gpg6_4ef2e055-ee39-422e-9cd0-5c7e2e29c7c1/serve-healthcheck-canary/0.log"
Jul 28 16:09:38.804070 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:09:38.804039    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-rsd4q_45da22e1-42e5-4dcf-b132-105b7b343538/kube-rbac-proxy/0.log"
Jul 28 16:09:38.826364 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:09:38.826334    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-rsd4q_45da22e1-42e5-4dcf-b132-105b7b343538/exporter/0.log"
Jul 28 16:09:38.849199 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:09:38.849170    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-rsd4q_45da22e1-42e5-4dcf-b132-105b7b343538/extractor/0.log"
Jul 28 16:09:41.083712 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:09:41.083680    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_llmisvc-controller-manager-84fc959947-wmbcg_60525da1-3d0a-4186-be33-0f94468ebc99/manager/0.log"
Jul 28 16:09:41.104459 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:09:41.104430    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_model-serving-api-7b954ffc95-p9c4h_5f5b50cc-e574-4f74-b27e-0e5c35923b80/server/0.log"
Jul 28 16:09:42.305691 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:09:42.305658    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-9zm2w/perf-node-gather-daemonset-qbxv5"
Jul 28 16:09:45.361312 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:09:45.361262    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-6865c9967-xwgqx_b940bfbc-30c7-4eff-9501-484fa354d3b5/migrator/0.log"
Jul 28 16:09:45.388128 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:09:45.388097    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-6865c9967-xwgqx_b940bfbc-30c7-4eff-9501-484fa354d3b5/graceful-termination/0.log"
Jul 28 16:09:46.658245 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:09:46.658207    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-ptrdx_dcab1041-8d3b-42e1-af05-55e847b831ad/kube-multus-additional-cni-plugins/0.log"
Jul 28 16:09:46.682079 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:09:46.682053    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-ptrdx_dcab1041-8d3b-42e1-af05-55e847b831ad/egress-router-binary-copy/0.log"
Jul 28 16:09:46.708993 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:09:46.708964    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-ptrdx_dcab1041-8d3b-42e1-af05-55e847b831ad/cni-plugins/0.log"
Jul 28 16:09:46.733199 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:09:46.733175    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-ptrdx_dcab1041-8d3b-42e1-af05-55e847b831ad/bond-cni-plugin/0.log"
Jul 28 16:09:46.756092 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:09:46.756062    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-ptrdx_dcab1041-8d3b-42e1-af05-55e847b831ad/routeoverride-cni/0.log"
Jul 28 16:09:46.782224 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:09:46.782198    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-ptrdx_dcab1041-8d3b-42e1-af05-55e847b831ad/whereabouts-cni-bincopy/0.log"
Jul 28 16:09:46.808030 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:09:46.808004    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-ptrdx_dcab1041-8d3b-42e1-af05-55e847b831ad/whereabouts-cni/0.log"
Jul 28 16:09:47.264607 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:09:47.264582    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-nvbhc_7e2caef2-3492-48d3-adbd-89fd3d4494d9/kube-multus/0.log"
Jul 28 16:09:47.362793 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:09:47.362728    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-ckpr5_6100b515-cfe9-47ef-b672-d698ae677bd0/network-metrics-daemon/0.log"
Jul 28 16:09:47.384130 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:09:47.384093    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-ckpr5_6100b515-cfe9-47ef-b672-d698ae677bd0/kube-rbac-proxy/0.log"
Jul 28 16:09:48.164268 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:09:48.164237    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2vrz4_3622f792-27df-4604-a1e3-47192c7302cb/ovn-controller/0.log"
Jul 28 16:09:48.182728 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:09:48.182698    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2vrz4_3622f792-27df-4604-a1e3-47192c7302cb/ovn-acl-logging/0.log"
Jul 28 16:09:48.207373 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:09:48.207338    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2vrz4_3622f792-27df-4604-a1e3-47192c7302cb/kube-rbac-proxy-node/0.log"
Jul 28 16:09:48.234854 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:09:48.234818    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2vrz4_3622f792-27df-4604-a1e3-47192c7302cb/kube-rbac-proxy-ovn-metrics/0.log"
Jul 28 16:09:48.257633 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:09:48.257588    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2vrz4_3622f792-27df-4604-a1e3-47192c7302cb/northd/0.log"
Jul 28 16:09:48.279030 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:09:48.279000    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2vrz4_3622f792-27df-4604-a1e3-47192c7302cb/nbdb/0.log"
Jul 28 16:09:48.299344 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:09:48.299318    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2vrz4_3622f792-27df-4604-a1e3-47192c7302cb/sbdb/0.log"
Jul 28 16:09:48.346167 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:09:48.346122    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2vrz4_3622f792-27df-4604-a1e3-47192c7302cb/ovnkube-controller/0.log"
Jul 28 16:09:49.816227 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:09:49.816199    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-diagnostics_network-check-target-85ncq_644d231e-7d3c-4aef-a201-87266e83d526/network-check-target-container/0.log"
Jul 28 16:09:50.712979 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:09:50.712950    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-operator_iptables-alerter-49vz8_de4142bb-fcb0-460c-9fbf-1025c61e2015/iptables-alerter/0.log"
Jul 28 16:09:51.463605 ip-10-0-131-121 kubenswrapper[2568]: I0728 16:09:51.463576    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-node-tuning-operator_tuned-n2w6d_ad8ebf58-b767-4341-b60e-868c5b8b637f/tuned/0.log"