Jul 27 21:04:31.299580 ip-10-0-131-133 systemd[1]: Starting Kubernetes Kubelet...
Jul 27 21:04:31.744550 ip-10-0-131-133 kubenswrapper[2559]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 27 21:04:31.744550 ip-10-0-131-133 kubenswrapper[2559]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Jul 27 21:04:31.744550 ip-10-0-131-133 kubenswrapper[2559]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 27 21:04:31.744550 ip-10-0-131-133 kubenswrapper[2559]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Jul 27 21:04:31.744550 ip-10-0-131-133 kubenswrapper[2559]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 27 21:04:31.746927 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:04:31.746731    2559 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Jul 27 21:04:31.750800 ip-10-0-131-133 kubenswrapper[2559]: W0727 21:04:31.750780    2559 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 27 21:04:31.750800 ip-10-0-131-133 kubenswrapper[2559]: W0727 21:04:31.750802    2559 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 27 21:04:31.750883 ip-10-0-131-133 kubenswrapper[2559]: W0727 21:04:31.750807    2559 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 27 21:04:31.750883 ip-10-0-131-133 kubenswrapper[2559]: W0727 21:04:31.750811    2559 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 27 21:04:31.750883 ip-10-0-131-133 kubenswrapper[2559]: W0727 21:04:31.750814    2559 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 27 21:04:31.750883 ip-10-0-131-133 kubenswrapper[2559]: W0727 21:04:31.750818    2559 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 27 21:04:31.750883 ip-10-0-131-133 kubenswrapper[2559]: W0727 21:04:31.750821    2559 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 27 21:04:31.750883 ip-10-0-131-133 kubenswrapper[2559]: W0727 21:04:31.750824    2559 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 27 21:04:31.750883 ip-10-0-131-133 kubenswrapper[2559]: W0727 21:04:31.750827    2559 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 27 21:04:31.750883 ip-10-0-131-133 kubenswrapper[2559]: W0727 21:04:31.750830    2559 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 27 21:04:31.750883 ip-10-0-131-133 kubenswrapper[2559]: W0727 21:04:31.750832    2559 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 27 21:04:31.750883 ip-10-0-131-133 kubenswrapper[2559]: W0727 21:04:31.750835    2559 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 27 21:04:31.750883 ip-10-0-131-133 kubenswrapper[2559]: W0727 21:04:31.750838    2559 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 27 21:04:31.750883 ip-10-0-131-133 kubenswrapper[2559]: W0727 21:04:31.750841    2559 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 27 21:04:31.750883 ip-10-0-131-133 kubenswrapper[2559]: W0727 21:04:31.750847    2559 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 27 21:04:31.750883 ip-10-0-131-133 kubenswrapper[2559]: W0727 21:04:31.750850    2559 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 27 21:04:31.750883 ip-10-0-131-133 kubenswrapper[2559]: W0727 21:04:31.750854    2559 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 27 21:04:31.750883 ip-10-0-131-133 kubenswrapper[2559]: W0727 21:04:31.750857    2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 27 21:04:31.750883 ip-10-0-131-133 kubenswrapper[2559]: W0727 21:04:31.750860    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 27 21:04:31.750883 ip-10-0-131-133 kubenswrapper[2559]: W0727 21:04:31.750863    2559 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 27 21:04:31.750883 ip-10-0-131-133 kubenswrapper[2559]: W0727 21:04:31.750865    2559 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 27 21:04:31.750883 ip-10-0-131-133 kubenswrapper[2559]: W0727 21:04:31.750869    2559 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 27 21:04:31.751425 ip-10-0-131-133 kubenswrapper[2559]: W0727 21:04:31.750872    2559 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 27 21:04:31.751425 ip-10-0-131-133 kubenswrapper[2559]: W0727 21:04:31.750875    2559 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 27 21:04:31.751425 ip-10-0-131-133 kubenswrapper[2559]: W0727 21:04:31.750877    2559 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 27 21:04:31.751425 ip-10-0-131-133 kubenswrapper[2559]: W0727 21:04:31.750880    2559 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 27 21:04:31.751425 ip-10-0-131-133 kubenswrapper[2559]: W0727 21:04:31.750883    2559 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 27 21:04:31.751425 ip-10-0-131-133 kubenswrapper[2559]: W0727 21:04:31.750886    2559 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 27 21:04:31.751425 ip-10-0-131-133 kubenswrapper[2559]: W0727 21:04:31.750889    2559 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 27 21:04:31.751425 ip-10-0-131-133 kubenswrapper[2559]: W0727 21:04:31.750892    2559 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 27 21:04:31.751425 ip-10-0-131-133 kubenswrapper[2559]: W0727 21:04:31.750894    2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 27 21:04:31.751425 ip-10-0-131-133 kubenswrapper[2559]: W0727 21:04:31.750897    2559 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 27 21:04:31.751425 ip-10-0-131-133 kubenswrapper[2559]: W0727 21:04:31.750900    2559 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 27 21:04:31.751425 ip-10-0-131-133 kubenswrapper[2559]: W0727 21:04:31.750903    2559 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 27 21:04:31.751425 ip-10-0-131-133 kubenswrapper[2559]: W0727 21:04:31.750905    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 27 21:04:31.751425 ip-10-0-131-133 kubenswrapper[2559]: W0727 21:04:31.750908    2559 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 27 21:04:31.751425 ip-10-0-131-133 kubenswrapper[2559]: W0727 21:04:31.750911    2559 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 27 21:04:31.751425 ip-10-0-131-133 kubenswrapper[2559]: W0727 21:04:31.750914    2559 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 27 21:04:31.751425 ip-10-0-131-133 kubenswrapper[2559]: W0727 21:04:31.750916    2559 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 27 21:04:31.751425 ip-10-0-131-133 kubenswrapper[2559]: W0727 21:04:31.750920    2559 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 27 21:04:31.751425 ip-10-0-131-133 kubenswrapper[2559]: W0727 21:04:31.750922    2559 feature_gate.go:328] unrecognized feature gate: Example2
Jul 27 21:04:31.751425 ip-10-0-131-133 kubenswrapper[2559]: W0727 21:04:31.750926    2559 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 27 21:04:31.751962 ip-10-0-131-133 kubenswrapper[2559]: W0727 21:04:31.750929    2559 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 27 21:04:31.751962 ip-10-0-131-133 kubenswrapper[2559]: W0727 21:04:31.750931    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 27 21:04:31.751962 ip-10-0-131-133 kubenswrapper[2559]: W0727 21:04:31.750934    2559 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 27 21:04:31.751962 ip-10-0-131-133 kubenswrapper[2559]: W0727 21:04:31.750936    2559 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 27 21:04:31.751962 ip-10-0-131-133 kubenswrapper[2559]: W0727 21:04:31.750939    2559 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 27 21:04:31.751962 ip-10-0-131-133 kubenswrapper[2559]: W0727 21:04:31.750942    2559 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 27 21:04:31.751962 ip-10-0-131-133 kubenswrapper[2559]: W0727 21:04:31.750945    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 27 21:04:31.751962 ip-10-0-131-133 kubenswrapper[2559]: W0727 21:04:31.750947    2559 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 27 21:04:31.751962 ip-10-0-131-133 kubenswrapper[2559]: W0727 21:04:31.750950    2559 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 27 21:04:31.751962 ip-10-0-131-133 kubenswrapper[2559]: W0727 21:04:31.750953    2559 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 27 21:04:31.751962 ip-10-0-131-133 kubenswrapper[2559]: W0727 21:04:31.750955    2559 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 27 21:04:31.751962 ip-10-0-131-133 kubenswrapper[2559]: W0727 21:04:31.750958    2559 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 27 21:04:31.751962 ip-10-0-131-133 kubenswrapper[2559]: W0727 21:04:31.750968    2559 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 27 21:04:31.751962 ip-10-0-131-133 kubenswrapper[2559]: W0727 21:04:31.750975    2559 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 27 21:04:31.751962 ip-10-0-131-133 kubenswrapper[2559]: W0727 21:04:31.750978    2559 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 27 21:04:31.751962 ip-10-0-131-133 kubenswrapper[2559]: W0727 21:04:31.750982    2559 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 27 21:04:31.751962 ip-10-0-131-133 kubenswrapper[2559]: W0727 21:04:31.750984    2559 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 27 21:04:31.751962 ip-10-0-131-133 kubenswrapper[2559]: W0727 21:04:31.750987    2559 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 27 21:04:31.751962 ip-10-0-131-133 kubenswrapper[2559]: W0727 21:04:31.750990    2559 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 27 21:04:31.751962 ip-10-0-131-133 kubenswrapper[2559]: W0727 21:04:31.750993    2559 feature_gate.go:328] unrecognized feature gate: Example
Jul 27 21:04:31.752525 ip-10-0-131-133 kubenswrapper[2559]: W0727 21:04:31.750995    2559 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 27 21:04:31.752525 ip-10-0-131-133 kubenswrapper[2559]: W0727 21:04:31.750998    2559 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 27 21:04:31.752525 ip-10-0-131-133 kubenswrapper[2559]: W0727 21:04:31.751001    2559 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 27 21:04:31.752525 ip-10-0-131-133 kubenswrapper[2559]: W0727 21:04:31.751003    2559 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 27 21:04:31.752525 ip-10-0-131-133 kubenswrapper[2559]: W0727 21:04:31.751006    2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 27 21:04:31.752525 ip-10-0-131-133 kubenswrapper[2559]: W0727 21:04:31.751009    2559 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 27 21:04:31.752525 ip-10-0-131-133 kubenswrapper[2559]: W0727 21:04:31.751011    2559 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 27 21:04:31.752525 ip-10-0-131-133 kubenswrapper[2559]: W0727 21:04:31.751014    2559 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 27 21:04:31.752525 ip-10-0-131-133 kubenswrapper[2559]: W0727 21:04:31.751017    2559 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 27 21:04:31.752525 ip-10-0-131-133 kubenswrapper[2559]: W0727 21:04:31.751021    2559 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 27 21:04:31.752525 ip-10-0-131-133 kubenswrapper[2559]: W0727 21:04:31.751023    2559 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 27 21:04:31.752525 ip-10-0-131-133 kubenswrapper[2559]: W0727 21:04:31.751026    2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 27 21:04:31.752525 ip-10-0-131-133 kubenswrapper[2559]: W0727 21:04:31.751029    2559 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 27 21:04:31.752525 ip-10-0-131-133 kubenswrapper[2559]: W0727 21:04:31.751032    2559 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 27 21:04:31.752525 ip-10-0-131-133 kubenswrapper[2559]: W0727 21:04:31.751034    2559 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 27 21:04:31.752525 ip-10-0-131-133 kubenswrapper[2559]: W0727 21:04:31.751037    2559 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 27 21:04:31.752525 ip-10-0-131-133 kubenswrapper[2559]: W0727 21:04:31.751040    2559 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 27 21:04:31.752525 ip-10-0-131-133 kubenswrapper[2559]: W0727 21:04:31.751042    2559 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 27 21:04:31.752525 ip-10-0-131-133 kubenswrapper[2559]: W0727 21:04:31.751045    2559 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 27 21:04:31.753077 ip-10-0-131-133 kubenswrapper[2559]: W0727 21:04:31.751049    2559 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 27 21:04:31.753077 ip-10-0-131-133 kubenswrapper[2559]: W0727 21:04:31.751053    2559 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 27 21:04:31.753077 ip-10-0-131-133 kubenswrapper[2559]: W0727 21:04:31.751056    2559 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 27 21:04:31.753077 ip-10-0-131-133 kubenswrapper[2559]: W0727 21:04:31.751060    2559 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 27 21:04:31.753077 ip-10-0-131-133 kubenswrapper[2559]: W0727 21:04:31.751063    2559 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 27 21:04:31.753077 ip-10-0-131-133 kubenswrapper[2559]: W0727 21:04:31.751065    2559 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 27 21:04:31.753077 ip-10-0-131-133 kubenswrapper[2559]: W0727 21:04:31.751069    2559 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 27 21:04:31.753077 ip-10-0-131-133 kubenswrapper[2559]: W0727 21:04:31.751073    2559 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 27 21:04:31.753077 ip-10-0-131-133 kubenswrapper[2559]: W0727 21:04:31.751077    2559 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 27 21:04:31.753077 ip-10-0-131-133 kubenswrapper[2559]: W0727 21:04:31.751080    2559 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 27 21:04:31.753077 ip-10-0-131-133 kubenswrapper[2559]: W0727 21:04:31.751083    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 27 21:04:31.753077 ip-10-0-131-133 kubenswrapper[2559]: W0727 21:04:31.751085    2559 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 27 21:04:31.753077 ip-10-0-131-133 kubenswrapper[2559]: W0727 21:04:31.751088    2559 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 27 21:04:31.753077 ip-10-0-131-133 kubenswrapper[2559]: W0727 21:04:31.751091    2559 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 27 21:04:31.753077 ip-10-0-131-133 kubenswrapper[2559]: W0727 21:04:31.751094    2559 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 27 21:04:31.753077 ip-10-0-131-133 kubenswrapper[2559]: W0727 21:04:31.751097    2559 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 27 21:04:31.753077 ip-10-0-131-133 kubenswrapper[2559]: W0727 21:04:31.751099    2559 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 27 21:04:31.753077 ip-10-0-131-133 kubenswrapper[2559]: W0727 21:04:31.751102    2559 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 27 21:04:31.753077 ip-10-0-131-133 kubenswrapper[2559]: W0727 21:04:31.751921    2559 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 27 21:04:31.753649 ip-10-0-131-133 kubenswrapper[2559]: W0727 21:04:31.751929    2559 feature_gate.go:328] unrecognized feature gate: Example
Jul 27 21:04:31.753649 ip-10-0-131-133 kubenswrapper[2559]: W0727 21:04:31.751933    2559 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 27 21:04:31.753649 ip-10-0-131-133 kubenswrapper[2559]: W0727 21:04:31.751936    2559 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 27 21:04:31.753649 ip-10-0-131-133 kubenswrapper[2559]: W0727 21:04:31.751939    2559 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 27 21:04:31.753649 ip-10-0-131-133 kubenswrapper[2559]: W0727 21:04:31.751943    2559 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 27 21:04:31.753649 ip-10-0-131-133 kubenswrapper[2559]: W0727 21:04:31.751948    2559 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 27 21:04:31.753649 ip-10-0-131-133 kubenswrapper[2559]: W0727 21:04:31.751952    2559 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 27 21:04:31.753649 ip-10-0-131-133 kubenswrapper[2559]: W0727 21:04:31.751955    2559 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 27 21:04:31.753649 ip-10-0-131-133 kubenswrapper[2559]: W0727 21:04:31.751958    2559 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 27 21:04:31.753649 ip-10-0-131-133 kubenswrapper[2559]: W0727 21:04:31.751961    2559 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 27 21:04:31.753649 ip-10-0-131-133 kubenswrapper[2559]: W0727 21:04:31.751964    2559 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 27 21:04:31.753649 ip-10-0-131-133 kubenswrapper[2559]: W0727 21:04:31.751967    2559 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 27 21:04:31.753649 ip-10-0-131-133 kubenswrapper[2559]: W0727 21:04:31.751970    2559 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 27 21:04:31.753649 ip-10-0-131-133 kubenswrapper[2559]: W0727 21:04:31.751973    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 27 21:04:31.753649 ip-10-0-131-133 kubenswrapper[2559]: W0727 21:04:31.751976    2559 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 27 21:04:31.753649 ip-10-0-131-133 kubenswrapper[2559]: W0727 21:04:31.751979    2559 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 27 21:04:31.753649 ip-10-0-131-133 kubenswrapper[2559]: W0727 21:04:31.751982    2559 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 27 21:04:31.753649 ip-10-0-131-133 kubenswrapper[2559]: W0727 21:04:31.751985    2559 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 27 21:04:31.753649 ip-10-0-131-133 kubenswrapper[2559]: W0727 21:04:31.751988    2559 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 27 21:04:31.753649 ip-10-0-131-133 kubenswrapper[2559]: W0727 21:04:31.751992    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 27 21:04:31.754196 ip-10-0-131-133 kubenswrapper[2559]: W0727 21:04:31.751995    2559 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 27 21:04:31.754196 ip-10-0-131-133 kubenswrapper[2559]: W0727 21:04:31.751999    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 27 21:04:31.754196 ip-10-0-131-133 kubenswrapper[2559]: W0727 21:04:31.752001    2559 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 27 21:04:31.754196 ip-10-0-131-133 kubenswrapper[2559]: W0727 21:04:31.752005    2559 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 27 21:04:31.754196 ip-10-0-131-133 kubenswrapper[2559]: W0727 21:04:31.752007    2559 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 27 21:04:31.754196 ip-10-0-131-133 kubenswrapper[2559]: W0727 21:04:31.752011    2559 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 27 21:04:31.754196 ip-10-0-131-133 kubenswrapper[2559]: W0727 21:04:31.752014    2559 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 27 21:04:31.754196 ip-10-0-131-133 kubenswrapper[2559]: W0727 21:04:31.752016    2559 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 27 21:04:31.754196 ip-10-0-131-133 kubenswrapper[2559]: W0727 21:04:31.752019    2559 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 27 21:04:31.754196 ip-10-0-131-133 kubenswrapper[2559]: W0727 21:04:31.752022    2559 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 27 21:04:31.754196 ip-10-0-131-133 kubenswrapper[2559]: W0727 21:04:31.752024    2559 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 27 21:04:31.754196 ip-10-0-131-133 kubenswrapper[2559]: W0727 21:04:31.752028    2559 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 27 21:04:31.754196 ip-10-0-131-133 kubenswrapper[2559]: W0727 21:04:31.752030    2559 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 27 21:04:31.754196 ip-10-0-131-133 kubenswrapper[2559]: W0727 21:04:31.752033    2559 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 27 21:04:31.754196 ip-10-0-131-133 kubenswrapper[2559]: W0727 21:04:31.752036    2559 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 27 21:04:31.754196 ip-10-0-131-133 kubenswrapper[2559]: W0727 21:04:31.752038    2559 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 27 21:04:31.754196 ip-10-0-131-133 kubenswrapper[2559]: W0727 21:04:31.752041    2559 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 27 21:04:31.754196 ip-10-0-131-133 kubenswrapper[2559]: W0727 21:04:31.752044    2559 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 27 21:04:31.754196 ip-10-0-131-133 kubenswrapper[2559]: W0727 21:04:31.752047    2559 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 27 21:04:31.754196 ip-10-0-131-133 kubenswrapper[2559]: W0727 21:04:31.752050    2559 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 27 21:04:31.754755 ip-10-0-131-133 kubenswrapper[2559]: W0727 21:04:31.752052    2559 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 27 21:04:31.754755 ip-10-0-131-133 kubenswrapper[2559]: W0727 21:04:31.752055    2559 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 27 21:04:31.754755 ip-10-0-131-133 kubenswrapper[2559]: W0727 21:04:31.752058    2559 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 27 21:04:31.754755 ip-10-0-131-133 kubenswrapper[2559]: W0727 21:04:31.752061    2559 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 27 21:04:31.754755 ip-10-0-131-133 kubenswrapper[2559]: W0727 21:04:31.752064    2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 27 21:04:31.754755 ip-10-0-131-133 kubenswrapper[2559]: W0727 21:04:31.752067    2559 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 27 21:04:31.754755 ip-10-0-131-133 kubenswrapper[2559]: W0727 21:04:31.752069    2559 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 27 21:04:31.754755 ip-10-0-131-133 kubenswrapper[2559]: W0727 21:04:31.752072    2559 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 27 21:04:31.754755 ip-10-0-131-133 kubenswrapper[2559]: W0727 21:04:31.752075    2559 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 27 21:04:31.754755 ip-10-0-131-133 kubenswrapper[2559]: W0727 21:04:31.752078    2559 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 27 21:04:31.754755 ip-10-0-131-133 kubenswrapper[2559]: W0727 21:04:31.752081    2559 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 27 21:04:31.754755 ip-10-0-131-133 kubenswrapper[2559]: W0727 21:04:31.752084    2559 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 27 21:04:31.754755 ip-10-0-131-133 kubenswrapper[2559]: W0727 21:04:31.752087    2559 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 27 21:04:31.754755 ip-10-0-131-133 kubenswrapper[2559]: W0727 21:04:31.752089    2559 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 27 21:04:31.754755 ip-10-0-131-133 kubenswrapper[2559]: W0727 21:04:31.752094    2559 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 27 21:04:31.754755 ip-10-0-131-133 kubenswrapper[2559]: W0727 21:04:31.752098    2559 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 27 21:04:31.754755 ip-10-0-131-133 kubenswrapper[2559]: W0727 21:04:31.752102    2559 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 27 21:04:31.754755 ip-10-0-131-133 kubenswrapper[2559]: W0727 21:04:31.752108    2559 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 27 21:04:31.754755 ip-10-0-131-133 kubenswrapper[2559]: W0727 21:04:31.752111    2559 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 27 21:04:31.755299 ip-10-0-131-133 kubenswrapper[2559]: W0727 21:04:31.752114    2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 27 21:04:31.755299 ip-10-0-131-133 kubenswrapper[2559]: W0727 21:04:31.752117    2559 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 27 21:04:31.755299 ip-10-0-131-133 kubenswrapper[2559]: W0727 21:04:31.752120    2559 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 27 21:04:31.755299 ip-10-0-131-133 kubenswrapper[2559]: W0727 21:04:31.752123    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 27 21:04:31.755299 ip-10-0-131-133 kubenswrapper[2559]: W0727 21:04:31.752126    2559 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 27 21:04:31.755299 ip-10-0-131-133 kubenswrapper[2559]: W0727 21:04:31.752130    2559 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 27 21:04:31.755299 ip-10-0-131-133 kubenswrapper[2559]: W0727 21:04:31.752132    2559 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 27 21:04:31.755299 ip-10-0-131-133 kubenswrapper[2559]: W0727 21:04:31.752135    2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 27 21:04:31.755299 ip-10-0-131-133 kubenswrapper[2559]: W0727 21:04:31.752138    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 27 21:04:31.755299 ip-10-0-131-133 kubenswrapper[2559]: W0727 21:04:31.752141    2559 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 27 21:04:31.755299 ip-10-0-131-133 kubenswrapper[2559]: W0727 21:04:31.752144    2559 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 27 21:04:31.755299 ip-10-0-131-133 kubenswrapper[2559]: W0727 21:04:31.752147    2559 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 27 21:04:31.755299 ip-10-0-131-133 kubenswrapper[2559]: W0727 21:04:31.752150    2559 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 27 21:04:31.755299 ip-10-0-131-133 kubenswrapper[2559]: W0727 21:04:31.752153    2559 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 27 21:04:31.755299 ip-10-0-131-133 kubenswrapper[2559]: W0727 21:04:31.752156    2559 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 27 21:04:31.755299 ip-10-0-131-133 kubenswrapper[2559]: W0727 21:04:31.752158    2559 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 27 21:04:31.755299 ip-10-0-131-133 kubenswrapper[2559]: W0727 21:04:31.752161    2559 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 27 21:04:31.755299 ip-10-0-131-133 kubenswrapper[2559]: W0727 21:04:31.752164    2559 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 27 21:04:31.755299 ip-10-0-131-133 kubenswrapper[2559]: W0727 21:04:31.752166    2559 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 27 21:04:31.755830 ip-10-0-131-133 kubenswrapper[2559]: W0727 21:04:31.752169    2559 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 27 21:04:31.755830 ip-10-0-131-133 kubenswrapper[2559]: W0727 21:04:31.752172    2559 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 27 21:04:31.755830 ip-10-0-131-133 kubenswrapper[2559]: W0727 21:04:31.752175    2559 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 27 21:04:31.755830 ip-10-0-131-133 kubenswrapper[2559]: W0727 21:04:31.752178    2559 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 27 21:04:31.755830 ip-10-0-131-133 kubenswrapper[2559]: W0727 21:04:31.752182    2559 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 27 21:04:31.755830 ip-10-0-131-133 kubenswrapper[2559]: W0727 21:04:31.752185    2559 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 27 21:04:31.755830 ip-10-0-131-133 kubenswrapper[2559]: W0727 21:04:31.752188    2559 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 27 21:04:31.755830 ip-10-0-131-133 kubenswrapper[2559]: W0727 21:04:31.752191    2559 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 27 21:04:31.755830 ip-10-0-131-133 kubenswrapper[2559]: W0727 21:04:31.752193    2559 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 27 21:04:31.755830 ip-10-0-131-133 kubenswrapper[2559]: W0727 21:04:31.752196    2559 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 27 21:04:31.755830 ip-10-0-131-133 kubenswrapper[2559]: W0727 21:04:31.752199    2559 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 27 21:04:31.755830 ip-10-0-131-133 kubenswrapper[2559]: W0727 21:04:31.752202    2559 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 27 21:04:31.755830 ip-10-0-131-133 kubenswrapper[2559]: W0727 21:04:31.752205    2559 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 27 21:04:31.755830 ip-10-0-131-133 kubenswrapper[2559]: W0727 21:04:31.752208    2559 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 27 21:04:31.755830 ip-10-0-131-133 kubenswrapper[2559]: W0727 21:04:31.752210    2559 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 27 21:04:31.755830 ip-10-0-131-133 kubenswrapper[2559]: W0727 21:04:31.752213    2559 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 27 21:04:31.755830 ip-10-0-131-133 kubenswrapper[2559]: W0727 21:04:31.752216    2559 feature_gate.go:328] unrecognized feature gate: Example2
Jul 27 21:04:31.755830 ip-10-0-131-133 kubenswrapper[2559]: W0727 21:04:31.752219    2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 27 21:04:31.755830 ip-10-0-131-133 kubenswrapper[2559]: W0727 21:04:31.752221    2559 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 27 21:04:31.755830 ip-10-0-131-133 kubenswrapper[2559]: W0727 21:04:31.752224    2559 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 27 21:04:31.756384 ip-10-0-131-133 kubenswrapper[2559]: W0727 21:04:31.752739    2559 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 27 21:04:31.756384 ip-10-0-131-133 kubenswrapper[2559]: W0727 21:04:31.752747    2559 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 27 21:04:31.756384 ip-10-0-131-133 kubenswrapper[2559]: W0727 21:04:31.752751    2559 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 27 21:04:31.756384 ip-10-0-131-133 kubenswrapper[2559]: W0727 21:04:31.752755    2559 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 27 21:04:31.756384 ip-10-0-131-133 kubenswrapper[2559]: W0727 21:04:31.752759    2559 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 27 21:04:31.756384 ip-10-0-131-133 kubenswrapper[2559]: W0727 21:04:31.752762    2559 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 27 21:04:31.756384 ip-10-0-131-133 kubenswrapper[2559]: W0727 21:04:31.752766    2559 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 27 21:04:31.756384 ip-10-0-131-133 kubenswrapper[2559]: W0727 21:04:31.752769    2559 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 27 21:04:31.756384 ip-10-0-131-133 kubenswrapper[2559]: W0727 21:04:31.752772    2559 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 27 21:04:31.756384 ip-10-0-131-133 kubenswrapper[2559]: W0727 21:04:31.752776    2559 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 27 21:04:31.756384 ip-10-0-131-133 kubenswrapper[2559]: W0727 21:04:31.752779    2559 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 27 21:04:31.756384 ip-10-0-131-133 kubenswrapper[2559]: W0727 21:04:31.752782    2559 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 27 21:04:31.756384 ip-10-0-131-133 kubenswrapper[2559]: W0727 21:04:31.752786    2559 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 27 21:04:31.756384 ip-10-0-131-133 kubenswrapper[2559]: W0727 21:04:31.752789    2559 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 27 21:04:31.756384 ip-10-0-131-133 kubenswrapper[2559]: W0727 21:04:31.752792    2559 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 27 21:04:31.756384 ip-10-0-131-133 kubenswrapper[2559]: W0727 21:04:31.752795    2559 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 27 21:04:31.756384 ip-10-0-131-133 kubenswrapper[2559]: W0727 21:04:31.752798    2559 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 27 21:04:31.756384 ip-10-0-131-133 kubenswrapper[2559]: W0727 21:04:31.752800    2559 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 27 21:04:31.756384 ip-10-0-131-133 kubenswrapper[2559]: W0727 21:04:31.752803    2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 27 21:04:31.756384 ip-10-0-131-133 kubenswrapper[2559]: W0727 21:04:31.752806    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 27 21:04:31.756924 ip-10-0-131-133 kubenswrapper[2559]: W0727 21:04:31.752809    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 27 21:04:31.756924 ip-10-0-131-133 kubenswrapper[2559]: W0727 21:04:31.752812    2559 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 27 21:04:31.756924 ip-10-0-131-133 kubenswrapper[2559]: W0727 21:04:31.752815    2559 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 27 21:04:31.756924 ip-10-0-131-133 kubenswrapper[2559]: W0727 21:04:31.752818    2559 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 27 21:04:31.756924 ip-10-0-131-133 kubenswrapper[2559]: W0727 21:04:31.752820    2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 27 21:04:31.756924 ip-10-0-131-133 kubenswrapper[2559]: W0727 21:04:31.752823    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 27 21:04:31.756924 ip-10-0-131-133 kubenswrapper[2559]: W0727 21:04:31.752826    2559 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 27 21:04:31.756924 ip-10-0-131-133 kubenswrapper[2559]: W0727 21:04:31.752829    2559 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 27 21:04:31.756924 ip-10-0-131-133 kubenswrapper[2559]: W0727 21:04:31.752832    2559 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 27 21:04:31.756924 ip-10-0-131-133 kubenswrapper[2559]: W0727 21:04:31.752834    2559 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 27 21:04:31.756924 ip-10-0-131-133 kubenswrapper[2559]: W0727 21:04:31.752837    2559 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 27 21:04:31.756924 ip-10-0-131-133 kubenswrapper[2559]: W0727 21:04:31.752839    2559 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 27 21:04:31.756924 ip-10-0-131-133 kubenswrapper[2559]: W0727 21:04:31.752842    2559 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 27 21:04:31.756924 ip-10-0-131-133 kubenswrapper[2559]: W0727 21:04:31.752845    2559 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 27 21:04:31.756924 ip-10-0-131-133 kubenswrapper[2559]: W0727 21:04:31.752848    2559 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 27 21:04:31.756924 ip-10-0-131-133 kubenswrapper[2559]: W0727 21:04:31.752851    2559 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 27 21:04:31.756924 ip-10-0-131-133 kubenswrapper[2559]: W0727 21:04:31.752853    2559 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 27 21:04:31.756924 ip-10-0-131-133 kubenswrapper[2559]: W0727 21:04:31.752860    2559 feature_gate.go:328] unrecognized feature gate: Example
Jul 27 21:04:31.756924 ip-10-0-131-133 kubenswrapper[2559]: W0727 21:04:31.752863    2559 feature_gate.go:328] unrecognized feature gate: Example2
Jul 27 21:04:31.756924 ip-10-0-131-133 kubenswrapper[2559]: W0727 21:04:31.752866    2559 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 27 21:04:31.757487 ip-10-0-131-133 kubenswrapper[2559]: W0727 21:04:31.752868    2559 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 27 21:04:31.757487 ip-10-0-131-133 kubenswrapper[2559]: W0727 21:04:31.752871    2559 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 27 21:04:31.757487 ip-10-0-131-133 kubenswrapper[2559]: W0727 21:04:31.752874    2559 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 27 21:04:31.757487 ip-10-0-131-133 kubenswrapper[2559]: W0727 21:04:31.752877    2559 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 27 21:04:31.757487 ip-10-0-131-133 kubenswrapper[2559]: W0727 21:04:31.752880    2559 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 27 21:04:31.757487 ip-10-0-131-133 kubenswrapper[2559]: W0727 21:04:31.752885    2559 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 27 21:04:31.757487 ip-10-0-131-133 kubenswrapper[2559]: W0727 21:04:31.752889    2559 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 27 21:04:31.757487 ip-10-0-131-133 kubenswrapper[2559]: W0727 21:04:31.752892    2559 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 27 21:04:31.757487 ip-10-0-131-133 kubenswrapper[2559]: W0727 21:04:31.752895    2559 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 27 21:04:31.757487 ip-10-0-131-133 kubenswrapper[2559]: W0727 21:04:31.752897    2559 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 27 21:04:31.757487 ip-10-0-131-133 kubenswrapper[2559]: W0727 21:04:31.752900    2559 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 27 21:04:31.757487 ip-10-0-131-133 kubenswrapper[2559]: W0727 21:04:31.752903    2559 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 27 21:04:31.757487 ip-10-0-131-133 kubenswrapper[2559]: W0727 21:04:31.752906    2559 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 27 21:04:31.757487 ip-10-0-131-133 kubenswrapper[2559]: W0727 21:04:31.752908    2559 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 27 21:04:31.757487 ip-10-0-131-133 kubenswrapper[2559]: W0727 21:04:31.752911    2559 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 27 21:04:31.757487 ip-10-0-131-133 kubenswrapper[2559]: W0727 21:04:31.752914    2559 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 27 21:04:31.757487 ip-10-0-131-133 kubenswrapper[2559]: W0727 21:04:31.752916    2559 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 27 21:04:31.757487 ip-10-0-131-133 kubenswrapper[2559]: W0727 21:04:31.752919    2559 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 27 21:04:31.757487 ip-10-0-131-133 kubenswrapper[2559]: W0727 21:04:31.752922    2559 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 27 21:04:31.757989 ip-10-0-131-133 kubenswrapper[2559]: W0727 21:04:31.752924    2559 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 27 21:04:31.757989 ip-10-0-131-133 kubenswrapper[2559]: W0727 21:04:31.752927    2559 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 27 21:04:31.757989 ip-10-0-131-133 kubenswrapper[2559]: W0727 21:04:31.752930    2559 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 27 21:04:31.757989 ip-10-0-131-133 kubenswrapper[2559]: W0727 21:04:31.752934    2559 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 27 21:04:31.757989 ip-10-0-131-133 kubenswrapper[2559]: W0727 21:04:31.752938    2559 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 27 21:04:31.757989 ip-10-0-131-133 kubenswrapper[2559]: W0727 21:04:31.752941    2559 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 27 21:04:31.757989 ip-10-0-131-133 kubenswrapper[2559]: W0727 21:04:31.752945    2559 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 27 21:04:31.757989 ip-10-0-131-133 kubenswrapper[2559]: W0727 21:04:31.752948    2559 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 27 21:04:31.757989 ip-10-0-131-133 kubenswrapper[2559]: W0727 21:04:31.752951    2559 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 27 21:04:31.757989 ip-10-0-131-133 kubenswrapper[2559]: W0727 21:04:31.752954    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 27 21:04:31.757989 ip-10-0-131-133 kubenswrapper[2559]: W0727 21:04:31.752957    2559 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 27 21:04:31.757989 ip-10-0-131-133 kubenswrapper[2559]: W0727 21:04:31.752960    2559 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 27 21:04:31.757989 ip-10-0-131-133 kubenswrapper[2559]: W0727 21:04:31.752962    2559 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 27 21:04:31.757989 ip-10-0-131-133 kubenswrapper[2559]: W0727 21:04:31.752965    2559 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 27 21:04:31.757989 ip-10-0-131-133 kubenswrapper[2559]: W0727 21:04:31.752968    2559 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 27 21:04:31.757989 ip-10-0-131-133 kubenswrapper[2559]: W0727 21:04:31.752973    2559 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 27 21:04:31.757989 ip-10-0-131-133 kubenswrapper[2559]: W0727 21:04:31.752976    2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 27 21:04:31.757989 ip-10-0-131-133 kubenswrapper[2559]: W0727 21:04:31.752978    2559 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 27 21:04:31.757989 ip-10-0-131-133 kubenswrapper[2559]: W0727 21:04:31.752981    2559 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 27 21:04:31.758557 ip-10-0-131-133 kubenswrapper[2559]: W0727 21:04:31.752984    2559 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 27 21:04:31.758557 ip-10-0-131-133 kubenswrapper[2559]: W0727 21:04:31.752987    2559 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 27 21:04:31.758557 ip-10-0-131-133 kubenswrapper[2559]: W0727 21:04:31.752990    2559 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 27 21:04:31.758557 ip-10-0-131-133 kubenswrapper[2559]: W0727 21:04:31.752992    2559 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 27 21:04:31.758557 ip-10-0-131-133 kubenswrapper[2559]: W0727 21:04:31.752995    2559 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 27 21:04:31.758557 ip-10-0-131-133 kubenswrapper[2559]: W0727 21:04:31.752998    2559 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 27 21:04:31.758557 ip-10-0-131-133 kubenswrapper[2559]: W0727 21:04:31.753001    2559 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 27 21:04:31.758557 ip-10-0-131-133 kubenswrapper[2559]: W0727 21:04:31.753004    2559 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 27 21:04:31.758557 ip-10-0-131-133 kubenswrapper[2559]: W0727 21:04:31.753006    2559 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 27 21:04:31.758557 ip-10-0-131-133 kubenswrapper[2559]: W0727 21:04:31.753009    2559 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 27 21:04:31.758557 ip-10-0-131-133 kubenswrapper[2559]: W0727 21:04:31.753012    2559 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 27 21:04:31.758557 ip-10-0-131-133 kubenswrapper[2559]: W0727 21:04:31.753015    2559 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 27 21:04:31.758557 ip-10-0-131-133 kubenswrapper[2559]: W0727 21:04:31.753017    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 27 21:04:31.758557 ip-10-0-131-133 kubenswrapper[2559]: W0727 21:04:31.753020    2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 27 21:04:31.758557 ip-10-0-131-133 kubenswrapper[2559]: W0727 21:04:31.753023    2559 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 27 21:04:31.758557 ip-10-0-131-133 kubenswrapper[2559]: W0727 21:04:31.753026    2559 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 27 21:04:31.758557 ip-10-0-131-133 kubenswrapper[2559]: W0727 21:04:31.753030    2559 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 27 21:04:31.758557 ip-10-0-131-133 kubenswrapper[2559]: W0727 21:04:31.753033    2559 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 27 21:04:31.758557 ip-10-0-131-133 kubenswrapper[2559]: W0727 21:04:31.753036    2559 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 27 21:04:31.758557 ip-10-0-131-133 kubenswrapper[2559]: W0727 21:04:31.753042    2559 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 27 21:04:31.759085 ip-10-0-131-133 kubenswrapper[2559]: W0727 21:04:31.753046    2559 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 27 21:04:31.759085 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:04:31.753121    2559 flags.go:64] FLAG: --address="0.0.0.0"
Jul 27 21:04:31.759085 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:04:31.753129    2559 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Jul 27 21:04:31.759085 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:04:31.753136    2559 flags.go:64] FLAG: --anonymous-auth="true"
Jul 27 21:04:31.759085 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:04:31.753141    2559 flags.go:64] FLAG: --application-metrics-count-limit="100"
Jul 27 21:04:31.759085 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:04:31.753146    2559 flags.go:64] FLAG: --authentication-token-webhook="false"
Jul 27 21:04:31.759085 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:04:31.753150    2559 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Jul 27 21:04:31.759085 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:04:31.753155    2559 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Jul 27 21:04:31.759085 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:04:31.753160    2559 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Jul 27 21:04:31.759085 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:04:31.753164    2559 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Jul 27 21:04:31.759085 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:04:31.753167    2559 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Jul 27 21:04:31.759085 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:04:31.753171    2559 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Jul 27 21:04:31.759085 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:04:31.753175    2559 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Jul 27 21:04:31.759085 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:04:31.753178    2559 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Jul 27 21:04:31.759085 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:04:31.753182    2559 flags.go:64] FLAG: --cgroup-root=""
Jul 27 21:04:31.759085 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:04:31.753185    2559 flags.go:64] FLAG: --cgroups-per-qos="true"
Jul 27 21:04:31.759085 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:04:31.753188    2559 flags.go:64] FLAG: --client-ca-file=""
Jul 27 21:04:31.759085 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:04:31.753191    2559 flags.go:64] FLAG: --cloud-provider="external"
Jul 27 21:04:31.759085 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:04:31.753195    2559 flags.go:64] FLAG: --cluster-dns="[]"
Jul 27 21:04:31.759085 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:04:31.753200    2559 flags.go:64] FLAG: --cluster-domain=""
Jul 27 21:04:31.759085 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:04:31.753203    2559 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Jul 27 21:04:31.759085 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:04:31.753207    2559 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Jul 27 21:04:31.759085 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:04:31.753211    2559 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Jul 27 21:04:31.759728 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:04:31.753214    2559 flags.go:64] FLAG: --container-log-max-files="5"
Jul 27 21:04:31.759728 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:04:31.753218    2559 flags.go:64] FLAG: --container-log-max-size="10Mi"
Jul 27 21:04:31.759728 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:04:31.753222    2559 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Jul 27 21:04:31.759728 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:04:31.753225    2559 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Jul 27 21:04:31.759728 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:04:31.753229    2559 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Jul 27 21:04:31.759728 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:04:31.753233    2559 flags.go:64] FLAG: --contention-profiling="false"
Jul 27 21:04:31.759728 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:04:31.753236    2559 flags.go:64] FLAG: --cpu-cfs-quota="true"
Jul 27 21:04:31.759728 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:04:31.753239    2559 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Jul 27 21:04:31.759728 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:04:31.753244    2559 flags.go:64] FLAG: --cpu-manager-policy="none"
Jul 27 21:04:31.759728 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:04:31.753248    2559 flags.go:64] FLAG: --cpu-manager-policy-options=""
Jul 27 21:04:31.759728 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:04:31.753252    2559 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Jul 27 21:04:31.759728 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:04:31.753255    2559 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Jul 27 21:04:31.759728 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:04:31.753258    2559 flags.go:64] FLAG: --enable-debugging-handlers="true"
Jul 27 21:04:31.759728 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:04:31.753262    2559 flags.go:64] FLAG: --enable-load-reader="false"
Jul 27 21:04:31.759728 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:04:31.753265    2559 flags.go:64] FLAG: --enable-server="true"
Jul 27 21:04:31.759728 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:04:31.753269    2559 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Jul 27 21:04:31.759728 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:04:31.753275    2559 flags.go:64] FLAG: --event-burst="100"
Jul 27 21:04:31.759728 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:04:31.753279    2559 flags.go:64] FLAG: --event-qps="50"
Jul 27 21:04:31.759728 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:04:31.753282    2559 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Jul 27 21:04:31.759728 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:04:31.753299    2559 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Jul 27 21:04:31.759728 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:04:31.753303    2559 flags.go:64] FLAG: --eviction-hard=""
Jul 27 21:04:31.759728 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:04:31.753307    2559 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Jul 27 21:04:31.759728 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:04:31.753310    2559 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Jul 27 21:04:31.759728 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:04:31.753313    2559 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Jul 27 21:04:31.759728 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:04:31.753317    2559 flags.go:64] FLAG: --eviction-soft=""
Jul 27 21:04:31.760414 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:04:31.753320    2559 flags.go:64] FLAG: --eviction-soft-grace-period=""
Jul 27 21:04:31.760414 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:04:31.753323    2559 flags.go:64] FLAG: --exit-on-lock-contention="false"
Jul 27 21:04:31.760414 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:04:31.753326    2559 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Jul 27 21:04:31.760414 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:04:31.753329    2559 flags.go:64] FLAG: --experimental-mounter-path=""
Jul 27 21:04:31.760414 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:04:31.753333    2559 flags.go:64] FLAG: --fail-cgroupv1="false"
Jul 27 21:04:31.760414 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:04:31.753336    2559 flags.go:64] FLAG: --fail-swap-on="true"
Jul 27 21:04:31.760414 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:04:31.753339    2559 flags.go:64] FLAG: --feature-gates=""
Jul 27 21:04:31.760414 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:04:31.753343    2559 flags.go:64] FLAG: --file-check-frequency="20s"
Jul 27 21:04:31.760414 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:04:31.753346    2559 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Jul 27 21:04:31.760414 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:04:31.753349    2559 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Jul 27 21:04:31.760414 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:04:31.753353    2559 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Jul 27 21:04:31.760414 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:04:31.753356    2559 flags.go:64] FLAG: --healthz-port="10248"
Jul 27 21:04:31.760414 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:04:31.753359    2559 flags.go:64] FLAG: --help="false"
Jul 27 21:04:31.760414 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:04:31.753362    2559 flags.go:64] FLAG: --hostname-override="ip-10-0-131-133.ec2.internal"
Jul 27 21:04:31.760414 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:04:31.753366    2559 flags.go:64] FLAG: --housekeeping-interval="10s"
Jul 27 21:04:31.760414 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:04:31.753370    2559 flags.go:64] FLAG: --http-check-frequency="20s"
Jul 27 21:04:31.760414 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:04:31.753374    2559 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Jul 27 21:04:31.760414 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:04:31.753377    2559 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Jul 27 21:04:31.760414 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:04:31.753381    2559 flags.go:64] FLAG: --image-gc-high-threshold="85"
Jul 27 21:04:31.760414 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:04:31.753384    2559 flags.go:64] FLAG: --image-gc-low-threshold="80"
Jul 27 21:04:31.760414 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:04:31.753387    2559 flags.go:64] FLAG: --image-service-endpoint=""
Jul 27 21:04:31.760414 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:04:31.753390    2559 flags.go:64] FLAG: --kernel-memcg-notification="false"
Jul 27 21:04:31.760414 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:04:31.753393    2559 flags.go:64] FLAG: --kube-api-burst="100"
Jul 27 21:04:31.761075 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:04:31.753398    2559 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Jul 27 21:04:31.761075 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:04:31.753401    2559 flags.go:64] FLAG: --kube-api-qps="50"
Jul 27 21:04:31.761075 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:04:31.753404    2559 flags.go:64] FLAG: --kube-reserved=""
Jul 27 21:04:31.761075 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:04:31.753407    2559 flags.go:64] FLAG: --kube-reserved-cgroup=""
Jul 27 21:04:31.761075 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:04:31.753410    2559 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Jul 27 21:04:31.761075 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:04:31.753414    2559 flags.go:64] FLAG: --kubelet-cgroups=""
Jul 27 21:04:31.761075 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:04:31.753416    2559 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Jul 27 21:04:31.761075 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:04:31.753419    2559 flags.go:64] FLAG: --lock-file=""
Jul 27 21:04:31.761075 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:04:31.753422    2559 flags.go:64] FLAG: --log-cadvisor-usage="false"
Jul 27 21:04:31.761075 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:04:31.753425    2559 flags.go:64] FLAG: --log-flush-frequency="5s"
Jul 27 21:04:31.761075 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:04:31.753429    2559 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Jul 27 21:04:31.761075 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:04:31.753435    2559 flags.go:64] FLAG: --log-json-split-stream="false"
Jul 27 21:04:31.761075 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:04:31.753438    2559 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Jul 27 21:04:31.761075 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:04:31.753441    2559 flags.go:64] FLAG: --log-text-split-stream="false"
Jul 27 21:04:31.761075 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:04:31.753444    2559 flags.go:64] FLAG: --logging-format="text"
Jul 27 21:04:31.761075 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:04:31.753447    2559 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Jul 27 21:04:31.761075 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:04:31.753450    2559 flags.go:64] FLAG: --make-iptables-util-chains="true"
Jul 27 21:04:31.761075 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:04:31.753453    2559 flags.go:64] FLAG: --manifest-url=""
Jul 27 21:04:31.761075 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:04:31.753456    2559 flags.go:64] FLAG: --manifest-url-header=""
Jul 27 21:04:31.761075 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:04:31.753461    2559 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Jul 27 21:04:31.761075 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:04:31.753464    2559 flags.go:64] FLAG: --max-open-files="1000000"
Jul 27 21:04:31.761075 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:04:31.753469    2559 flags.go:64] FLAG: --max-pods="110"
Jul 27 21:04:31.761075 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:04:31.753472    2559 flags.go:64] FLAG: --maximum-dead-containers="-1"
Jul 27 21:04:31.761075 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:04:31.753475    2559 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Jul 27 21:04:31.761075 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:04:31.753479    2559 flags.go:64] FLAG: --memory-manager-policy="None"
Jul 27 21:04:31.761762 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:04:31.753483    2559 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Jul 27 21:04:31.761762 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:04:31.753486    2559 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Jul 27 21:04:31.761762 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:04:31.753489    2559 flags.go:64] FLAG: --node-ip="0.0.0.0"
Jul 27 21:04:31.761762 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:04:31.753492    2559 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Jul 27 21:04:31.761762 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:04:31.753500    2559 flags.go:64] FLAG: --node-status-max-images="50"
Jul 27 21:04:31.761762 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:04:31.753503    2559 flags.go:64] FLAG: --node-status-update-frequency="10s"
Jul 27 21:04:31.761762 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:04:31.753506    2559 flags.go:64] FLAG: --oom-score-adj="-999"
Jul 27 21:04:31.761762 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:04:31.753511    2559 flags.go:64] FLAG: --pod-cidr=""
Jul 27 21:04:31.761762 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:04:31.753515    2559 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:7505336a406394b8c1c77d14d890a9f628aa4cfff172312179cb9af8c28c30bb"
Jul 27 21:04:31.761762 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:04:31.753522    2559 flags.go:64] FLAG: --pod-manifest-path=""
Jul 27 21:04:31.761762 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:04:31.753526    2559 flags.go:64] FLAG: --pod-max-pids="-1"
Jul 27 21:04:31.761762 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:04:31.753529    2559 flags.go:64] FLAG: --pods-per-core="0"
Jul 27 21:04:31.761762 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:04:31.753532    2559 flags.go:64] FLAG: --port="10250"
Jul 27 21:04:31.761762 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:04:31.753536    2559 flags.go:64] FLAG: --protect-kernel-defaults="false"
Jul 27 21:04:31.761762 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:04:31.753538    2559 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-00600c7530b871cf5"
Jul 27 21:04:31.761762 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:04:31.753542    2559 flags.go:64] FLAG: --qos-reserved=""
Jul 27 21:04:31.761762 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:04:31.753545    2559 flags.go:64] FLAG: --read-only-port="10255"
Jul 27 21:04:31.761762 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:04:31.753548    2559 flags.go:64] FLAG: --register-node="true"
Jul 27 21:04:31.761762 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:04:31.753552    2559 flags.go:64] FLAG: --register-with-taints=""
Jul 27 21:04:31.761762 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:04:31.753555    2559 flags.go:64] FLAG: --registry-burst="10"
Jul 27 21:04:31.761762 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:04:31.753559    2559 flags.go:64] FLAG: --registry-qps="5"
Jul 27 21:04:31.761762 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:04:31.753562    2559 flags.go:64] FLAG: --reserved-cpus=""
Jul 27 21:04:31.761762 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:04:31.753565    2559 flags.go:64] FLAG: --reserved-memory=""
Jul 27 21:04:31.761762 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:04:31.753568    2559 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Jul 27 21:04:31.761762 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:04:31.753572    2559 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Jul 27 21:04:31.762482 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:04:31.753575    2559 flags.go:64] FLAG: --rotate-certificates="false"
Jul 27 21:04:31.762482 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:04:31.753578    2559 flags.go:64] FLAG: --rotate-server-certificates="false"
Jul 27 21:04:31.762482 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:04:31.753581    2559 flags.go:64] FLAG: --runonce="false"
Jul 27 21:04:31.762482 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:04:31.753584    2559 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Jul 27 21:04:31.762482 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:04:31.753587    2559 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Jul 27 21:04:31.762482 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:04:31.753590    2559 flags.go:64] FLAG: --seccomp-default="false"
Jul 27 21:04:31.762482 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:04:31.753595    2559 flags.go:64] FLAG: --serialize-image-pulls="true"
Jul 27 21:04:31.762482 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:04:31.753598    2559 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Jul 27 21:04:31.762482 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:04:31.753601    2559 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Jul 27 21:04:31.762482 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:04:31.753605    2559 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Jul 27 21:04:31.762482 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:04:31.753608    2559 flags.go:64] FLAG: --storage-driver-password="root"
Jul 27 21:04:31.762482 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:04:31.753611    2559 flags.go:64] FLAG: --storage-driver-secure="false"
Jul 27 21:04:31.762482 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:04:31.753614    2559 flags.go:64] FLAG: --storage-driver-table="stats"
Jul 27 21:04:31.762482 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:04:31.753617    2559 flags.go:64] FLAG: --storage-driver-user="root"
Jul 27 21:04:31.762482 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:04:31.753620    2559 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Jul 27 21:04:31.762482 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:04:31.753624    2559 flags.go:64] FLAG: --sync-frequency="1m0s"
Jul 27 21:04:31.762482 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:04:31.753628    2559 flags.go:64] FLAG: --system-cgroups=""
Jul 27 21:04:31.762482 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:04:31.753631    2559 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Jul 27 21:04:31.762482 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:04:31.753637    2559 flags.go:64] FLAG: --system-reserved-cgroup=""
Jul 27 21:04:31.762482 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:04:31.753640    2559 flags.go:64] FLAG: --tls-cert-file=""
Jul 27 21:04:31.762482 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:04:31.753643    2559 flags.go:64] FLAG: --tls-cipher-suites="[]"
Jul 27 21:04:31.762482 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:04:31.753647    2559 flags.go:64] FLAG: --tls-min-version=""
Jul 27 21:04:31.762482 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:04:31.753651    2559 flags.go:64] FLAG: --tls-private-key-file=""
Jul 27 21:04:31.762482 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:04:31.753654    2559 flags.go:64] FLAG: --topology-manager-policy="none"
Jul 27 21:04:31.762482 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:04:31.753657    2559 flags.go:64] FLAG: --topology-manager-policy-options=""
Jul 27 21:04:31.763155 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:04:31.753661    2559 flags.go:64] FLAG: --topology-manager-scope="container"
Jul 27 21:04:31.763155 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:04:31.753664    2559 flags.go:64] FLAG: --v="2"
Jul 27 21:04:31.763155 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:04:31.753668    2559 flags.go:64] FLAG: --version="false"
Jul 27 21:04:31.763155 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:04:31.753672    2559 flags.go:64] FLAG: --vmodule=""
Jul 27 21:04:31.763155 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:04:31.753677    2559 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Jul 27 21:04:31.763155 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:04:31.753680    2559 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Jul 27 21:04:31.763155 ip-10-0-131-133 kubenswrapper[2559]: W0727 21:04:31.753827    2559 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 27 21:04:31.763155 ip-10-0-131-133 kubenswrapper[2559]: W0727 21:04:31.753831    2559 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 27 21:04:31.763155 ip-10-0-131-133 kubenswrapper[2559]: W0727 21:04:31.753834    2559 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 27 21:04:31.763155 ip-10-0-131-133 kubenswrapper[2559]: W0727 21:04:31.753837    2559 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 27 21:04:31.763155 ip-10-0-131-133 kubenswrapper[2559]: W0727 21:04:31.753840    2559 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 27 21:04:31.763155 ip-10-0-131-133 kubenswrapper[2559]: W0727 21:04:31.753843    2559 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 27 21:04:31.763155 ip-10-0-131-133 kubenswrapper[2559]: W0727 21:04:31.753846    2559 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 27 21:04:31.763155 ip-10-0-131-133 kubenswrapper[2559]: W0727 21:04:31.753850    2559 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 27 21:04:31.763155 ip-10-0-131-133 kubenswrapper[2559]: W0727 21:04:31.753853    2559 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 27 21:04:31.763155 ip-10-0-131-133 kubenswrapper[2559]: W0727 21:04:31.753858    2559 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 27 21:04:31.763155 ip-10-0-131-133 kubenswrapper[2559]: W0727 21:04:31.753862    2559 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 27 21:04:31.763155 ip-10-0-131-133 kubenswrapper[2559]: W0727 21:04:31.753866    2559 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 27 21:04:31.763155 ip-10-0-131-133 kubenswrapper[2559]: W0727 21:04:31.753869    2559 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 27 21:04:31.763155 ip-10-0-131-133 kubenswrapper[2559]: W0727 21:04:31.753872    2559 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 27 21:04:31.763155 ip-10-0-131-133 kubenswrapper[2559]: W0727 21:04:31.753874    2559 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 27 21:04:31.763752 ip-10-0-131-133 kubenswrapper[2559]: W0727 21:04:31.753877    2559 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 27 21:04:31.763752 ip-10-0-131-133 kubenswrapper[2559]: W0727 21:04:31.753882    2559 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 27 21:04:31.763752 ip-10-0-131-133 kubenswrapper[2559]: W0727 21:04:31.753885    2559 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 27 21:04:31.763752 ip-10-0-131-133 kubenswrapper[2559]: W0727 21:04:31.753889    2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 27 21:04:31.763752 ip-10-0-131-133 kubenswrapper[2559]: W0727 21:04:31.753891    2559 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 27 21:04:31.763752 ip-10-0-131-133 kubenswrapper[2559]: W0727 21:04:31.753894    2559 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 27 21:04:31.763752 ip-10-0-131-133 kubenswrapper[2559]: W0727 21:04:31.753897    2559 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 27 21:04:31.763752 ip-10-0-131-133 kubenswrapper[2559]: W0727 21:04:31.753900    2559 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 27 21:04:31.763752 ip-10-0-131-133 kubenswrapper[2559]: W0727 21:04:31.753903    2559 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 27 21:04:31.763752 ip-10-0-131-133 kubenswrapper[2559]: W0727 21:04:31.753906    2559 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 27 21:04:31.763752 ip-10-0-131-133 kubenswrapper[2559]: W0727 21:04:31.753908    2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 27 21:04:31.763752 ip-10-0-131-133 kubenswrapper[2559]: W0727 21:04:31.753911    2559 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 27 21:04:31.763752 ip-10-0-131-133 kubenswrapper[2559]: W0727 21:04:31.753914    2559 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 27 21:04:31.763752 ip-10-0-131-133 kubenswrapper[2559]: W0727 21:04:31.753917    2559 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 27 21:04:31.763752 ip-10-0-131-133 kubenswrapper[2559]: W0727 21:04:31.753919    2559 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 27 21:04:31.763752 ip-10-0-131-133 kubenswrapper[2559]: W0727 21:04:31.753922    2559 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 27 21:04:31.763752 ip-10-0-131-133 kubenswrapper[2559]: W0727 21:04:31.753925    2559 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 27 21:04:31.763752 ip-10-0-131-133 kubenswrapper[2559]: W0727 21:04:31.753928    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 27 21:04:31.763752 ip-10-0-131-133 kubenswrapper[2559]: W0727 21:04:31.753932    2559 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 27 21:04:31.764258 ip-10-0-131-133 kubenswrapper[2559]: W0727 21:04:31.753935    2559 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 27 21:04:31.764258 ip-10-0-131-133 kubenswrapper[2559]: W0727 21:04:31.753939    2559 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 27 21:04:31.764258 ip-10-0-131-133 kubenswrapper[2559]: W0727 21:04:31.753942    2559 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 27 21:04:31.764258 ip-10-0-131-133 kubenswrapper[2559]: W0727 21:04:31.753946    2559 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 27 21:04:31.764258 ip-10-0-131-133 kubenswrapper[2559]: W0727 21:04:31.753950    2559 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 27 21:04:31.764258 ip-10-0-131-133 kubenswrapper[2559]: W0727 21:04:31.753955    2559 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 27 21:04:31.764258 ip-10-0-131-133 kubenswrapper[2559]: W0727 21:04:31.753958    2559 feature_gate.go:328] unrecognized feature gate: Example2
Jul 27 21:04:31.764258 ip-10-0-131-133 kubenswrapper[2559]: W0727 21:04:31.753961    2559 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 27 21:04:31.764258 ip-10-0-131-133 kubenswrapper[2559]: W0727 21:04:31.753964    2559 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 27 21:04:31.764258 ip-10-0-131-133 kubenswrapper[2559]: W0727 21:04:31.753967    2559 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 27 21:04:31.764258 ip-10-0-131-133 kubenswrapper[2559]: W0727 21:04:31.753970    2559 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 27 21:04:31.764258 ip-10-0-131-133 kubenswrapper[2559]: W0727 21:04:31.753973    2559 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 27 21:04:31.764258 ip-10-0-131-133 kubenswrapper[2559]: W0727 21:04:31.753976    2559 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 27 21:04:31.764258 ip-10-0-131-133 kubenswrapper[2559]: W0727 21:04:31.753979    2559 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 27 21:04:31.764258 ip-10-0-131-133 kubenswrapper[2559]: W0727 21:04:31.753985    2559 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 27 21:04:31.764258 ip-10-0-131-133 kubenswrapper[2559]: W0727 21:04:31.753988    2559 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 27 21:04:31.764258 ip-10-0-131-133 kubenswrapper[2559]: W0727 21:04:31.753991    2559 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 27 21:04:31.764258 ip-10-0-131-133 kubenswrapper[2559]: W0727 21:04:31.753995    2559 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 27 21:04:31.764258 ip-10-0-131-133 kubenswrapper[2559]: W0727 21:04:31.753998    2559 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 27 21:04:31.764258 ip-10-0-131-133 kubenswrapper[2559]: W0727 21:04:31.754001    2559 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 27 21:04:31.764820 ip-10-0-131-133 kubenswrapper[2559]: W0727 21:04:31.754004    2559 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 27 21:04:31.764820 ip-10-0-131-133 kubenswrapper[2559]: W0727 21:04:31.754007    2559 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 27 21:04:31.764820 ip-10-0-131-133 kubenswrapper[2559]: W0727 21:04:31.754010    2559 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 27 21:04:31.764820 ip-10-0-131-133 kubenswrapper[2559]: W0727 21:04:31.754013    2559 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 27 21:04:31.764820 ip-10-0-131-133 kubenswrapper[2559]: W0727 21:04:31.754016    2559 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 27 21:04:31.764820 ip-10-0-131-133 kubenswrapper[2559]: W0727 21:04:31.754018    2559 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 27 21:04:31.764820 ip-10-0-131-133 kubenswrapper[2559]: W0727 21:04:31.754021    2559 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 27 21:04:31.764820 ip-10-0-131-133 kubenswrapper[2559]: W0727 21:04:31.754023    2559 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 27 21:04:31.764820 ip-10-0-131-133 kubenswrapper[2559]: W0727 21:04:31.754026    2559 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 27 21:04:31.764820 ip-10-0-131-133 kubenswrapper[2559]: W0727 21:04:31.754029    2559 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 27 21:04:31.764820 ip-10-0-131-133 kubenswrapper[2559]: W0727 21:04:31.754032    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 27 21:04:31.764820 ip-10-0-131-133 kubenswrapper[2559]: W0727 21:04:31.754035    2559 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 27 21:04:31.764820 ip-10-0-131-133 kubenswrapper[2559]: W0727 21:04:31.754037    2559 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 27 21:04:31.764820 ip-10-0-131-133 kubenswrapper[2559]: W0727 21:04:31.754040    2559 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 27 21:04:31.764820 ip-10-0-131-133 kubenswrapper[2559]: W0727 21:04:31.754043    2559 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 27 21:04:31.764820 ip-10-0-131-133 kubenswrapper[2559]: W0727 21:04:31.754045    2559 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 27 21:04:31.764820 ip-10-0-131-133 kubenswrapper[2559]: W0727 21:04:31.754048    2559 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 27 21:04:31.764820 ip-10-0-131-133 kubenswrapper[2559]: W0727 21:04:31.754052    2559 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 27 21:04:31.764820 ip-10-0-131-133 kubenswrapper[2559]: W0727 21:04:31.754055    2559 feature_gate.go:328] unrecognized feature gate: Example
Jul 27 21:04:31.764820 ip-10-0-131-133 kubenswrapper[2559]: W0727 21:04:31.754058    2559 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 27 21:04:31.765536 ip-10-0-131-133 kubenswrapper[2559]: W0727 21:04:31.754060    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 27 21:04:31.765536 ip-10-0-131-133 kubenswrapper[2559]: W0727 21:04:31.754064    2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 27 21:04:31.765536 ip-10-0-131-133 kubenswrapper[2559]: W0727 21:04:31.754067    2559 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 27 21:04:31.765536 ip-10-0-131-133 kubenswrapper[2559]: W0727 21:04:31.754069    2559 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 27 21:04:31.765536 ip-10-0-131-133 kubenswrapper[2559]: W0727 21:04:31.754072    2559 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 27 21:04:31.765536 ip-10-0-131-133 kubenswrapper[2559]: W0727 21:04:31.754075    2559 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 27 21:04:31.765536 ip-10-0-131-133 kubenswrapper[2559]: W0727 21:04:31.754079    2559 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 27 21:04:31.765536 ip-10-0-131-133 kubenswrapper[2559]: W0727 21:04:31.754081    2559 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 27 21:04:31.765536 ip-10-0-131-133 kubenswrapper[2559]: W0727 21:04:31.754084    2559 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 27 21:04:31.765536 ip-10-0-131-133 kubenswrapper[2559]: W0727 21:04:31.754087    2559 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 27 21:04:31.765536 ip-10-0-131-133 kubenswrapper[2559]: W0727 21:04:31.754091    2559 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 27 21:04:31.765536 ip-10-0-131-133 kubenswrapper[2559]: W0727 21:04:31.754093    2559 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 27 21:04:31.765536 ip-10-0-131-133 kubenswrapper[2559]: W0727 21:04:31.754096    2559 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 27 21:04:31.765536 ip-10-0-131-133 kubenswrapper[2559]: W0727 21:04:31.754099    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 27 21:04:31.765536 ip-10-0-131-133 kubenswrapper[2559]: W0727 21:04:31.754101    2559 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 27 21:04:31.765536 ip-10-0-131-133 kubenswrapper[2559]: W0727 21:04:31.754104    2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 27 21:04:31.765536 ip-10-0-131-133 kubenswrapper[2559]: W0727 21:04:31.754107    2559 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 27 21:04:31.765536 ip-10-0-131-133 kubenswrapper[2559]: W0727 21:04:31.754109    2559 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 27 21:04:31.765536 ip-10-0-131-133 kubenswrapper[2559]: W0727 21:04:31.754112    2559 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 27 21:04:31.765536 ip-10-0-131-133 kubenswrapper[2559]: W0727 21:04:31.754115    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 27 21:04:31.766076 ip-10-0-131-133 kubenswrapper[2559]: W0727 21:04:31.754118    2559 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 27 21:04:31.766076 ip-10-0-131-133 kubenswrapper[2559]: W0727 21:04:31.754120    2559 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 27 21:04:31.766076 ip-10-0-131-133 kubenswrapper[2559]: W0727 21:04:31.754123    2559 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 27 21:04:31.766076 ip-10-0-131-133 kubenswrapper[2559]: W0727 21:04:31.754126    2559 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 27 21:04:31.766076 ip-10-0-131-133 kubenswrapper[2559]: W0727 21:04:31.754129    2559 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 27 21:04:31.766076 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:04:31.754823    2559 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 27 21:04:31.766076 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:04:31.763054    2559 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Jul 27 21:04:31.766076 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:04:31.763080    2559 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Jul 27 21:04:31.766076 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:04:31.763102    2559 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Jul 27 21:04:31.766076 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:04:31.763109    2559 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Jul 27 21:04:31.766076 ip-10-0-131-133 kubenswrapper[2559]: W0727 21:04:31.763192    2559 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 27 21:04:31.766076 ip-10-0-131-133 kubenswrapper[2559]: W0727 21:04:31.763199    2559 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 27 21:04:31.766076 ip-10-0-131-133 kubenswrapper[2559]: W0727 21:04:31.763203    2559 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 27 21:04:31.766076 ip-10-0-131-133 kubenswrapper[2559]: W0727 21:04:31.763206    2559 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 27 21:04:31.766076 ip-10-0-131-133 kubenswrapper[2559]: W0727 21:04:31.763210    2559 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 27 21:04:31.766502 ip-10-0-131-133 kubenswrapper[2559]: W0727 21:04:31.763213    2559 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 27 21:04:31.766502 ip-10-0-131-133 kubenswrapper[2559]: W0727 21:04:31.763216    2559 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 27 21:04:31.766502 ip-10-0-131-133 kubenswrapper[2559]: W0727 21:04:31.763219    2559 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 27 21:04:31.766502 ip-10-0-131-133 kubenswrapper[2559]: W0727 21:04:31.763221    2559 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 27 21:04:31.766502 ip-10-0-131-133 kubenswrapper[2559]: W0727 21:04:31.763224    2559 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 27 21:04:31.766502 ip-10-0-131-133 kubenswrapper[2559]: W0727 21:04:31.763227    2559 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 27 21:04:31.766502 ip-10-0-131-133 kubenswrapper[2559]: W0727 21:04:31.763230    2559 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 27 21:04:31.766502 ip-10-0-131-133 kubenswrapper[2559]: W0727 21:04:31.763232    2559 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 27 21:04:31.766502 ip-10-0-131-133 kubenswrapper[2559]: W0727 21:04:31.763235    2559 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 27 21:04:31.766502 ip-10-0-131-133 kubenswrapper[2559]: W0727 21:04:31.763238    2559 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 27 21:04:31.766502 ip-10-0-131-133 kubenswrapper[2559]: W0727 21:04:31.763241    2559 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 27 21:04:31.766502 ip-10-0-131-133 kubenswrapper[2559]: W0727 21:04:31.763244    2559 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 27 21:04:31.766502 ip-10-0-131-133 kubenswrapper[2559]: W0727 21:04:31.763247    2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 27 21:04:31.766502 ip-10-0-131-133 kubenswrapper[2559]: W0727 21:04:31.763250    2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 27 21:04:31.766502 ip-10-0-131-133 kubenswrapper[2559]: W0727 21:04:31.763253    2559 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 27 21:04:31.766502 ip-10-0-131-133 kubenswrapper[2559]: W0727 21:04:31.763256    2559 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 27 21:04:31.766502 ip-10-0-131-133 kubenswrapper[2559]: W0727 21:04:31.763259    2559 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 27 21:04:31.766502 ip-10-0-131-133 kubenswrapper[2559]: W0727 21:04:31.763274    2559 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 27 21:04:31.766502 ip-10-0-131-133 kubenswrapper[2559]: W0727 21:04:31.763279    2559 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 27 21:04:31.767003 ip-10-0-131-133 kubenswrapper[2559]: W0727 21:04:31.763283    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 27 21:04:31.767003 ip-10-0-131-133 kubenswrapper[2559]: W0727 21:04:31.763286    2559 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 27 21:04:31.767003 ip-10-0-131-133 kubenswrapper[2559]: W0727 21:04:31.763289    2559 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 27 21:04:31.767003 ip-10-0-131-133 kubenswrapper[2559]: W0727 21:04:31.763293    2559 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 27 21:04:31.767003 ip-10-0-131-133 kubenswrapper[2559]: W0727 21:04:31.763296    2559 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 27 21:04:31.767003 ip-10-0-131-133 kubenswrapper[2559]: W0727 21:04:31.763307    2559 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 27 21:04:31.767003 ip-10-0-131-133 kubenswrapper[2559]: W0727 21:04:31.763310    2559 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 27 21:04:31.767003 ip-10-0-131-133 kubenswrapper[2559]: W0727 21:04:31.763313    2559 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 27 21:04:31.767003 ip-10-0-131-133 kubenswrapper[2559]: W0727 21:04:31.763316    2559 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 27 21:04:31.767003 ip-10-0-131-133 kubenswrapper[2559]: W0727 21:04:31.763319    2559 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 27 21:04:31.767003 ip-10-0-131-133 kubenswrapper[2559]: W0727 21:04:31.763322    2559 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 27 21:04:31.767003 ip-10-0-131-133 kubenswrapper[2559]: W0727 21:04:31.763325    2559 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 27 21:04:31.767003 ip-10-0-131-133 kubenswrapper[2559]: W0727 21:04:31.763327    2559 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 27 21:04:31.767003 ip-10-0-131-133 kubenswrapper[2559]: W0727 21:04:31.763330    2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 27 21:04:31.767003 ip-10-0-131-133 kubenswrapper[2559]: W0727 21:04:31.763333    2559 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 27 21:04:31.767003 ip-10-0-131-133 kubenswrapper[2559]: W0727 21:04:31.763336    2559 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 27 21:04:31.767003 ip-10-0-131-133 kubenswrapper[2559]: W0727 21:04:31.763340    2559 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 27 21:04:31.767003 ip-10-0-131-133 kubenswrapper[2559]: W0727 21:04:31.763343    2559 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 27 21:04:31.767003 ip-10-0-131-133 kubenswrapper[2559]: W0727 21:04:31.763345    2559 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 27 21:04:31.767003 ip-10-0-131-133 kubenswrapper[2559]: W0727 21:04:31.763348    2559 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 27 21:04:31.767641 ip-10-0-131-133 kubenswrapper[2559]: W0727 21:04:31.763351    2559 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 27 21:04:31.767641 ip-10-0-131-133 kubenswrapper[2559]: W0727 21:04:31.763354    2559 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 27 21:04:31.767641 ip-10-0-131-133 kubenswrapper[2559]: W0727 21:04:31.763357    2559 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 27 21:04:31.767641 ip-10-0-131-133 kubenswrapper[2559]: W0727 21:04:31.763360    2559 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 27 21:04:31.767641 ip-10-0-131-133 kubenswrapper[2559]: W0727 21:04:31.763362    2559 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 27 21:04:31.767641 ip-10-0-131-133 kubenswrapper[2559]: W0727 21:04:31.763365    2559 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 27 21:04:31.767641 ip-10-0-131-133 kubenswrapper[2559]: W0727 21:04:31.763368    2559 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 27 21:04:31.767641 ip-10-0-131-133 kubenswrapper[2559]: W0727 21:04:31.763371    2559 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 27 21:04:31.767641 ip-10-0-131-133 kubenswrapper[2559]: W0727 21:04:31.763373    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 27 21:04:31.767641 ip-10-0-131-133 kubenswrapper[2559]: W0727 21:04:31.763377    2559 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 27 21:04:31.767641 ip-10-0-131-133 kubenswrapper[2559]: W0727 21:04:31.763379    2559 feature_gate.go:328] unrecognized feature gate: Example2
Jul 27 21:04:31.767641 ip-10-0-131-133 kubenswrapper[2559]: W0727 21:04:31.763382    2559 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 27 21:04:31.767641 ip-10-0-131-133 kubenswrapper[2559]: W0727 21:04:31.763385    2559 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 27 21:04:31.767641 ip-10-0-131-133 kubenswrapper[2559]: W0727 21:04:31.763389    2559 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 27 21:04:31.767641 ip-10-0-131-133 kubenswrapper[2559]: W0727 21:04:31.763392    2559 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 27 21:04:31.767641 ip-10-0-131-133 kubenswrapper[2559]: W0727 21:04:31.763394    2559 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 27 21:04:31.767641 ip-10-0-131-133 kubenswrapper[2559]: W0727 21:04:31.763397    2559 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 27 21:04:31.767641 ip-10-0-131-133 kubenswrapper[2559]: W0727 21:04:31.763400    2559 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 27 21:04:31.767641 ip-10-0-131-133 kubenswrapper[2559]: W0727 21:04:31.763409    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 27 21:04:31.768168 ip-10-0-131-133 kubenswrapper[2559]: W0727 21:04:31.763412    2559 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 27 21:04:31.768168 ip-10-0-131-133 kubenswrapper[2559]: W0727 21:04:31.763415    2559 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 27 21:04:31.768168 ip-10-0-131-133 kubenswrapper[2559]: W0727 21:04:31.763418    2559 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 27 21:04:31.768168 ip-10-0-131-133 kubenswrapper[2559]: W0727 21:04:31.763420    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 27 21:04:31.768168 ip-10-0-131-133 kubenswrapper[2559]: W0727 21:04:31.763423    2559 feature_gate.go:328] unrecognized feature gate: Example
Jul 27 21:04:31.768168 ip-10-0-131-133 kubenswrapper[2559]: W0727 21:04:31.763434    2559 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 27 21:04:31.768168 ip-10-0-131-133 kubenswrapper[2559]: W0727 21:04:31.763437    2559 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 27 21:04:31.768168 ip-10-0-131-133 kubenswrapper[2559]: W0727 21:04:31.763439    2559 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 27 21:04:31.768168 ip-10-0-131-133 kubenswrapper[2559]: W0727 21:04:31.763443    2559 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 27 21:04:31.768168 ip-10-0-131-133 kubenswrapper[2559]: W0727 21:04:31.763445    2559 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 27 21:04:31.768168 ip-10-0-131-133 kubenswrapper[2559]: W0727 21:04:31.763448    2559 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 27 21:04:31.768168 ip-10-0-131-133 kubenswrapper[2559]: W0727 21:04:31.763451    2559 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 27 21:04:31.768168 ip-10-0-131-133 kubenswrapper[2559]: W0727 21:04:31.763454    2559 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 27 21:04:31.768168 ip-10-0-131-133 kubenswrapper[2559]: W0727 21:04:31.763457    2559 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 27 21:04:31.768168 ip-10-0-131-133 kubenswrapper[2559]: W0727 21:04:31.763460    2559 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 27 21:04:31.768168 ip-10-0-131-133 kubenswrapper[2559]: W0727 21:04:31.763462    2559 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 27 21:04:31.768168 ip-10-0-131-133 kubenswrapper[2559]: W0727 21:04:31.763465    2559 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 27 21:04:31.768168 ip-10-0-131-133 kubenswrapper[2559]: W0727 21:04:31.763468    2559 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 27 21:04:31.768168 ip-10-0-131-133 kubenswrapper[2559]: W0727 21:04:31.763470    2559 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 27 21:04:31.768168 ip-10-0-131-133 kubenswrapper[2559]: W0727 21:04:31.763473    2559 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 27 21:04:31.768721 ip-10-0-131-133 kubenswrapper[2559]: W0727 21:04:31.763476    2559 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 27 21:04:31.768721 ip-10-0-131-133 kubenswrapper[2559]: W0727 21:04:31.763479    2559 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 27 21:04:31.768721 ip-10-0-131-133 kubenswrapper[2559]: W0727 21:04:31.763482    2559 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 27 21:04:31.768721 ip-10-0-131-133 kubenswrapper[2559]: W0727 21:04:31.763485    2559 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 27 21:04:31.768721 ip-10-0-131-133 kubenswrapper[2559]: W0727 21:04:31.763489    2559 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 27 21:04:31.768721 ip-10-0-131-133 kubenswrapper[2559]: W0727 21:04:31.763493    2559 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 27 21:04:31.768721 ip-10-0-131-133 kubenswrapper[2559]: W0727 21:04:31.763496    2559 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 27 21:04:31.768721 ip-10-0-131-133 kubenswrapper[2559]: W0727 21:04:31.763499    2559 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 27 21:04:31.768721 ip-10-0-131-133 kubenswrapper[2559]: W0727 21:04:31.763502    2559 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 27 21:04:31.768721 ip-10-0-131-133 kubenswrapper[2559]: W0727 21:04:31.763505    2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 27 21:04:31.768721 ip-10-0-131-133 kubenswrapper[2559]: W0727 21:04:31.763508    2559 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 27 21:04:31.768721 ip-10-0-131-133 kubenswrapper[2559]: W0727 21:04:31.763511    2559 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 27 21:04:31.768721 ip-10-0-131-133 kubenswrapper[2559]: W0727 21:04:31.763523    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 27 21:04:31.768721 ip-10-0-131-133 kubenswrapper[2559]: W0727 21:04:31.763526    2559 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 27 21:04:31.768721 ip-10-0-131-133 kubenswrapper[2559]: W0727 21:04:31.763530    2559 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 27 21:04:31.768721 ip-10-0-131-133 kubenswrapper[2559]: W0727 21:04:31.763534    2559 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 27 21:04:31.769153 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:04:31.763540    2559 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 27 21:04:31.769153 ip-10-0-131-133 kubenswrapper[2559]: W0727 21:04:31.763727    2559 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 27 21:04:31.769153 ip-10-0-131-133 kubenswrapper[2559]: W0727 21:04:31.763735    2559 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 27 21:04:31.769153 ip-10-0-131-133 kubenswrapper[2559]: W0727 21:04:31.763739    2559 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 27 21:04:31.769153 ip-10-0-131-133 kubenswrapper[2559]: W0727 21:04:31.763742    2559 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 27 21:04:31.769153 ip-10-0-131-133 kubenswrapper[2559]: W0727 21:04:31.763745    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 27 21:04:31.769153 ip-10-0-131-133 kubenswrapper[2559]: W0727 21:04:31.763748    2559 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 27 21:04:31.769153 ip-10-0-131-133 kubenswrapper[2559]: W0727 21:04:31.763751    2559 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 27 21:04:31.769153 ip-10-0-131-133 kubenswrapper[2559]: W0727 21:04:31.763754    2559 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 27 21:04:31.769153 ip-10-0-131-133 kubenswrapper[2559]: W0727 21:04:31.763758    2559 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 27 21:04:31.769153 ip-10-0-131-133 kubenswrapper[2559]: W0727 21:04:31.763761    2559 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 27 21:04:31.769153 ip-10-0-131-133 kubenswrapper[2559]: W0727 21:04:31.763764    2559 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 27 21:04:31.769153 ip-10-0-131-133 kubenswrapper[2559]: W0727 21:04:31.763768    2559 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 27 21:04:31.769153 ip-10-0-131-133 kubenswrapper[2559]: W0727 21:04:31.763771    2559 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 27 21:04:31.769153 ip-10-0-131-133 kubenswrapper[2559]: W0727 21:04:31.763774    2559 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 27 21:04:31.769569 ip-10-0-131-133 kubenswrapper[2559]: W0727 21:04:31.763779    2559 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 27 21:04:31.769569 ip-10-0-131-133 kubenswrapper[2559]: W0727 21:04:31.763782    2559 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 27 21:04:31.769569 ip-10-0-131-133 kubenswrapper[2559]: W0727 21:04:31.763785    2559 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 27 21:04:31.769569 ip-10-0-131-133 kubenswrapper[2559]: W0727 21:04:31.763788    2559 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 27 21:04:31.769569 ip-10-0-131-133 kubenswrapper[2559]: W0727 21:04:31.763791    2559 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 27 21:04:31.769569 ip-10-0-131-133 kubenswrapper[2559]: W0727 21:04:31.763793    2559 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 27 21:04:31.769569 ip-10-0-131-133 kubenswrapper[2559]: W0727 21:04:31.763797    2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 27 21:04:31.769569 ip-10-0-131-133 kubenswrapper[2559]: W0727 21:04:31.763799    2559 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 27 21:04:31.769569 ip-10-0-131-133 kubenswrapper[2559]: W0727 21:04:31.763802    2559 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 27 21:04:31.769569 ip-10-0-131-133 kubenswrapper[2559]: W0727 21:04:31.763805    2559 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 27 21:04:31.769569 ip-10-0-131-133 kubenswrapper[2559]: W0727 21:04:31.763808    2559 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 27 21:04:31.769569 ip-10-0-131-133 kubenswrapper[2559]: W0727 21:04:31.763813    2559 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 27 21:04:31.769569 ip-10-0-131-133 kubenswrapper[2559]: W0727 21:04:31.763816    2559 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 27 21:04:31.769569 ip-10-0-131-133 kubenswrapper[2559]: W0727 21:04:31.763819    2559 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 27 21:04:31.769569 ip-10-0-131-133 kubenswrapper[2559]: W0727 21:04:31.763828    2559 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 27 21:04:31.769569 ip-10-0-131-133 kubenswrapper[2559]: W0727 21:04:31.763832    2559 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 27 21:04:31.769569 ip-10-0-131-133 kubenswrapper[2559]: W0727 21:04:31.763835    2559 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 27 21:04:31.769569 ip-10-0-131-133 kubenswrapper[2559]: W0727 21:04:31.763838    2559 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 27 21:04:31.769569 ip-10-0-131-133 kubenswrapper[2559]: W0727 21:04:31.763841    2559 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 27 21:04:31.769569 ip-10-0-131-133 kubenswrapper[2559]: W0727 21:04:31.763843    2559 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 27 21:04:31.770094 ip-10-0-131-133 kubenswrapper[2559]: W0727 21:04:31.763846    2559 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 27 21:04:31.770094 ip-10-0-131-133 kubenswrapper[2559]: W0727 21:04:31.763849    2559 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 27 21:04:31.770094 ip-10-0-131-133 kubenswrapper[2559]: W0727 21:04:31.763852    2559 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 27 21:04:31.770094 ip-10-0-131-133 kubenswrapper[2559]: W0727 21:04:31.763854    2559 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 27 21:04:31.770094 ip-10-0-131-133 kubenswrapper[2559]: W0727 21:04:31.763857    2559 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 27 21:04:31.770094 ip-10-0-131-133 kubenswrapper[2559]: W0727 21:04:31.763860    2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 27 21:04:31.770094 ip-10-0-131-133 kubenswrapper[2559]: W0727 21:04:31.763863    2559 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 27 21:04:31.770094 ip-10-0-131-133 kubenswrapper[2559]: W0727 21:04:31.763866    2559 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 27 21:04:31.770094 ip-10-0-131-133 kubenswrapper[2559]: W0727 21:04:31.763869    2559 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 27 21:04:31.770094 ip-10-0-131-133 kubenswrapper[2559]: W0727 21:04:31.763871    2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 27 21:04:31.770094 ip-10-0-131-133 kubenswrapper[2559]: W0727 21:04:31.763874    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 27 21:04:31.770094 ip-10-0-131-133 kubenswrapper[2559]: W0727 21:04:31.763876    2559 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 27 21:04:31.770094 ip-10-0-131-133 kubenswrapper[2559]: W0727 21:04:31.763879    2559 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 27 21:04:31.770094 ip-10-0-131-133 kubenswrapper[2559]: W0727 21:04:31.763882    2559 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 27 21:04:31.770094 ip-10-0-131-133 kubenswrapper[2559]: W0727 21:04:31.763886    2559 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 27 21:04:31.770094 ip-10-0-131-133 kubenswrapper[2559]: W0727 21:04:31.763890    2559 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 27 21:04:31.770094 ip-10-0-131-133 kubenswrapper[2559]: W0727 21:04:31.763893    2559 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 27 21:04:31.770094 ip-10-0-131-133 kubenswrapper[2559]: W0727 21:04:31.763896    2559 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 27 21:04:31.770094 ip-10-0-131-133 kubenswrapper[2559]: W0727 21:04:31.763899    2559 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 27 21:04:31.770615 ip-10-0-131-133 kubenswrapper[2559]: W0727 21:04:31.763902    2559 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 27 21:04:31.770615 ip-10-0-131-133 kubenswrapper[2559]: W0727 21:04:31.763906    2559 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 27 21:04:31.770615 ip-10-0-131-133 kubenswrapper[2559]: W0727 21:04:31.763909    2559 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 27 21:04:31.770615 ip-10-0-131-133 kubenswrapper[2559]: W0727 21:04:31.763912    2559 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 27 21:04:31.770615 ip-10-0-131-133 kubenswrapper[2559]: W0727 21:04:31.763915    2559 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 27 21:04:31.770615 ip-10-0-131-133 kubenswrapper[2559]: W0727 21:04:31.763918    2559 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 27 21:04:31.770615 ip-10-0-131-133 kubenswrapper[2559]: W0727 21:04:31.763920    2559 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 27 21:04:31.770615 ip-10-0-131-133 kubenswrapper[2559]: W0727 21:04:31.763923    2559 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 27 21:04:31.770615 ip-10-0-131-133 kubenswrapper[2559]: W0727 21:04:31.763932    2559 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 27 21:04:31.770615 ip-10-0-131-133 kubenswrapper[2559]: W0727 21:04:31.763935    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 27 21:04:31.770615 ip-10-0-131-133 kubenswrapper[2559]: W0727 21:04:31.763938    2559 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 27 21:04:31.770615 ip-10-0-131-133 kubenswrapper[2559]: W0727 21:04:31.763941    2559 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 27 21:04:31.770615 ip-10-0-131-133 kubenswrapper[2559]: W0727 21:04:31.763944    2559 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 27 21:04:31.770615 ip-10-0-131-133 kubenswrapper[2559]: W0727 21:04:31.763946    2559 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 27 21:04:31.770615 ip-10-0-131-133 kubenswrapper[2559]: W0727 21:04:31.763949    2559 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 27 21:04:31.770615 ip-10-0-131-133 kubenswrapper[2559]: W0727 21:04:31.763952    2559 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 27 21:04:31.770615 ip-10-0-131-133 kubenswrapper[2559]: W0727 21:04:31.763954    2559 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 27 21:04:31.770615 ip-10-0-131-133 kubenswrapper[2559]: W0727 21:04:31.763964    2559 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 27 21:04:31.770615 ip-10-0-131-133 kubenswrapper[2559]: W0727 21:04:31.763967    2559 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 27 21:04:31.771118 ip-10-0-131-133 kubenswrapper[2559]: W0727 21:04:31.763970    2559 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 27 21:04:31.771118 ip-10-0-131-133 kubenswrapper[2559]: W0727 21:04:31.763973    2559 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 27 21:04:31.771118 ip-10-0-131-133 kubenswrapper[2559]: W0727 21:04:31.763975    2559 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 27 21:04:31.771118 ip-10-0-131-133 kubenswrapper[2559]: W0727 21:04:31.763978    2559 feature_gate.go:328] unrecognized feature gate: Example2
Jul 27 21:04:31.771118 ip-10-0-131-133 kubenswrapper[2559]: W0727 21:04:31.763981    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 27 21:04:31.771118 ip-10-0-131-133 kubenswrapper[2559]: W0727 21:04:31.763983    2559 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 27 21:04:31.771118 ip-10-0-131-133 kubenswrapper[2559]: W0727 21:04:31.763986    2559 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 27 21:04:31.771118 ip-10-0-131-133 kubenswrapper[2559]: W0727 21:04:31.763988    2559 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 27 21:04:31.771118 ip-10-0-131-133 kubenswrapper[2559]: W0727 21:04:31.763991    2559 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 27 21:04:31.771118 ip-10-0-131-133 kubenswrapper[2559]: W0727 21:04:31.763994    2559 feature_gate.go:328] unrecognized feature gate: Example
Jul 27 21:04:31.771118 ip-10-0-131-133 kubenswrapper[2559]: W0727 21:04:31.763997    2559 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 27 21:04:31.771118 ip-10-0-131-133 kubenswrapper[2559]: W0727 21:04:31.764000    2559 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 27 21:04:31.771118 ip-10-0-131-133 kubenswrapper[2559]: W0727 21:04:31.764003    2559 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 27 21:04:31.771118 ip-10-0-131-133 kubenswrapper[2559]: W0727 21:04:31.764006    2559 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 27 21:04:31.771118 ip-10-0-131-133 kubenswrapper[2559]: W0727 21:04:31.764009    2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 27 21:04:31.771118 ip-10-0-131-133 kubenswrapper[2559]: W0727 21:04:31.764012    2559 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 27 21:04:31.771118 ip-10-0-131-133 kubenswrapper[2559]: W0727 21:04:31.764014    2559 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 27 21:04:31.771118 ip-10-0-131-133 kubenswrapper[2559]: W0727 21:04:31.764018    2559 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 27 21:04:31.771118 ip-10-0-131-133 kubenswrapper[2559]: W0727 21:04:31.764020    2559 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 27 21:04:31.771118 ip-10-0-131-133 kubenswrapper[2559]: W0727 21:04:31.764023    2559 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 27 21:04:31.771820 ip-10-0-131-133 kubenswrapper[2559]: W0727 21:04:31.764026    2559 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 27 21:04:31.771820 ip-10-0-131-133 kubenswrapper[2559]: W0727 21:04:31.764028    2559 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 27 21:04:31.771820 ip-10-0-131-133 kubenswrapper[2559]: W0727 21:04:31.764037    2559 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 27 21:04:31.771820 ip-10-0-131-133 kubenswrapper[2559]: W0727 21:04:31.764040    2559 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 27 21:04:31.771820 ip-10-0-131-133 kubenswrapper[2559]: W0727 21:04:31.764043    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 27 21:04:31.771820 ip-10-0-131-133 kubenswrapper[2559]: W0727 21:04:31.764045    2559 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 27 21:04:31.771820 ip-10-0-131-133 kubenswrapper[2559]: W0727 21:04:31.764048    2559 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 27 21:04:31.771820 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:04:31.764053    2559 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 27 21:04:31.771820 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:04:31.764204    2559 server.go:962] "Client rotation is on, will bootstrap in background"
Jul 27 21:04:31.771820 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:04:31.766702    2559 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Jul 27 21:04:31.771820 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:04:31.768170    2559 server.go:1019] "Starting client certificate rotation"
Jul 27 21:04:31.771820 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:04:31.768332    2559 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 27 21:04:31.771820 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:04:31.768367    2559 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 27 21:04:31.773988 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:04:31.773961    2559 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 27 21:04:31.780734 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:04:31.780706    2559 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 27 21:04:31.790590 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:04:31.790570    2559 log.go:25] "Validated CRI v1 runtime API"
Jul 27 21:04:31.793037 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:04:31.793022    2559 log.go:25] "Validated CRI v1 image API"
Jul 27 21:04:31.793990 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:04:31.793974    2559 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Jul 27 21:04:31.797911 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:04:31.797889    2559 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 7d99e333-345e-45cd-8734-df52516f3bc3:/dev/nvme0n1p4 fbfa0886-6d37-4451-93d8-fba466f2b1ec:/dev/nvme0n1p3]
Jul 27 21:04:31.797983 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:04:31.797912    2559 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Jul 27 21:04:31.801540 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:04:31.801519    2559 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 27 21:04:31.803019 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:04:31.802900    2559 manager.go:216] Machine: {Timestamp:2026-07-27 21:04:31.802342594 +0000 UTC m=+0.381526609 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3109392 MemoryCapacity:33164496896 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2492fbba93cb070b34e547d189a6f1 SystemUUID:ec2492fb-ba93-cb07-0b34-e547d189a6f1 BootID:ef7c8450-df38-4995-b26e-78d8606de5bd Filesystems:[{Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632902656 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582250496 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582246400 Type:vfs Inodes:4048400 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:f3:88:66:a7:31 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:f3:88:66:a7:31 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:ba:c6:0f:2a:42:2e Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164496896 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Jul 27 21:04:31.803019 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:04:31.803015    2559 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Jul 27 21:04:31.803124 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:04:31.803103    2559 manager.go:232] Version: {KernelVersion:5.14.0-570.129.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260717-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Jul 27 21:04:31.806796 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:04:31.806765    2559 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Jul 27 21:04:31.806948 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:04:31.806800    2559 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-131-133.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Jul 27 21:04:31.807720 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:04:31.807709    2559 topology_manager.go:138] "Creating topology manager with none policy"
Jul 27 21:04:31.807766 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:04:31.807723    2559 container_manager_linux.go:309] "Creating device plugin manager"
Jul 27 21:04:31.807766 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:04:31.807732    2559 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 27 21:04:31.807766 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:04:31.807748    2559 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 27 21:04:31.808690 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:04:31.808679    2559 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Jul 27 21:04:31.810786 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:04:31.810773    2559 state_mem.go:36] "Initialized new in-memory state store"
Jul 27 21:04:31.810924 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:04:31.810914    2559 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Jul 27 21:04:31.810968 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:04:31.810960    2559 kubelet.go:486] "Attempting to sync node with API server"
Jul 27 21:04:31.811003 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:04:31.810976    2559 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Jul 27 21:04:31.811003 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:04:31.811002    2559 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Jul 27 21:04:31.811065 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:04:31.811014    2559 kubelet.go:398] "Adding apiserver pod source"
Jul 27 21:04:31.811065 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:04:31.811029    2559 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Jul 27 21:04:31.811468 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:04:31.811455    2559 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 27 21:04:31.811526 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:04:31.811480    2559 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 27 21:04:31.814689 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:04:31.814653    2559 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.10-2.rhaos4.21.git83ee25b.el9" apiVersion="v1"
Jul 27 21:04:31.817680 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:04:31.817662    2559 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Jul 27 21:04:31.817771 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:04:31.817704    2559 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Jul 27 21:04:31.819658 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:04:31.819642    2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Jul 27 21:04:31.819736 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:04:31.819663    2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Jul 27 21:04:31.819736 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:04:31.819673    2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Jul 27 21:04:31.819736 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:04:31.819681    2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Jul 27 21:04:31.819736 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:04:31.819694    2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Jul 27 21:04:31.819736 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:04:31.819703    2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Jul 27 21:04:31.819736 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:04:31.819715    2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Jul 27 21:04:31.819736 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:04:31.819724    2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Jul 27 21:04:31.819736 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:04:31.819735    2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Jul 27 21:04:31.820020 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:04:31.819744    2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Jul 27 21:04:31.820020 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:04:31.819769    2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Jul 27 21:04:31.820020 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:04:31.819778    2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Jul 27 21:04:31.820736 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:04:31.820724    2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Jul 27 21:04:31.820790 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:04:31.820739    2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Jul 27 21:04:31.820945 ip-10-0-131-133 kubenswrapper[2559]: E0727 21:04:31.820923    2559 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Jul 27 21:04:31.821010 ip-10-0-131-133 kubenswrapper[2559]: E0727 21:04:31.820923    2559 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-131-133.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Jul 27 21:04:31.822691 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:04:31.822629    2559 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-131-133.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Jul 27 21:04:31.824197 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:04:31.824179    2559 server.go:1268] "Started kubelet"
Jul 27 21:04:31.824337 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:04:31.824276    2559 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Jul 27 21:04:31.824398 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:04:31.824361    2559 server_v1.go:49] "podresources" method="list" useActivePods=true
Jul 27 21:04:31.824473 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:04:31.824309    2559 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Jul 27 21:04:31.825213 ip-10-0-131-133 systemd[1]: Started Kubernetes Kubelet.
Jul 27 21:04:31.825872 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:04:31.825685    2559 server.go:310] "Adding debug handlers to kubelet server"
Jul 27 21:04:31.826263 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:04:31.826051    2559 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Jul 27 21:04:31.829359 ip-10-0-131-133 kubenswrapper[2559]: E0727 21:04:31.829340    2559 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Jul 27 21:04:31.835533 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:04:31.834820    2559 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Jul 27 21:04:31.835533 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:04:31.834836    2559 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Jul 27 21:04:31.835533 ip-10-0-131-133 kubenswrapper[2559]: E0727 21:04:31.835015    2559 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-133.ec2.internal\" not found"
Jul 27 21:04:31.835747 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:04:31.835602    2559 volume_manager.go:311] "The desired_state_of_world populator starts"
Jul 27 21:04:31.835747 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:04:31.835617    2559 volume_manager.go:313] "Starting Kubelet Volume Manager"
Jul 27 21:04:31.835747 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:04:31.835704    2559 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Jul 27 21:04:31.835876 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:04:31.835764    2559 reconstruct.go:97] "Volume reconstruction finished"
Jul 27 21:04:31.835876 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:04:31.835771    2559 reconciler.go:29] "Reconciler: start to sync state"
Jul 27 21:04:31.836062 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:04:31.835985    2559 factory.go:55] Registering systemd factory
Jul 27 21:04:31.836062 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:04:31.836005    2559 factory.go:223] Registration of the systemd container factory successfully
Jul 27 21:04:31.836322 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:04:31.836304    2559 factory.go:153] Registering CRI-O factory
Jul 27 21:04:31.836322 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:04:31.836322    2559 factory.go:223] Registration of the crio container factory successfully
Jul 27 21:04:31.836423 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:04:31.836373    2559 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Jul 27 21:04:31.836423 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:04:31.836394    2559 factory.go:103] Registering Raw factory
Jul 27 21:04:31.836423 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:04:31.836405    2559 manager.go:1196] Started watching for new ooms in manager
Jul 27 21:04:31.837121 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:04:31.837089    2559 manager.go:349] Starting recovery of all containers
Jul 27 21:04:31.839627 ip-10-0-131-133 kubenswrapper[2559]: E0727 21:04:31.839592    2559 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-131-133.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Jul 27 21:04:31.839842 ip-10-0-131-133 kubenswrapper[2559]: E0727 21:04:31.839817    2559 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Jul 27 21:04:31.840978 ip-10-0-131-133 kubenswrapper[2559]: E0727 21:04:31.839945    2559 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-131-133.ec2.internal.18c64174b5fb7e14  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-131-133.ec2.internal,UID:ip-10-0-131-133.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-131-133.ec2.internal,},FirstTimestamp:2026-07-27 21:04:31.824150036 +0000 UTC m=+0.403334037,LastTimestamp:2026-07-27 21:04:31.824150036 +0000 UTC m=+0.403334037,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-131-133.ec2.internal,}"
Jul 27 21:04:31.850915 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:04:31.850787    2559 manager.go:354] Recovery completed
Jul 27 21:04:31.855381 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:04:31.855365    2559 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 27 21:04:31.857378 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:04:31.857353    2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-133.ec2.internal" event="NodeHasSufficientMemory"
Jul 27 21:04:31.857437 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:04:31.857393    2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-133.ec2.internal" event="NodeHasNoDiskPressure"
Jul 27 21:04:31.857437 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:04:31.857405    2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-133.ec2.internal" event="NodeHasSufficientPID"
Jul 27 21:04:31.857955 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:04:31.857943    2559 cpu_manager.go:222] "Starting CPU manager" policy="none"
Jul 27 21:04:31.857955 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:04:31.857953    2559 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Jul 27 21:04:31.858050 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:04:31.857957    2559 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 27 21:04:31.858050 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:04:31.857969    2559 state_mem.go:36] "Initialized new in-memory state store"
Jul 27 21:04:31.858709 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:04:31.858691    2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-133.ec2.internal" event="NodeHasSufficientMemory"
Jul 27 21:04:31.858774 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:04:31.858722    2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-133.ec2.internal" event="NodeHasNoDiskPressure"
Jul 27 21:04:31.858774 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:04:31.858732    2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-133.ec2.internal" event="NodeHasSufficientPID"
Jul 27 21:04:31.859802 ip-10-0-131-133 kubenswrapper[2559]: E0727 21:04:31.859711    2559 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-131-133.ec2.internal.18c64174b7f68323  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-131-133.ec2.internal,UID:ip-10-0-131-133.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-131-133.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-131-133.ec2.internal,},FirstTimestamp:2026-07-27 21:04:31.857378083 +0000 UTC m=+0.436562084,LastTimestamp:2026-07-27 21:04:31.857378083 +0000 UTC m=+0.436562084,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-131-133.ec2.internal,}"
Jul 27 21:04:31.859941 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:04:31.859925    2559 policy_none.go:49] "None policy: Start"
Jul 27 21:04:31.859997 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:04:31.859959    2559 memory_manager.go:187] "Starting memorymanager" policy="None"
Jul 27 21:04:31.859997 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:04:31.859976    2559 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Jul 27 21:04:31.861199 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:04:31.861187    2559 policy_none.go:47] "Start"
Jul 27 21:04:31.861675 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:04:31.861658    2559 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-plldg"
Jul 27 21:04:31.868376 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:04:31.868356    2559 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-plldg"
Jul 27 21:04:31.868642 ip-10-0-131-133 kubenswrapper[2559]: E0727 21:04:31.868568    2559 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-131-133.ec2.internal.18c64174b7f6d555  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-131-133.ec2.internal,UID:ip-10-0-131-133.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-131-133.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-131-133.ec2.internal,},FirstTimestamp:2026-07-27 21:04:31.857399125 +0000 UTC m=+0.436583127,LastTimestamp:2026-07-27 21:04:31.857399125 +0000 UTC m=+0.436583127,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-131-133.ec2.internal,}"
Jul 27 21:04:31.902973 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:04:31.902944    2559 manager.go:337] "Starting Device Plugin manager"
Jul 27 21:04:31.903108 ip-10-0-131-133 kubenswrapper[2559]: E0727 21:04:31.902998    2559 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Jul 27 21:04:31.903108 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:04:31.903012    2559 server.go:87] "Starting device plugin registration server"
Jul 27 21:04:31.903360 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:04:31.903344    2559 eviction_manager.go:189] "Eviction manager: starting control loop"
Jul 27 21:04:31.903433 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:04:31.903361    2559 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Jul 27 21:04:31.903501 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:04:31.903484    2559 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Jul 27 21:04:31.903587 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:04:31.903575    2559 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Jul 27 21:04:31.903637 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:04:31.903588    2559 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Jul 27 21:04:31.904119 ip-10-0-131-133 kubenswrapper[2559]: E0727 21:04:31.904098    2559 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Jul 27 21:04:31.904207 ip-10-0-131-133 kubenswrapper[2559]: E0727 21:04:31.904154    2559 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-131-133.ec2.internal\" not found"
Jul 27 21:04:31.974051 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:04:31.974008    2559 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Jul 27 21:04:31.975496 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:04:31.975403    2559 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Jul 27 21:04:31.975496 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:04:31.975451    2559 status_manager.go:244] "Starting to sync pod status with apiserver"
Jul 27 21:04:31.975496 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:04:31.975479    2559 kubelet.go:2448] "Starting kubelet main sync loop"
Jul 27 21:04:31.975666 ip-10-0-131-133 kubenswrapper[2559]: E0727 21:04:31.975524    2559 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Jul 27 21:04:31.978267 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:04:31.978236    2559 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 27 21:04:32.004442 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:04:32.004402    2559 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 27 21:04:32.005469 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:04:32.005452    2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-133.ec2.internal" event="NodeHasSufficientMemory"
Jul 27 21:04:32.005576 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:04:32.005489    2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-133.ec2.internal" event="NodeHasNoDiskPressure"
Jul 27 21:04:32.005576 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:04:32.005503    2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-133.ec2.internal" event="NodeHasSufficientPID"
Jul 27 21:04:32.005576 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:04:32.005537    2559 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-131-133.ec2.internal"
Jul 27 21:04:32.012835 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:04:32.012815    2559 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-131-133.ec2.internal"
Jul 27 21:04:32.012907 ip-10-0-131-133 kubenswrapper[2559]: E0727 21:04:32.012842    2559 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-131-133.ec2.internal\": node \"ip-10-0-131-133.ec2.internal\" not found"
Jul 27 21:04:32.046692 ip-10-0-131-133 kubenswrapper[2559]: E0727 21:04:32.046655    2559 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-133.ec2.internal\" not found"
Jul 27 21:04:32.075782 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:04:32.075697    2559 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-133.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-131-133.ec2.internal"]
Jul 27 21:04:32.075782 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:04:32.075780    2559 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 27 21:04:32.076869 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:04:32.076851    2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-133.ec2.internal" event="NodeHasSufficientMemory"
Jul 27 21:04:32.076971 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:04:32.076887    2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-133.ec2.internal" event="NodeHasNoDiskPressure"
Jul 27 21:04:32.076971 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:04:32.076902    2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-133.ec2.internal" event="NodeHasSufficientPID"
Jul 27 21:04:32.078504 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:04:32.078487    2559 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 27 21:04:32.078638 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:04:32.078621    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-133.ec2.internal"
Jul 27 21:04:32.078695 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:04:32.078657    2559 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 27 21:04:32.079308 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:04:32.079283    2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-133.ec2.internal" event="NodeHasSufficientMemory"
Jul 27 21:04:32.079308 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:04:32.079298    2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-133.ec2.internal" event="NodeHasSufficientMemory"
Jul 27 21:04:32.079420 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:04:32.079312    2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-133.ec2.internal" event="NodeHasNoDiskPressure"
Jul 27 21:04:32.079420 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:04:32.079320    2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-133.ec2.internal" event="NodeHasNoDiskPressure"
Jul 27 21:04:32.079420 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:04:32.079325    2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-133.ec2.internal" event="NodeHasSufficientPID"
Jul 27 21:04:32.079420 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:04:32.079331    2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-133.ec2.internal" event="NodeHasSufficientPID"
Jul 27 21:04:32.080535 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:04:32.080514    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-133.ec2.internal"
Jul 27 21:04:32.080639 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:04:32.080543    2559 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 27 21:04:32.081253 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:04:32.081236    2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-133.ec2.internal" event="NodeHasSufficientMemory"
Jul 27 21:04:32.081347 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:04:32.081268    2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-133.ec2.internal" event="NodeHasNoDiskPressure"
Jul 27 21:04:32.081347 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:04:32.081282    2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-133.ec2.internal" event="NodeHasSufficientPID"
Jul 27 21:04:32.105230 ip-10-0-131-133 kubenswrapper[2559]: E0727 21:04:32.105201    2559 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-131-133.ec2.internal\" not found" node="ip-10-0-131-133.ec2.internal"
Jul 27 21:04:32.110074 ip-10-0-131-133 kubenswrapper[2559]: E0727 21:04:32.110056    2559 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-131-133.ec2.internal\" not found" node="ip-10-0-131-133.ec2.internal"
Jul 27 21:04:32.137844 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:04:32.137815    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/47a35874b1255611552eed095cad3eb8-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-131-133.ec2.internal\" (UID: \"47a35874b1255611552eed095cad3eb8\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-133.ec2.internal"
Jul 27 21:04:32.137963 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:04:32.137852    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/47a35874b1255611552eed095cad3eb8-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-131-133.ec2.internal\" (UID: \"47a35874b1255611552eed095cad3eb8\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-133.ec2.internal"
Jul 27 21:04:32.137963 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:04:32.137873    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/c500a9e213c47a39763b09a5e1d404dc-config\") pod \"kube-apiserver-proxy-ip-10-0-131-133.ec2.internal\" (UID: \"c500a9e213c47a39763b09a5e1d404dc\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-131-133.ec2.internal"
Jul 27 21:04:32.147229 ip-10-0-131-133 kubenswrapper[2559]: E0727 21:04:32.147206    2559 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-133.ec2.internal\" not found"
Jul 27 21:04:32.238036 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:04:32.237997    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/47a35874b1255611552eed095cad3eb8-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-131-133.ec2.internal\" (UID: \"47a35874b1255611552eed095cad3eb8\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-133.ec2.internal"
Jul 27 21:04:32.238122 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:04:32.238043    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/47a35874b1255611552eed095cad3eb8-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-131-133.ec2.internal\" (UID: \"47a35874b1255611552eed095cad3eb8\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-133.ec2.internal"
Jul 27 21:04:32.238122 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:04:32.238061    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/c500a9e213c47a39763b09a5e1d404dc-config\") pod \"kube-apiserver-proxy-ip-10-0-131-133.ec2.internal\" (UID: \"c500a9e213c47a39763b09a5e1d404dc\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-131-133.ec2.internal"
Jul 27 21:04:32.238122 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:04:32.238078    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/47a35874b1255611552eed095cad3eb8-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-131-133.ec2.internal\" (UID: \"47a35874b1255611552eed095cad3eb8\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-133.ec2.internal"
Jul 27 21:04:32.238122 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:04:32.238093    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/c500a9e213c47a39763b09a5e1d404dc-config\") pod \"kube-apiserver-proxy-ip-10-0-131-133.ec2.internal\" (UID: \"c500a9e213c47a39763b09a5e1d404dc\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-131-133.ec2.internal"
Jul 27 21:04:32.238122 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:04:32.238093    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/47a35874b1255611552eed095cad3eb8-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-131-133.ec2.internal\" (UID: \"47a35874b1255611552eed095cad3eb8\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-133.ec2.internal"
Jul 27 21:04:32.247659 ip-10-0-131-133 kubenswrapper[2559]: E0727 21:04:32.247636    2559 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-133.ec2.internal\" not found"
Jul 27 21:04:32.348297 ip-10-0-131-133 kubenswrapper[2559]: E0727 21:04:32.348205    2559 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-133.ec2.internal\" not found"
Jul 27 21:04:32.413380 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:04:32.413336    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-133.ec2.internal"
Jul 27 21:04:32.414529 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:04:32.414511    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-133.ec2.internal"
Jul 27 21:04:32.448337 ip-10-0-131-133 kubenswrapper[2559]: E0727 21:04:32.448294    2559 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-133.ec2.internal\" not found"
Jul 27 21:04:32.548742 ip-10-0-131-133 kubenswrapper[2559]: E0727 21:04:32.548685    2559 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-133.ec2.internal\" not found"
Jul 27 21:04:32.649823 ip-10-0-131-133 kubenswrapper[2559]: E0727 21:04:32.649733    2559 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-133.ec2.internal\" not found"
Jul 27 21:04:32.750336 ip-10-0-131-133 kubenswrapper[2559]: E0727 21:04:32.750294    2559 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-133.ec2.internal\" not found"
Jul 27 21:04:32.767487 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:04:32.767452    2559 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Jul 27 21:04:32.767684 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:04:32.767646    2559 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Jul 27 21:04:32.850952 ip-10-0-131-133 kubenswrapper[2559]: E0727 21:04:32.850925    2559 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-133.ec2.internal\" not found"
Jul 27 21:04:32.873274 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:04:32.873235    2559 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-26 20:59:32 +0000 UTC" deadline="2028-03-21 13:29:58.398204031 +0000 UTC"
Jul 27 21:04:32.873274 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:04:32.873271    2559 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14464h25m25.524937742s"
Jul 27 21:04:32.951421 ip-10-0-131-133 kubenswrapper[2559]: E0727 21:04:32.951346    2559 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-133.ec2.internal\" not found"
Jul 27 21:04:32.974495 ip-10-0-131-133 kubenswrapper[2559]: W0727 21:04:32.974460    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod47a35874b1255611552eed095cad3eb8.slice/crio-3260d4f77f1566540d2a7c1c25801ac373a519a743dcd4fadf847c5a33fab314 WatchSource:0}: Error finding container 3260d4f77f1566540d2a7c1c25801ac373a519a743dcd4fadf847c5a33fab314: Status 404 returned error can't find the container with id 3260d4f77f1566540d2a7c1c25801ac373a519a743dcd4fadf847c5a33fab314
Jul 27 21:04:32.974849 ip-10-0-131-133 kubenswrapper[2559]: W0727 21:04:32.974832    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc500a9e213c47a39763b09a5e1d404dc.slice/crio-86d409f6ab56d6e9a39124b2073f8f8db7c835df23cf757ecebb359017b0b03c WatchSource:0}: Error finding container 86d409f6ab56d6e9a39124b2073f8f8db7c835df23cf757ecebb359017b0b03c: Status 404 returned error can't find the container with id 86d409f6ab56d6e9a39124b2073f8f8db7c835df23cf757ecebb359017b0b03c
Jul 27 21:04:32.979979 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:04:32.979963    2559 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 27 21:04:32.991351 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:04:32.991302    2559 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 27 21:04:32.992629 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:04:32.992606    2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-133.ec2.internal" event="NodeHasSufficientMemory"
Jul 27 21:04:32.992735 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:04:32.992642    2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-133.ec2.internal" event="NodeHasNoDiskPressure"
Jul 27 21:04:32.992735 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:04:32.992657    2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-133.ec2.internal" event="NodeHasSufficientPID"
Jul 27 21:04:33.006371 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:04:33.006339    2559 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 27 21:04:33.052482 ip-10-0-131-133 kubenswrapper[2559]: E0727 21:04:33.052443    2559 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-133.ec2.internal\" not found"
Jul 27 21:04:33.096413 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:04:33.096383    2559 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 27 21:04:33.136322 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:04:33.136291    2559 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-133.ec2.internal"
Jul 27 21:04:33.150614 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:04:33.150590    2559 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 27 21:04:33.152908 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:04:33.152893    2559 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-133.ec2.internal"
Jul 27 21:04:33.162088 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:04:33.162065    2559 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 27 21:04:33.402510 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:04:33.402431    2559 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 27 21:04:33.813125 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:04:33.813034    2559 apiserver.go:52] "Watching apiserver"
Jul 27 21:04:33.819377 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:04:33.819341    2559 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Jul 27 21:04:33.819774 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:04:33.819749    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kube-system/konnectivity-agent-tfvtq","openshift-image-registry/node-ca-qxzfs","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-133.ec2.internal","openshift-multus/multus-additional-cni-plugins-2pbtk","openshift-multus/multus-dchr6","openshift-network-diagnostics/network-check-target-hmjv6","kube-system/global-pull-secret-syncer-rxl8w","kube-system/kube-apiserver-proxy-ip-10-0-131-133.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gs462","openshift-cluster-node-tuning-operator/tuned-tfsvr","openshift-dns/node-resolver-fvbmc","openshift-multus/network-metrics-daemon-pr4c2","openshift-network-operator/iptables-alerter-cd6w9","openshift-ovn-kubernetes/ovnkube-node-qs89h"]
Jul 27 21:04:33.821496 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:04:33.821476    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rxl8w"
Jul 27 21:04:33.821597 ip-10-0-131-133 kubenswrapper[2559]: E0727 21:04:33.821564    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rxl8w" podUID="00ef4d6d-0b8e-4ece-ba97-36a9e236ec53"
Jul 27 21:04:33.822668 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:04:33.822649    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-cd6w9"
Jul 27 21:04:33.824276 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:04:33.824258    2559 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 27 21:04:33.824626 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:04:33.824605    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-qs89h"
Jul 27 21:04:33.825010 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:04:33.824990    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Jul 27 21:04:33.825455 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:04:33.825434    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-lx9fb\""
Jul 27 21:04:33.825744 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:04:33.825727    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Jul 27 21:04:33.825808 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:04:33.825744    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Jul 27 21:04:33.826066 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:04:33.826051    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-2pbtk"
Jul 27 21:04:33.826706 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:04:33.826688    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Jul 27 21:04:33.827992 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:04:33.827737    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-w7zbc\""
Jul 27 21:04:33.828465 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:04:33.828441    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Jul 27 21:04:33.828606 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:04:33.828516    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Jul 27 21:04:33.828722 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:04:33.828706    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Jul 27 21:04:33.828804 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:04:33.828446    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Jul 27 21:04:33.829094 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:04:33.829072    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Jul 27 21:04:33.829715 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:04:33.829691    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Jul 27 21:04:33.829866 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:04:33.829842    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Jul 27 21:04:33.830002 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:04:33.829980    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Jul 27 21:04:33.830088 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:04:33.830072    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-dchr6"
Jul 27 21:04:33.830143 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:04:33.830084    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-dtzg5\""
Jul 27 21:04:33.830194 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:04:33.830170    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Jul 27 21:04:33.830281 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:04:33.830078    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Jul 27 21:04:33.831589 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:04:33.831558    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hmjv6"
Jul 27 21:04:33.831685 ip-10-0-131-133 kubenswrapper[2559]: E0727 21:04:33.831654    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hmjv6" podUID="95b2123e-132a-42a8-8edb-33ce19c2df51"
Jul 27 21:04:33.832623 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:04:33.832604    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-5z67m\""
Jul 27 21:04:33.832742 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:04:33.832657    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Jul 27 21:04:33.833190 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:04:33.833168    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gs462"
Jul 27 21:04:33.834985 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:04:33.834962    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-tfvtq"
Jul 27 21:04:33.835147 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:04:33.835130    2559 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Jul 27 21:04:33.835914 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:04:33.835894    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Jul 27 21:04:33.836112 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:04:33.836092    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Jul 27 21:04:33.836296 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:04:33.836270    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Jul 27 21:04:33.836602 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:04:33.836579    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-8qv7t\""
Jul 27 21:04:33.837181 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:04:33.837160    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-qxzfs"
Jul 27 21:04:33.837407 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:04:33.837389    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Jul 27 21:04:33.837674 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:04:33.837437    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Jul 27 21:04:33.837909 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:04:33.837872    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-7hkr2\""
Jul 27 21:04:33.838608 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:04:33.838590    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-tfsvr"
Jul 27 21:04:33.839224 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:04:33.839208    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Jul 27 21:04:33.839754 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:04:33.839735    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Jul 27 21:04:33.839884 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:04:33.839865    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Jul 27 21:04:33.839947 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:04:33.839936    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-2tz4d\""
Jul 27 21:04:33.840209 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:04:33.840190    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-fvbmc"
Jul 27 21:04:33.841569 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:04:33.841506    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Jul 27 21:04:33.841569 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:04:33.841531    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Jul 27 21:04:33.841793 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:04:33.841725    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pr4c2"
Jul 27 21:04:33.841862 ip-10-0-131-133 kubenswrapper[2559]: E0727 21:04:33.841788    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pr4c2" podUID="5851de39-848d-4a52-bc17-fefe1bf5407a"
Jul 27 21:04:33.841862 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:04:33.841840    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-4nkch\""
Jul 27 21:04:33.843124 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:04:33.843046    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Jul 27 21:04:33.844826 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:04:33.844775    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Jul 27 21:04:33.845444 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:04:33.845418    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-hj6cq\""
Jul 27 21:04:33.846682 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:04:33.846469    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/3d80fd9e-83d7-4c96-b196-1c9afac0bad4-registration-dir\") pod \"aws-ebs-csi-driver-node-gs462\" (UID: \"3d80fd9e-83d7-4c96-b196-1c9afac0bad4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gs462"
Jul 27 21:04:33.846682 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:04:33.846509    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/4ff24566-f9aa-4eab-8795-a2808831cec1-konnectivity-ca\") pod \"konnectivity-agent-tfvtq\" (UID: \"4ff24566-f9aa-4eab-8795-a2808831cec1\") " pod="kube-system/konnectivity-agent-tfvtq"
Jul 27 21:04:33.846682 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:04:33.846542    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7f66b2c1-260e-4a55-902b-c4a060b723f9-var-lib-openvswitch\") pod \"ovnkube-node-qs89h\" (UID: \"7f66b2c1-260e-4a55-902b-c4a060b723f9\") " pod="openshift-ovn-kubernetes/ovnkube-node-qs89h"
Jul 27 21:04:33.846682 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:04:33.846567    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/7f66b2c1-260e-4a55-902b-c4a060b723f9-host-cni-netd\") pod \"ovnkube-node-qs89h\" (UID: \"7f66b2c1-260e-4a55-902b-c4a060b723f9\") " pod="openshift-ovn-kubernetes/ovnkube-node-qs89h"
Jul 27 21:04:33.846682 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:04:33.846592    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/6b7af06d-65e7-45ca-80ef-e88bcee2282a-host-run-netns\") pod \"multus-dchr6\" (UID: \"6b7af06d-65e7-45ca-80ef-e88bcee2282a\") " pod="openshift-multus/multus-dchr6"
Jul 27 21:04:33.846682 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:04:33.846616    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j6h2z\" (UniqueName: \"kubernetes.io/projected/6b7af06d-65e7-45ca-80ef-e88bcee2282a-kube-api-access-j6h2z\") pod \"multus-dchr6\" (UID: \"6b7af06d-65e7-45ca-80ef-e88bcee2282a\") " pod="openshift-multus/multus-dchr6"
Jul 27 21:04:33.846682 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:04:33.846641    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mtwdg\" (UniqueName: \"kubernetes.io/projected/95b2123e-132a-42a8-8edb-33ce19c2df51-kube-api-access-mtwdg\") pod \"network-check-target-hmjv6\" (UID: \"95b2123e-132a-42a8-8edb-33ce19c2df51\") " pod="openshift-network-diagnostics/network-check-target-hmjv6"
Jul 27 21:04:33.847072 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:04:33.846711    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/00ef4d6d-0b8e-4ece-ba97-36a9e236ec53-kubelet-config\") pod \"global-pull-secret-syncer-rxl8w\" (UID: \"00ef4d6d-0b8e-4ece-ba97-36a9e236ec53\") " pod="kube-system/global-pull-secret-syncer-rxl8w"
Jul 27 21:04:33.847072 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:04:33.846735    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/00ef4d6d-0b8e-4ece-ba97-36a9e236ec53-dbus\") pod \"global-pull-secret-syncer-rxl8w\" (UID: \"00ef4d6d-0b8e-4ece-ba97-36a9e236ec53\") " pod="kube-system/global-pull-secret-syncer-rxl8w"
Jul 27 21:04:33.847072 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:04:33.846762    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/4275e4fb-a8e5-43a5-8b8c-ed4cc2094674-cnibin\") pod \"multus-additional-cni-plugins-2pbtk\" (UID: \"4275e4fb-a8e5-43a5-8b8c-ed4cc2094674\") " pod="openshift-multus/multus-additional-cni-plugins-2pbtk"
Jul 27 21:04:33.847072 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:04:33.846785    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/3d80fd9e-83d7-4c96-b196-1c9afac0bad4-socket-dir\") pod \"aws-ebs-csi-driver-node-gs462\" (UID: \"3d80fd9e-83d7-4c96-b196-1c9afac0bad4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gs462"
Jul 27 21:04:33.847072 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:04:33.846812    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-np6md\" (UniqueName: \"kubernetes.io/projected/3d80fd9e-83d7-4c96-b196-1c9afac0bad4-kube-api-access-np6md\") pod \"aws-ebs-csi-driver-node-gs462\" (UID: \"3d80fd9e-83d7-4c96-b196-1c9afac0bad4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gs462"
Jul 27 21:04:33.847072 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:04:33.846840    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/6b7af06d-65e7-45ca-80ef-e88bcee2282a-hostroot\") pod \"multus-dchr6\" (UID: \"6b7af06d-65e7-45ca-80ef-e88bcee2282a\") " pod="openshift-multus/multus-dchr6"
Jul 27 21:04:33.847072 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:04:33.846863    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/3d80fd9e-83d7-4c96-b196-1c9afac0bad4-etc-selinux\") pod \"aws-ebs-csi-driver-node-gs462\" (UID: \"3d80fd9e-83d7-4c96-b196-1c9afac0bad4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gs462"
Jul 27 21:04:33.847072 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:04:33.846887    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7f66b2c1-260e-4a55-902b-c4a060b723f9-etc-openvswitch\") pod \"ovnkube-node-qs89h\" (UID: \"7f66b2c1-260e-4a55-902b-c4a060b723f9\") " pod="openshift-ovn-kubernetes/ovnkube-node-qs89h"
Jul 27 21:04:33.847072 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:04:33.846937    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/7f66b2c1-260e-4a55-902b-c4a060b723f9-node-log\") pod \"ovnkube-node-qs89h\" (UID: \"7f66b2c1-260e-4a55-902b-c4a060b723f9\") " pod="openshift-ovn-kubernetes/ovnkube-node-qs89h"
Jul 27 21:04:33.847072 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:04:33.846964    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7f66b2c1-260e-4a55-902b-c4a060b723f9-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-qs89h\" (UID: \"7f66b2c1-260e-4a55-902b-c4a060b723f9\") " pod="openshift-ovn-kubernetes/ovnkube-node-qs89h"
Jul 27 21:04:33.847072 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:04:33.847038    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/c2d67a6e-7963-4c73-9983-40b775a8134a-host-slash\") pod \"iptables-alerter-cd6w9\" (UID: \"c2d67a6e-7963-4c73-9983-40b775a8134a\") " pod="openshift-network-operator/iptables-alerter-cd6w9"
Jul 27 21:04:33.847660 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:04:33.847103    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/4275e4fb-a8e5-43a5-8b8c-ed4cc2094674-cni-binary-copy\") pod \"multus-additional-cni-plugins-2pbtk\" (UID: \"4275e4fb-a8e5-43a5-8b8c-ed4cc2094674\") " pod="openshift-multus/multus-additional-cni-plugins-2pbtk"
Jul 27 21:04:33.847660 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:04:33.847156    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/7f66b2c1-260e-4a55-902b-c4a060b723f9-host-slash\") pod \"ovnkube-node-qs89h\" (UID: \"7f66b2c1-260e-4a55-902b-c4a060b723f9\") " pod="openshift-ovn-kubernetes/ovnkube-node-qs89h"
Jul 27 21:04:33.847660 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:04:33.847183    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p2xrm\" (UniqueName: \"kubernetes.io/projected/7f66b2c1-260e-4a55-902b-c4a060b723f9-kube-api-access-p2xrm\") pod \"ovnkube-node-qs89h\" (UID: \"7f66b2c1-260e-4a55-902b-c4a060b723f9\") " pod="openshift-ovn-kubernetes/ovnkube-node-qs89h"
Jul 27 21:04:33.847660 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:04:33.847226    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/6b7af06d-65e7-45ca-80ef-e88bcee2282a-multus-daemon-config\") pod \"multus-dchr6\" (UID: \"6b7af06d-65e7-45ca-80ef-e88bcee2282a\") " pod="openshift-multus/multus-dchr6"
Jul 27 21:04:33.847660 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:04:33.847266    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wz54v\" (UniqueName: \"kubernetes.io/projected/3e4c5c3f-db85-4e88-a1d5-48e602b0018a-kube-api-access-wz54v\") pod \"node-ca-qxzfs\" (UID: \"3e4c5c3f-db85-4e88-a1d5-48e602b0018a\") " pod="openshift-image-registry/node-ca-qxzfs"
Jul 27 21:04:33.847660 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:04:33.847322    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/6b7af06d-65e7-45ca-80ef-e88bcee2282a-host-var-lib-cni-bin\") pod \"multus-dchr6\" (UID: \"6b7af06d-65e7-45ca-80ef-e88bcee2282a\") " pod="openshift-multus/multus-dchr6"
Jul 27 21:04:33.847660 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:04:33.847449    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xcn66\" (UniqueName: \"kubernetes.io/projected/c2d67a6e-7963-4c73-9983-40b775a8134a-kube-api-access-xcn66\") pod \"iptables-alerter-cd6w9\" (UID: \"c2d67a6e-7963-4c73-9983-40b775a8134a\") " pod="openshift-network-operator/iptables-alerter-cd6w9"
Jul 27 21:04:33.847660 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:04:33.847497    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/7f66b2c1-260e-4a55-902b-c4a060b723f9-run-systemd\") pod \"ovnkube-node-qs89h\" (UID: \"7f66b2c1-260e-4a55-902b-c4a060b723f9\") " pod="openshift-ovn-kubernetes/ovnkube-node-qs89h"
Jul 27 21:04:33.847660 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:04:33.847539    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/7f66b2c1-260e-4a55-902b-c4a060b723f9-ovn-node-metrics-cert\") pod \"ovnkube-node-qs89h\" (UID: \"7f66b2c1-260e-4a55-902b-c4a060b723f9\") " pod="openshift-ovn-kubernetes/ovnkube-node-qs89h"
Jul 27 21:04:33.847660 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:04:33.847570    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/6b7af06d-65e7-45ca-80ef-e88bcee2282a-cnibin\") pod \"multus-dchr6\" (UID: \"6b7af06d-65e7-45ca-80ef-e88bcee2282a\") " pod="openshift-multus/multus-dchr6"
Jul 27 21:04:33.847660 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:04:33.847602    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/6b7af06d-65e7-45ca-80ef-e88bcee2282a-host-var-lib-kubelet\") pod \"multus-dchr6\" (UID: \"6b7af06d-65e7-45ca-80ef-e88bcee2282a\") " pod="openshift-multus/multus-dchr6"
Jul 27 21:04:33.847660 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:04:33.847626    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/3e4c5c3f-db85-4e88-a1d5-48e602b0018a-serviceca\") pod \"node-ca-qxzfs\" (UID: \"3e4c5c3f-db85-4e88-a1d5-48e602b0018a\") " pod="openshift-image-registry/node-ca-qxzfs"
Jul 27 21:04:33.847660 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:04:33.847650    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/7f66b2c1-260e-4a55-902b-c4a060b723f9-env-overrides\") pod \"ovnkube-node-qs89h\" (UID: \"7f66b2c1-260e-4a55-902b-c4a060b723f9\") " pod="openshift-ovn-kubernetes/ovnkube-node-qs89h"
Jul 27 21:04:33.848402 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:04:33.847684    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/6b7af06d-65e7-45ca-80ef-e88bcee2282a-host-var-lib-cni-multus\") pod \"multus-dchr6\" (UID: \"6b7af06d-65e7-45ca-80ef-e88bcee2282a\") " pod="openshift-multus/multus-dchr6"
Jul 27 21:04:33.848402 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:04:33.847714    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/4275e4fb-a8e5-43a5-8b8c-ed4cc2094674-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-2pbtk\" (UID: \"4275e4fb-a8e5-43a5-8b8c-ed4cc2094674\") " pod="openshift-multus/multus-additional-cni-plugins-2pbtk"
Jul 27 21:04:33.848402 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:04:33.847744    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/3d80fd9e-83d7-4c96-b196-1c9afac0bad4-sys-fs\") pod \"aws-ebs-csi-driver-node-gs462\" (UID: \"3d80fd9e-83d7-4c96-b196-1c9afac0bad4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gs462"
Jul 27 21:04:33.848402 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:04:33.847779    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7f66b2c1-260e-4a55-902b-c4a060b723f9-host-run-ovn-kubernetes\") pod \"ovnkube-node-qs89h\" (UID: \"7f66b2c1-260e-4a55-902b-c4a060b723f9\") " pod="openshift-ovn-kubernetes/ovnkube-node-qs89h"
Jul 27 21:04:33.848402 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:04:33.847804    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/6b7af06d-65e7-45ca-80ef-e88bcee2282a-cni-binary-copy\") pod \"multus-dchr6\" (UID: \"6b7af06d-65e7-45ca-80ef-e88bcee2282a\") " pod="openshift-multus/multus-dchr6"
Jul 27 21:04:33.848402 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:04:33.847841    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/6b7af06d-65e7-45ca-80ef-e88bcee2282a-host-run-multus-certs\") pod \"multus-dchr6\" (UID: \"6b7af06d-65e7-45ca-80ef-e88bcee2282a\") " pod="openshift-multus/multus-dchr6"
Jul 27 21:04:33.848402 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:04:33.847888    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/4275e4fb-a8e5-43a5-8b8c-ed4cc2094674-tuning-conf-dir\") pod \"multus-additional-cni-plugins-2pbtk\" (UID: \"4275e4fb-a8e5-43a5-8b8c-ed4cc2094674\") " pod="openshift-multus/multus-additional-cni-plugins-2pbtk"
Jul 27 21:04:33.848402 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:04:33.847915    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/3d80fd9e-83d7-4c96-b196-1c9afac0bad4-device-dir\") pod \"aws-ebs-csi-driver-node-gs462\" (UID: \"3d80fd9e-83d7-4c96-b196-1c9afac0bad4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gs462"
Jul 27 21:04:33.848402 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:04:33.847939    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7f66b2c1-260e-4a55-902b-c4a060b723f9-run-openvswitch\") pod \"ovnkube-node-qs89h\" (UID: \"7f66b2c1-260e-4a55-902b-c4a060b723f9\") " pod="openshift-ovn-kubernetes/ovnkube-node-qs89h"
Jul 27 21:04:33.848402 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:04:33.847960    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/7f66b2c1-260e-4a55-902b-c4a060b723f9-host-cni-bin\") pod \"ovnkube-node-qs89h\" (UID: \"7f66b2c1-260e-4a55-902b-c4a060b723f9\") " pod="openshift-ovn-kubernetes/ovnkube-node-qs89h"
Jul 27 21:04:33.848402 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:04:33.848008    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6b7af06d-65e7-45ca-80ef-e88bcee2282a-multus-cni-dir\") pod \"multus-dchr6\" (UID: \"6b7af06d-65e7-45ca-80ef-e88bcee2282a\") " pod="openshift-multus/multus-dchr6"
Jul 27 21:04:33.848402 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:04:33.848049    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/3d80fd9e-83d7-4c96-b196-1c9afac0bad4-kubelet-dir\") pod \"aws-ebs-csi-driver-node-gs462\" (UID: \"3d80fd9e-83d7-4c96-b196-1c9afac0bad4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gs462"
Jul 27 21:04:33.848402 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:04:33.848082    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/7f66b2c1-260e-4a55-902b-c4a060b723f9-ovnkube-config\") pod \"ovnkube-node-qs89h\" (UID: \"7f66b2c1-260e-4a55-902b-c4a060b723f9\") " pod="openshift-ovn-kubernetes/ovnkube-node-qs89h"
Jul 27 21:04:33.848402 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:04:33.848106    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/6b7af06d-65e7-45ca-80ef-e88bcee2282a-host-run-k8s-cni-cncf-io\") pod \"multus-dchr6\" (UID: \"6b7af06d-65e7-45ca-80ef-e88bcee2282a\") " pod="openshift-multus/multus-dchr6"
Jul 27 21:04:33.848402 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:04:33.848134    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6b7af06d-65e7-45ca-80ef-e88bcee2282a-etc-kubernetes\") pod \"multus-dchr6\" (UID: \"6b7af06d-65e7-45ca-80ef-e88bcee2282a\") " pod="openshift-multus/multus-dchr6"
Jul 27 21:04:33.848402 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:04:33.848158    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/00ef4d6d-0b8e-4ece-ba97-36a9e236ec53-original-pull-secret\") pod \"global-pull-secret-syncer-rxl8w\" (UID: \"00ef4d6d-0b8e-4ece-ba97-36a9e236ec53\") " pod="kube-system/global-pull-secret-syncer-rxl8w"
Jul 27 21:04:33.849315 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:04:33.848174    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/3e4c5c3f-db85-4e88-a1d5-48e602b0018a-host\") pod \"node-ca-qxzfs\" (UID: \"3e4c5c3f-db85-4e88-a1d5-48e602b0018a\") " pod="openshift-image-registry/node-ca-qxzfs"
Jul 27 21:04:33.849315 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:04:33.848191    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/7f66b2c1-260e-4a55-902b-c4a060b723f9-systemd-units\") pod \"ovnkube-node-qs89h\" (UID: \"7f66b2c1-260e-4a55-902b-c4a060b723f9\") " pod="openshift-ovn-kubernetes/ovnkube-node-qs89h"
Jul 27 21:04:33.849315 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:04:33.848221    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/7f66b2c1-260e-4a55-902b-c4a060b723f9-log-socket\") pod \"ovnkube-node-qs89h\" (UID: \"7f66b2c1-260e-4a55-902b-c4a060b723f9\") " pod="openshift-ovn-kubernetes/ovnkube-node-qs89h"
Jul 27 21:04:33.849315 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:04:33.848264    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6b7af06d-65e7-45ca-80ef-e88bcee2282a-system-cni-dir\") pod \"multus-dchr6\" (UID: \"6b7af06d-65e7-45ca-80ef-e88bcee2282a\") " pod="openshift-multus/multus-dchr6"
Jul 27 21:04:33.849315 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:04:33.848289    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/6b7af06d-65e7-45ca-80ef-e88bcee2282a-multus-socket-dir-parent\") pod \"multus-dchr6\" (UID: \"6b7af06d-65e7-45ca-80ef-e88bcee2282a\") " pod="openshift-multus/multus-dchr6"
Jul 27 21:04:33.849315 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:04:33.848313    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/6b7af06d-65e7-45ca-80ef-e88bcee2282a-os-release\") pod \"multus-dchr6\" (UID: \"6b7af06d-65e7-45ca-80ef-e88bcee2282a\") " pod="openshift-multus/multus-dchr6"
Jul 27 21:04:33.849315 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:04:33.848339    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/4275e4fb-a8e5-43a5-8b8c-ed4cc2094674-system-cni-dir\") pod \"multus-additional-cni-plugins-2pbtk\" (UID: \"4275e4fb-a8e5-43a5-8b8c-ed4cc2094674\") " pod="openshift-multus/multus-additional-cni-plugins-2pbtk"
Jul 27 21:04:33.849315 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:04:33.848387    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/4275e4fb-a8e5-43a5-8b8c-ed4cc2094674-os-release\") pod \"multus-additional-cni-plugins-2pbtk\" (UID: \"4275e4fb-a8e5-43a5-8b8c-ed4cc2094674\") " pod="openshift-multus/multus-additional-cni-plugins-2pbtk"
Jul 27 21:04:33.849315 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:04:33.848420    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/4275e4fb-a8e5-43a5-8b8c-ed4cc2094674-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-2pbtk\" (UID: \"4275e4fb-a8e5-43a5-8b8c-ed4cc2094674\") " pod="openshift-multus/multus-additional-cni-plugins-2pbtk"
Jul 27 21:04:33.849315 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:04:33.848448    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ws2v2\" (UniqueName: \"kubernetes.io/projected/4275e4fb-a8e5-43a5-8b8c-ed4cc2094674-kube-api-access-ws2v2\") pod \"multus-additional-cni-plugins-2pbtk\" (UID: \"4275e4fb-a8e5-43a5-8b8c-ed4cc2094674\") " pod="openshift-multus/multus-additional-cni-plugins-2pbtk"
Jul 27 21:04:33.849315 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:04:33.848486    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/6b7af06d-65e7-45ca-80ef-e88bcee2282a-multus-conf-dir\") pod \"multus-dchr6\" (UID: \"6b7af06d-65e7-45ca-80ef-e88bcee2282a\") " pod="openshift-multus/multus-dchr6"
Jul 27 21:04:33.849315 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:04:33.848514    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/4ff24566-f9aa-4eab-8795-a2808831cec1-agent-certs\") pod \"konnectivity-agent-tfvtq\" (UID: \"4ff24566-f9aa-4eab-8795-a2808831cec1\") " pod="kube-system/konnectivity-agent-tfvtq"
Jul 27 21:04:33.849315 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:04:33.848540    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/7f66b2c1-260e-4a55-902b-c4a060b723f9-run-ovn\") pod \"ovnkube-node-qs89h\" (UID: \"7f66b2c1-260e-4a55-902b-c4a060b723f9\") " pod="openshift-ovn-kubernetes/ovnkube-node-qs89h"
Jul 27 21:04:33.849315 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:04:33.848566    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/7f66b2c1-260e-4a55-902b-c4a060b723f9-ovnkube-script-lib\") pod \"ovnkube-node-qs89h\" (UID: \"7f66b2c1-260e-4a55-902b-c4a060b723f9\") " pod="openshift-ovn-kubernetes/ovnkube-node-qs89h"
Jul 27 21:04:33.849315 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:04:33.848589    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/7f66b2c1-260e-4a55-902b-c4a060b723f9-host-kubelet\") pod \"ovnkube-node-qs89h\" (UID: \"7f66b2c1-260e-4a55-902b-c4a060b723f9\") " pod="openshift-ovn-kubernetes/ovnkube-node-qs89h"
Jul 27 21:04:33.849315 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:04:33.848614    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/7f66b2c1-260e-4a55-902b-c4a060b723f9-host-run-netns\") pod \"ovnkube-node-qs89h\" (UID: \"7f66b2c1-260e-4a55-902b-c4a060b723f9\") " pod="openshift-ovn-kubernetes/ovnkube-node-qs89h"
Jul 27 21:04:33.850189 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:04:33.848640    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/c2d67a6e-7963-4c73-9983-40b775a8134a-iptables-alerter-script\") pod \"iptables-alerter-cd6w9\" (UID: \"c2d67a6e-7963-4c73-9983-40b775a8134a\") " pod="openshift-network-operator/iptables-alerter-cd6w9"
Jul 27 21:04:33.851565 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:04:33.851544    2559 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 27 21:04:33.876317 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:04:33.876281    2559 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-bzsrq"
Jul 27 21:04:33.886125 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:04:33.886106    2559 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-bzsrq"
Jul 27 21:04:33.936855 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:04:33.936826    2559 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Jul 27 21:04:33.949683 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:04:33.949650    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-np6md\" (UniqueName: \"kubernetes.io/projected/3d80fd9e-83d7-4c96-b196-1c9afac0bad4-kube-api-access-np6md\") pod \"aws-ebs-csi-driver-node-gs462\" (UID: \"3d80fd9e-83d7-4c96-b196-1c9afac0bad4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gs462"
Jul 27 21:04:33.949885 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:04:33.949696    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/6b7af06d-65e7-45ca-80ef-e88bcee2282a-hostroot\") pod \"multus-dchr6\" (UID: \"6b7af06d-65e7-45ca-80ef-e88bcee2282a\") " pod="openshift-multus/multus-dchr6"
Jul 27 21:04:33.949885 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:04:33.949723    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e84e95da-4c32-4514-9c49-aaa0f96abea2-etc-kubernetes\") pod \"tuned-tfsvr\" (UID: \"e84e95da-4c32-4514-9c49-aaa0f96abea2\") " pod="openshift-cluster-node-tuning-operator/tuned-tfsvr"
Jul 27 21:04:33.949885 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:04:33.949751    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/e84e95da-4c32-4514-9c49-aaa0f96abea2-etc-tuned\") pod \"tuned-tfsvr\" (UID: \"e84e95da-4c32-4514-9c49-aaa0f96abea2\") " pod="openshift-cluster-node-tuning-operator/tuned-tfsvr"
Jul 27 21:04:33.949885 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:04:33.949776    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/3d80fd9e-83d7-4c96-b196-1c9afac0bad4-etc-selinux\") pod \"aws-ebs-csi-driver-node-gs462\" (UID: \"3d80fd9e-83d7-4c96-b196-1c9afac0bad4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gs462"
Jul 27 21:04:33.949885 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:04:33.949801    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7f66b2c1-260e-4a55-902b-c4a060b723f9-etc-openvswitch\") pod \"ovnkube-node-qs89h\" (UID: \"7f66b2c1-260e-4a55-902b-c4a060b723f9\") " pod="openshift-ovn-kubernetes/ovnkube-node-qs89h"
Jul 27 21:04:33.949885 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:04:33.949843    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/6b7af06d-65e7-45ca-80ef-e88bcee2282a-hostroot\") pod \"multus-dchr6\" (UID: \"6b7af06d-65e7-45ca-80ef-e88bcee2282a\") " pod="openshift-multus/multus-dchr6"
Jul 27 21:04:33.949885 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:04:33.949852    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7f66b2c1-260e-4a55-902b-c4a060b723f9-etc-openvswitch\") pod \"ovnkube-node-qs89h\" (UID: \"7f66b2c1-260e-4a55-902b-c4a060b723f9\") " pod="openshift-ovn-kubernetes/ovnkube-node-qs89h"
Jul 27 21:04:33.949885 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:04:33.949877    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/7f66b2c1-260e-4a55-902b-c4a060b723f9-node-log\") pod \"ovnkube-node-qs89h\" (UID: \"7f66b2c1-260e-4a55-902b-c4a060b723f9\") " pod="openshift-ovn-kubernetes/ovnkube-node-qs89h"
Jul 27 21:04:33.950362 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:04:33.949920    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/7f66b2c1-260e-4a55-902b-c4a060b723f9-node-log\") pod \"ovnkube-node-qs89h\" (UID: \"7f66b2c1-260e-4a55-902b-c4a060b723f9\") " pod="openshift-ovn-kubernetes/ovnkube-node-qs89h"
Jul 27 21:04:33.950362 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:04:33.949919    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7f66b2c1-260e-4a55-902b-c4a060b723f9-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-qs89h\" (UID: \"7f66b2c1-260e-4a55-902b-c4a060b723f9\") " pod="openshift-ovn-kubernetes/ovnkube-node-qs89h"
Jul 27 21:04:33.950362 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:04:33.949934    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/3d80fd9e-83d7-4c96-b196-1c9afac0bad4-etc-selinux\") pod \"aws-ebs-csi-driver-node-gs462\" (UID: \"3d80fd9e-83d7-4c96-b196-1c9afac0bad4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gs462"
Jul 27 21:04:33.950362 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:04:33.949958    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7f66b2c1-260e-4a55-902b-c4a060b723f9-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-qs89h\" (UID: \"7f66b2c1-260e-4a55-902b-c4a060b723f9\") " pod="openshift-ovn-kubernetes/ovnkube-node-qs89h"
Jul 27 21:04:33.950362 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:04:33.949963    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/e84e95da-4c32-4514-9c49-aaa0f96abea2-run\") pod \"tuned-tfsvr\" (UID: \"e84e95da-4c32-4514-9c49-aaa0f96abea2\") " pod="openshift-cluster-node-tuning-operator/tuned-tfsvr"
Jul 27 21:04:33.950362 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:04:33.949998    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/e84e95da-4c32-4514-9c49-aaa0f96abea2-host\") pod \"tuned-tfsvr\" (UID: \"e84e95da-4c32-4514-9c49-aaa0f96abea2\") " pod="openshift-cluster-node-tuning-operator/tuned-tfsvr"
Jul 27 21:04:33.950362 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:04:33.950045    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/c2d67a6e-7963-4c73-9983-40b775a8134a-host-slash\") pod \"iptables-alerter-cd6w9\" (UID: \"c2d67a6e-7963-4c73-9983-40b775a8134a\") " pod="openshift-network-operator/iptables-alerter-cd6w9"
Jul 27 21:04:33.950362 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:04:33.950077    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/4275e4fb-a8e5-43a5-8b8c-ed4cc2094674-cni-binary-copy\") pod \"multus-additional-cni-plugins-2pbtk\" (UID: \"4275e4fb-a8e5-43a5-8b8c-ed4cc2094674\") " pod="openshift-multus/multus-additional-cni-plugins-2pbtk"
Jul 27 21:04:33.950362 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:04:33.950102    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/7f66b2c1-260e-4a55-902b-c4a060b723f9-host-slash\") pod \"ovnkube-node-qs89h\" (UID: \"7f66b2c1-260e-4a55-902b-c4a060b723f9\") " pod="openshift-ovn-kubernetes/ovnkube-node-qs89h"
Jul 27 21:04:33.950362 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:04:33.950128    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p2xrm\" (UniqueName: \"kubernetes.io/projected/7f66b2c1-260e-4a55-902b-c4a060b723f9-kube-api-access-p2xrm\") pod \"ovnkube-node-qs89h\" (UID: \"7f66b2c1-260e-4a55-902b-c4a060b723f9\") " pod="openshift-ovn-kubernetes/ovnkube-node-qs89h"
Jul 27 21:04:33.950362 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:04:33.950160    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/6b7af06d-65e7-45ca-80ef-e88bcee2282a-multus-daemon-config\") pod \"multus-dchr6\" (UID: \"6b7af06d-65e7-45ca-80ef-e88bcee2282a\") " pod="openshift-multus/multus-dchr6"
Jul 27 21:04:33.950362 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:04:33.950159    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/c2d67a6e-7963-4c73-9983-40b775a8134a-host-slash\") pod \"iptables-alerter-cd6w9\" (UID: \"c2d67a6e-7963-4c73-9983-40b775a8134a\") " pod="openshift-network-operator/iptables-alerter-cd6w9"
Jul 27 21:04:33.950362 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:04:33.950167    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/7f66b2c1-260e-4a55-902b-c4a060b723f9-host-slash\") pod \"ovnkube-node-qs89h\" (UID: \"7f66b2c1-260e-4a55-902b-c4a060b723f9\") " pod="openshift-ovn-kubernetes/ovnkube-node-qs89h"
Jul 27 21:04:33.950362 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:04:33.950189    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g25kk\" (UniqueName: \"kubernetes.io/projected/84a7d738-f486-43b0-ac43-3401891da8d1-kube-api-access-g25kk\") pod \"node-resolver-fvbmc\" (UID: \"84a7d738-f486-43b0-ac43-3401891da8d1\") " pod="openshift-dns/node-resolver-fvbmc"
Jul 27 21:04:33.950362 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:04:33.950222    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wz54v\" (UniqueName: \"kubernetes.io/projected/3e4c5c3f-db85-4e88-a1d5-48e602b0018a-kube-api-access-wz54v\") pod \"node-ca-qxzfs\" (UID: \"3e4c5c3f-db85-4e88-a1d5-48e602b0018a\") " pod="openshift-image-registry/node-ca-qxzfs"
Jul 27 21:04:33.950362 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:04:33.950248    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/6b7af06d-65e7-45ca-80ef-e88bcee2282a-host-var-lib-cni-bin\") pod \"multus-dchr6\" (UID: \"6b7af06d-65e7-45ca-80ef-e88bcee2282a\") " pod="openshift-multus/multus-dchr6"
Jul 27 21:04:33.950362 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:04:33.950299    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/e84e95da-4c32-4514-9c49-aaa0f96abea2-etc-sysconfig\") pod \"tuned-tfsvr\" (UID: \"e84e95da-4c32-4514-9c49-aaa0f96abea2\") " pod="openshift-cluster-node-tuning-operator/tuned-tfsvr"
Jul 27 21:04:33.951263 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:04:33.950324    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rqj57\" (UniqueName: \"kubernetes.io/projected/e84e95da-4c32-4514-9c49-aaa0f96abea2-kube-api-access-rqj57\") pod \"tuned-tfsvr\" (UID: \"e84e95da-4c32-4514-9c49-aaa0f96abea2\") " pod="openshift-cluster-node-tuning-operator/tuned-tfsvr"
Jul 27 21:04:33.951263 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:04:33.950349    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xcn66\" (UniqueName: \"kubernetes.io/projected/c2d67a6e-7963-4c73-9983-40b775a8134a-kube-api-access-xcn66\") pod \"iptables-alerter-cd6w9\" (UID: \"c2d67a6e-7963-4c73-9983-40b775a8134a\") " pod="openshift-network-operator/iptables-alerter-cd6w9"
Jul 27 21:04:33.951263 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:04:33.950373    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/7f66b2c1-260e-4a55-902b-c4a060b723f9-run-systemd\") pod \"ovnkube-node-qs89h\" (UID: \"7f66b2c1-260e-4a55-902b-c4a060b723f9\") " pod="openshift-ovn-kubernetes/ovnkube-node-qs89h"
Jul 27 21:04:33.951263 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:04:33.950398    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/7f66b2c1-260e-4a55-902b-c4a060b723f9-ovn-node-metrics-cert\") pod \"ovnkube-node-qs89h\" (UID: \"7f66b2c1-260e-4a55-902b-c4a060b723f9\") " pod="openshift-ovn-kubernetes/ovnkube-node-qs89h"
Jul 27 21:04:33.951263 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:04:33.950423    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/6b7af06d-65e7-45ca-80ef-e88bcee2282a-cnibin\") pod \"multus-dchr6\" (UID: \"6b7af06d-65e7-45ca-80ef-e88bcee2282a\") " pod="openshift-multus/multus-dchr6"
Jul 27 21:04:33.951263 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:04:33.950456    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/6b7af06d-65e7-45ca-80ef-e88bcee2282a-host-var-lib-kubelet\") pod \"multus-dchr6\" (UID: \"6b7af06d-65e7-45ca-80ef-e88bcee2282a\") " pod="openshift-multus/multus-dchr6"
Jul 27 21:04:33.951263 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:04:33.950464    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/6b7af06d-65e7-45ca-80ef-e88bcee2282a-host-var-lib-cni-bin\") pod \"multus-dchr6\" (UID: \"6b7af06d-65e7-45ca-80ef-e88bcee2282a\") " pod="openshift-multus/multus-dchr6"
Jul 27 21:04:33.951263 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:04:33.950482    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/3e4c5c3f-db85-4e88-a1d5-48e602b0018a-serviceca\") pod \"node-ca-qxzfs\" (UID: \"3e4c5c3f-db85-4e88-a1d5-48e602b0018a\") " pod="openshift-image-registry/node-ca-qxzfs"
Jul 27 21:04:33.951263 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:04:33.950509    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/7f66b2c1-260e-4a55-902b-c4a060b723f9-env-overrides\") pod \"ovnkube-node-qs89h\" (UID: \"7f66b2c1-260e-4a55-902b-c4a060b723f9\") " pod="openshift-ovn-kubernetes/ovnkube-node-qs89h"
Jul 27 21:04:33.951263 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:04:33.950511    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/7f66b2c1-260e-4a55-902b-c4a060b723f9-run-systemd\") pod \"ovnkube-node-qs89h\" (UID: \"7f66b2c1-260e-4a55-902b-c4a060b723f9\") " pod="openshift-ovn-kubernetes/ovnkube-node-qs89h"
Jul 27 21:04:33.951263 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:04:33.950535    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/6b7af06d-65e7-45ca-80ef-e88bcee2282a-host-var-lib-cni-multus\") pod \"multus-dchr6\" (UID: \"6b7af06d-65e7-45ca-80ef-e88bcee2282a\") " pod="openshift-multus/multus-dchr6"
Jul 27 21:04:33.951263 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:04:33.950560    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e84e95da-4c32-4514-9c49-aaa0f96abea2-tmp\") pod \"tuned-tfsvr\" (UID: \"e84e95da-4c32-4514-9c49-aaa0f96abea2\") " pod="openshift-cluster-node-tuning-operator/tuned-tfsvr"
Jul 27 21:04:33.951263 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:04:33.950568    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/6b7af06d-65e7-45ca-80ef-e88bcee2282a-host-var-lib-kubelet\") pod \"multus-dchr6\" (UID: \"6b7af06d-65e7-45ca-80ef-e88bcee2282a\") " pod="openshift-multus/multus-dchr6"
Jul 27 21:04:33.951263 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:04:33.950586    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5851de39-848d-4a52-bc17-fefe1bf5407a-metrics-certs\") pod \"network-metrics-daemon-pr4c2\" (UID: \"5851de39-848d-4a52-bc17-fefe1bf5407a\") " pod="openshift-multus/network-metrics-daemon-pr4c2"
Jul 27 21:04:33.951263 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:04:33.950613    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/4275e4fb-a8e5-43a5-8b8c-ed4cc2094674-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-2pbtk\" (UID: \"4275e4fb-a8e5-43a5-8b8c-ed4cc2094674\") " pod="openshift-multus/multus-additional-cni-plugins-2pbtk"
Jul 27 21:04:33.951263 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:04:33.950647    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/3d80fd9e-83d7-4c96-b196-1c9afac0bad4-sys-fs\") pod \"aws-ebs-csi-driver-node-gs462\" (UID: \"3d80fd9e-83d7-4c96-b196-1c9afac0bad4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gs462"
Jul 27 21:04:33.951263 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:04:33.950674    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7f66b2c1-260e-4a55-902b-c4a060b723f9-host-run-ovn-kubernetes\") pod \"ovnkube-node-qs89h\" (UID: \"7f66b2c1-260e-4a55-902b-c4a060b723f9\") " pod="openshift-ovn-kubernetes/ovnkube-node-qs89h"
Jul 27 21:04:33.952173 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:04:33.950698    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/6b7af06d-65e7-45ca-80ef-e88bcee2282a-cni-binary-copy\") pod \"multus-dchr6\" (UID: \"6b7af06d-65e7-45ca-80ef-e88bcee2282a\") " pod="openshift-multus/multus-dchr6"
Jul 27 21:04:33.952173 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:04:33.950725    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/6b7af06d-65e7-45ca-80ef-e88bcee2282a-host-run-multus-certs\") pod \"multus-dchr6\" (UID: \"6b7af06d-65e7-45ca-80ef-e88bcee2282a\") " pod="openshift-multus/multus-dchr6"
Jul 27 21:04:33.952173 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:04:33.950750    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bpmt9\" (UniqueName: \"kubernetes.io/projected/5851de39-848d-4a52-bc17-fefe1bf5407a-kube-api-access-bpmt9\") pod \"network-metrics-daemon-pr4c2\" (UID: \"5851de39-848d-4a52-bc17-fefe1bf5407a\") " pod="openshift-multus/network-metrics-daemon-pr4c2"
Jul 27 21:04:33.952173 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:04:33.950778    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/4275e4fb-a8e5-43a5-8b8c-ed4cc2094674-tuning-conf-dir\") pod \"multus-additional-cni-plugins-2pbtk\" (UID: \"4275e4fb-a8e5-43a5-8b8c-ed4cc2094674\") " pod="openshift-multus/multus-additional-cni-plugins-2pbtk"
Jul 27 21:04:33.952173 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:04:33.950803    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/3d80fd9e-83d7-4c96-b196-1c9afac0bad4-device-dir\") pod \"aws-ebs-csi-driver-node-gs462\" (UID: \"3d80fd9e-83d7-4c96-b196-1c9afac0bad4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gs462"
Jul 27 21:04:33.952173 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:04:33.950834    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/6b7af06d-65e7-45ca-80ef-e88bcee2282a-multus-daemon-config\") pod \"multus-dchr6\" (UID: \"6b7af06d-65e7-45ca-80ef-e88bcee2282a\") " pod="openshift-multus/multus-dchr6"
Jul 27 21:04:33.952173 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:04:33.950840    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/4275e4fb-a8e5-43a5-8b8c-ed4cc2094674-cni-binary-copy\") pod \"multus-additional-cni-plugins-2pbtk\" (UID: \"4275e4fb-a8e5-43a5-8b8c-ed4cc2094674\") " pod="openshift-multus/multus-additional-cni-plugins-2pbtk"
Jul 27 21:04:33.952173 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:04:33.950697    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/6b7af06d-65e7-45ca-80ef-e88bcee2282a-cnibin\") pod \"multus-dchr6\" (UID: \"6b7af06d-65e7-45ca-80ef-e88bcee2282a\") " pod="openshift-multus/multus-dchr6"
Jul 27 21:04:33.952173 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:04:33.950829    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7f66b2c1-260e-4a55-902b-c4a060b723f9-run-openvswitch\") pod \"ovnkube-node-qs89h\" (UID: \"7f66b2c1-260e-4a55-902b-c4a060b723f9\") " pod="openshift-ovn-kubernetes/ovnkube-node-qs89h"
Jul 27 21:04:33.952173 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:04:33.950916    2559 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Jul 27 21:04:33.952173 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:04:33.950938    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/7f66b2c1-260e-4a55-902b-c4a060b723f9-host-cni-bin\") pod \"ovnkube-node-qs89h\" (UID: \"7f66b2c1-260e-4a55-902b-c4a060b723f9\") " pod="openshift-ovn-kubernetes/ovnkube-node-qs89h"
Jul 27 21:04:33.952173 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:04:33.950947    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/3d80fd9e-83d7-4c96-b196-1c9afac0bad4-sys-fs\") pod \"aws-ebs-csi-driver-node-gs462\" (UID: \"3d80fd9e-83d7-4c96-b196-1c9afac0bad4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gs462"
Jul 27 21:04:33.952173 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:04:33.950970    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6b7af06d-65e7-45ca-80ef-e88bcee2282a-multus-cni-dir\") pod \"multus-dchr6\" (UID: \"6b7af06d-65e7-45ca-80ef-e88bcee2282a\") " pod="openshift-multus/multus-dchr6"
Jul 27 21:04:33.952173 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:04:33.951000    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7f66b2c1-260e-4a55-902b-c4a060b723f9-host-run-ovn-kubernetes\") pod \"ovnkube-node-qs89h\" (UID: \"7f66b2c1-260e-4a55-902b-c4a060b723f9\") " pod="openshift-ovn-kubernetes/ovnkube-node-qs89h"
Jul 27 21:04:33.952173 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:04:33.951000    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/e84e95da-4c32-4514-9c49-aaa0f96abea2-etc-sysctl-d\") pod \"tuned-tfsvr\" (UID: \"e84e95da-4c32-4514-9c49-aaa0f96abea2\") " pod="openshift-cluster-node-tuning-operator/tuned-tfsvr"
Jul 27 21:04:33.952173 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:04:33.951027    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/4275e4fb-a8e5-43a5-8b8c-ed4cc2094674-tuning-conf-dir\") pod \"multus-additional-cni-plugins-2pbtk\" (UID: \"4275e4fb-a8e5-43a5-8b8c-ed4cc2094674\") " pod="openshift-multus/multus-additional-cni-plugins-2pbtk"
Jul 27 21:04:33.952173 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:04:33.951052    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7f66b2c1-260e-4a55-902b-c4a060b723f9-run-openvswitch\") pod \"ovnkube-node-qs89h\" (UID: \"7f66b2c1-260e-4a55-902b-c4a060b723f9\") " pod="openshift-ovn-kubernetes/ovnkube-node-qs89h"
Jul 27 21:04:33.953032 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:04:33.951077    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/3d80fd9e-83d7-4c96-b196-1c9afac0bad4-device-dir\") pod \"aws-ebs-csi-driver-node-gs462\" (UID: \"3d80fd9e-83d7-4c96-b196-1c9afac0bad4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gs462"
Jul 27 21:04:33.953032 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:04:33.951082    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/6b7af06d-65e7-45ca-80ef-e88bcee2282a-host-run-multus-certs\") pod \"multus-dchr6\" (UID: \"6b7af06d-65e7-45ca-80ef-e88bcee2282a\") " pod="openshift-multus/multus-dchr6"
Jul 27 21:04:33.953032 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:04:33.951115    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/7f66b2c1-260e-4a55-902b-c4a060b723f9-host-cni-bin\") pod \"ovnkube-node-qs89h\" (UID: \"7f66b2c1-260e-4a55-902b-c4a060b723f9\") " pod="openshift-ovn-kubernetes/ovnkube-node-qs89h"
Jul 27 21:04:33.953032 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:04:33.951154    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/3e4c5c3f-db85-4e88-a1d5-48e602b0018a-serviceca\") pod \"node-ca-qxzfs\" (UID: \"3e4c5c3f-db85-4e88-a1d5-48e602b0018a\") " pod="openshift-image-registry/node-ca-qxzfs"
Jul 27 21:04:33.953032 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:04:33.951201    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6b7af06d-65e7-45ca-80ef-e88bcee2282a-multus-cni-dir\") pod \"multus-dchr6\" (UID: \"6b7af06d-65e7-45ca-80ef-e88bcee2282a\") " pod="openshift-multus/multus-dchr6"
Jul 27 21:04:33.953032 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:04:33.951204    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/6b7af06d-65e7-45ca-80ef-e88bcee2282a-host-var-lib-cni-multus\") pod \"multus-dchr6\" (UID: \"6b7af06d-65e7-45ca-80ef-e88bcee2282a\") " pod="openshift-multus/multus-dchr6"
Jul 27 21:04:33.953032 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:04:33.951199    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e84e95da-4c32-4514-9c49-aaa0f96abea2-var-lib-kubelet\") pod \"tuned-tfsvr\" (UID: \"e84e95da-4c32-4514-9c49-aaa0f96abea2\") " pod="openshift-cluster-node-tuning-operator/tuned-tfsvr"
Jul 27 21:04:33.953032 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:04:33.951245    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/3d80fd9e-83d7-4c96-b196-1c9afac0bad4-kubelet-dir\") pod \"aws-ebs-csi-driver-node-gs462\" (UID: \"3d80fd9e-83d7-4c96-b196-1c9afac0bad4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gs462"
Jul 27 21:04:33.953032 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:04:33.951287    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/7f66b2c1-260e-4a55-902b-c4a060b723f9-ovnkube-config\") pod \"ovnkube-node-qs89h\" (UID: \"7f66b2c1-260e-4a55-902b-c4a060b723f9\") " pod="openshift-ovn-kubernetes/ovnkube-node-qs89h"
Jul 27 21:04:33.953032 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:04:33.951332    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/6b7af06d-65e7-45ca-80ef-e88bcee2282a-host-run-k8s-cni-cncf-io\") pod \"multus-dchr6\" (UID: \"6b7af06d-65e7-45ca-80ef-e88bcee2282a\") " pod="openshift-multus/multus-dchr6"
Jul 27 21:04:33.953032 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:04:33.951421    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6b7af06d-65e7-45ca-80ef-e88bcee2282a-etc-kubernetes\") pod \"multus-dchr6\" (UID: \"6b7af06d-65e7-45ca-80ef-e88bcee2282a\") " pod="openshift-multus/multus-dchr6"
Jul 27 21:04:33.953032 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:04:33.951462    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/e84e95da-4c32-4514-9c49-aaa0f96abea2-lib-modules\") pod \"tuned-tfsvr\" (UID: \"e84e95da-4c32-4514-9c49-aaa0f96abea2\") " pod="openshift-cluster-node-tuning-operator/tuned-tfsvr"
Jul 27 21:04:33.953032 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:04:33.951492    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/00ef4d6d-0b8e-4ece-ba97-36a9e236ec53-original-pull-secret\") pod \"global-pull-secret-syncer-rxl8w\" (UID: \"00ef4d6d-0b8e-4ece-ba97-36a9e236ec53\") " pod="kube-system/global-pull-secret-syncer-rxl8w"
Jul 27 21:04:33.953032 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:04:33.951536    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/3e4c5c3f-db85-4e88-a1d5-48e602b0018a-host\") pod \"node-ca-qxzfs\" (UID: \"3e4c5c3f-db85-4e88-a1d5-48e602b0018a\") " pod="openshift-image-registry/node-ca-qxzfs"
Jul 27 21:04:33.953032 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:04:33.951567    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/6b7af06d-65e7-45ca-80ef-e88bcee2282a-cni-binary-copy\") pod \"multus-dchr6\" (UID: \"6b7af06d-65e7-45ca-80ef-e88bcee2282a\") " pod="openshift-multus/multus-dchr6"
Jul 27 21:04:33.953032 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:04:33.951603    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/7f66b2c1-260e-4a55-902b-c4a060b723f9-systemd-units\") pod \"ovnkube-node-qs89h\" (UID: \"7f66b2c1-260e-4a55-902b-c4a060b723f9\") " pod="openshift-ovn-kubernetes/ovnkube-node-qs89h"
Jul 27 21:04:33.953032 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:04:33.951618    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/6b7af06d-65e7-45ca-80ef-e88bcee2282a-host-run-k8s-cni-cncf-io\") pod \"multus-dchr6\" (UID: \"6b7af06d-65e7-45ca-80ef-e88bcee2282a\") " pod="openshift-multus/multus-dchr6"
Jul 27 21:04:33.953032 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:04:33.951641    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/7f66b2c1-260e-4a55-902b-c4a060b723f9-log-socket\") pod \"ovnkube-node-qs89h\" (UID: \"7f66b2c1-260e-4a55-902b-c4a060b723f9\") " pod="openshift-ovn-kubernetes/ovnkube-node-qs89h"
Jul 27 21:04:33.953935 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:04:33.951655    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/3e4c5c3f-db85-4e88-a1d5-48e602b0018a-host\") pod \"node-ca-qxzfs\" (UID: \"3e4c5c3f-db85-4e88-a1d5-48e602b0018a\") " pod="openshift-image-registry/node-ca-qxzfs"
Jul 27 21:04:33.953935 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:04:33.951695    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6b7af06d-65e7-45ca-80ef-e88bcee2282a-system-cni-dir\") pod \"multus-dchr6\" (UID: \"6b7af06d-65e7-45ca-80ef-e88bcee2282a\") " pod="openshift-multus/multus-dchr6"
Jul 27 21:04:33.953935 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:04:33.951722    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/6b7af06d-65e7-45ca-80ef-e88bcee2282a-multus-socket-dir-parent\") pod \"multus-dchr6\" (UID: \"6b7af06d-65e7-45ca-80ef-e88bcee2282a\") " pod="openshift-multus/multus-dchr6"
Jul 27 21:04:33.953935 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:04:33.951729    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/4275e4fb-a8e5-43a5-8b8c-ed4cc2094674-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-2pbtk\" (UID: \"4275e4fb-a8e5-43a5-8b8c-ed4cc2094674\") " pod="openshift-multus/multus-additional-cni-plugins-2pbtk"
Jul 27 21:04:33.953935 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:04:33.951753    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/6b7af06d-65e7-45ca-80ef-e88bcee2282a-os-release\") pod \"multus-dchr6\" (UID: \"6b7af06d-65e7-45ca-80ef-e88bcee2282a\") " pod="openshift-multus/multus-dchr6"
Jul 27 21:04:33.953935 ip-10-0-131-133 kubenswrapper[2559]: E0727 21:04:33.951779    2559 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 27 21:04:33.953935 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:04:33.951793    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/7f66b2c1-260e-4a55-902b-c4a060b723f9-systemd-units\") pod \"ovnkube-node-qs89h\" (UID: \"7f66b2c1-260e-4a55-902b-c4a060b723f9\") " pod="openshift-ovn-kubernetes/ovnkube-node-qs89h"
Jul 27 21:04:33.953935 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:04:33.951800    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/7f66b2c1-260e-4a55-902b-c4a060b723f9-env-overrides\") pod \"ovnkube-node-qs89h\" (UID: \"7f66b2c1-260e-4a55-902b-c4a060b723f9\") " pod="openshift-ovn-kubernetes/ovnkube-node-qs89h"
Jul 27 21:04:33.953935 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:04:33.951837    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/7f66b2c1-260e-4a55-902b-c4a060b723f9-log-socket\") pod \"ovnkube-node-qs89h\" (UID: \"7f66b2c1-260e-4a55-902b-c4a060b723f9\") " pod="openshift-ovn-kubernetes/ovnkube-node-qs89h"
Jul 27 21:04:33.953935 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:04:33.951844    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/6b7af06d-65e7-45ca-80ef-e88bcee2282a-multus-socket-dir-parent\") pod \"multus-dchr6\" (UID: \"6b7af06d-65e7-45ca-80ef-e88bcee2282a\") " pod="openshift-multus/multus-dchr6"
Jul 27 21:04:33.953935 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:04:33.951858    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/3d80fd9e-83d7-4c96-b196-1c9afac0bad4-kubelet-dir\") pod \"aws-ebs-csi-driver-node-gs462\" (UID: \"3d80fd9e-83d7-4c96-b196-1c9afac0bad4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gs462"
Jul 27 21:04:33.953935 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:04:33.951870    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/84a7d738-f486-43b0-ac43-3401891da8d1-tmp-dir\") pod \"node-resolver-fvbmc\" (UID: \"84a7d738-f486-43b0-ac43-3401891da8d1\") " pod="openshift-dns/node-resolver-fvbmc"
Jul 27 21:04:33.953935 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:04:33.951898    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6b7af06d-65e7-45ca-80ef-e88bcee2282a-etc-kubernetes\") pod \"multus-dchr6\" (UID: \"6b7af06d-65e7-45ca-80ef-e88bcee2282a\") " pod="openshift-multus/multus-dchr6"
Jul 27 21:04:33.953935 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:04:33.951902    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/4275e4fb-a8e5-43a5-8b8c-ed4cc2094674-system-cni-dir\") pod \"multus-additional-cni-plugins-2pbtk\" (UID: \"4275e4fb-a8e5-43a5-8b8c-ed4cc2094674\") " pod="openshift-multus/multus-additional-cni-plugins-2pbtk"
Jul 27 21:04:33.953935 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:04:33.951906    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6b7af06d-65e7-45ca-80ef-e88bcee2282a-system-cni-dir\") pod \"multus-dchr6\" (UID: \"6b7af06d-65e7-45ca-80ef-e88bcee2282a\") " pod="openshift-multus/multus-dchr6"
Jul 27 21:04:33.953935 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:04:33.951946    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/4275e4fb-a8e5-43a5-8b8c-ed4cc2094674-system-cni-dir\") pod \"multus-additional-cni-plugins-2pbtk\" (UID: \"4275e4fb-a8e5-43a5-8b8c-ed4cc2094674\") " pod="openshift-multus/multus-additional-cni-plugins-2pbtk"
Jul 27 21:04:33.953935 ip-10-0-131-133 kubenswrapper[2559]: E0727 21:04:33.951949    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/00ef4d6d-0b8e-4ece-ba97-36a9e236ec53-original-pull-secret podName:00ef4d6d-0b8e-4ece-ba97-36a9e236ec53 nodeName:}" failed. No retries permitted until 2026-07-27 21:04:34.451909472 +0000 UTC m=+3.031093477 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/00ef4d6d-0b8e-4ece-ba97-36a9e236ec53-original-pull-secret") pod "global-pull-secret-syncer-rxl8w" (UID: "00ef4d6d-0b8e-4ece-ba97-36a9e236ec53") : object "kube-system"/"original-pull-secret" not registered
Jul 27 21:04:33.954906 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:04:33.951970    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/6b7af06d-65e7-45ca-80ef-e88bcee2282a-os-release\") pod \"multus-dchr6\" (UID: \"6b7af06d-65e7-45ca-80ef-e88bcee2282a\") " pod="openshift-multus/multus-dchr6"
Jul 27 21:04:33.954906 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:04:33.951996    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/4275e4fb-a8e5-43a5-8b8c-ed4cc2094674-os-release\") pod \"multus-additional-cni-plugins-2pbtk\" (UID: \"4275e4fb-a8e5-43a5-8b8c-ed4cc2094674\") " pod="openshift-multus/multus-additional-cni-plugins-2pbtk"
Jul 27 21:04:33.954906 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:04:33.952029    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/4275e4fb-a8e5-43a5-8b8c-ed4cc2094674-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-2pbtk\" (UID: \"4275e4fb-a8e5-43a5-8b8c-ed4cc2094674\") " pod="openshift-multus/multus-additional-cni-plugins-2pbtk"
Jul 27 21:04:33.954906 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:04:33.952074    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/7f66b2c1-260e-4a55-902b-c4a060b723f9-ovnkube-config\") pod \"ovnkube-node-qs89h\" (UID: \"7f66b2c1-260e-4a55-902b-c4a060b723f9\") " pod="openshift-ovn-kubernetes/ovnkube-node-qs89h"
Jul 27 21:04:33.954906 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:04:33.952103    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/4275e4fb-a8e5-43a5-8b8c-ed4cc2094674-os-release\") pod \"multus-additional-cni-plugins-2pbtk\" (UID: \"4275e4fb-a8e5-43a5-8b8c-ed4cc2094674\") " pod="openshift-multus/multus-additional-cni-plugins-2pbtk"
Jul 27 21:04:33.954906 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:04:33.952103    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ws2v2\" (UniqueName: \"kubernetes.io/projected/4275e4fb-a8e5-43a5-8b8c-ed4cc2094674-kube-api-access-ws2v2\") pod \"multus-additional-cni-plugins-2pbtk\" (UID: \"4275e4fb-a8e5-43a5-8b8c-ed4cc2094674\") " pod="openshift-multus/multus-additional-cni-plugins-2pbtk"
Jul 27 21:04:33.954906 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:04:33.952199    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/6b7af06d-65e7-45ca-80ef-e88bcee2282a-multus-conf-dir\") pod \"multus-dchr6\" (UID: \"6b7af06d-65e7-45ca-80ef-e88bcee2282a\") " pod="openshift-multus/multus-dchr6"
Jul 27 21:04:33.954906 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:04:33.952256    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/e84e95da-4c32-4514-9c49-aaa0f96abea2-etc-modprobe-d\") pod \"tuned-tfsvr\" (UID: \"e84e95da-4c32-4514-9c49-aaa0f96abea2\") " pod="openshift-cluster-node-tuning-operator/tuned-tfsvr"
Jul 27 21:04:33.954906 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:04:33.952268    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/6b7af06d-65e7-45ca-80ef-e88bcee2282a-multus-conf-dir\") pod \"multus-dchr6\" (UID: \"6b7af06d-65e7-45ca-80ef-e88bcee2282a\") " pod="openshift-multus/multus-dchr6"
Jul 27 21:04:33.954906 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:04:33.952309    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/4ff24566-f9aa-4eab-8795-a2808831cec1-agent-certs\") pod \"konnectivity-agent-tfvtq\" (UID: \"4ff24566-f9aa-4eab-8795-a2808831cec1\") " pod="kube-system/konnectivity-agent-tfvtq"
Jul 27 21:04:33.954906 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:04:33.952334    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/7f66b2c1-260e-4a55-902b-c4a060b723f9-run-ovn\") pod \"ovnkube-node-qs89h\" (UID: \"7f66b2c1-260e-4a55-902b-c4a060b723f9\") " pod="openshift-ovn-kubernetes/ovnkube-node-qs89h"
Jul 27 21:04:33.954906 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:04:33.952370    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/7f66b2c1-260e-4a55-902b-c4a060b723f9-ovnkube-script-lib\") pod \"ovnkube-node-qs89h\" (UID: \"7f66b2c1-260e-4a55-902b-c4a060b723f9\") " pod="openshift-ovn-kubernetes/ovnkube-node-qs89h"
Jul 27 21:04:33.954906 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:04:33.952398    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/7f66b2c1-260e-4a55-902b-c4a060b723f9-host-kubelet\") pod \"ovnkube-node-qs89h\" (UID: \"7f66b2c1-260e-4a55-902b-c4a060b723f9\") " pod="openshift-ovn-kubernetes/ovnkube-node-qs89h"
Jul 27 21:04:33.954906 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:04:33.952446    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/7f66b2c1-260e-4a55-902b-c4a060b723f9-host-run-netns\") pod \"ovnkube-node-qs89h\" (UID: \"7f66b2c1-260e-4a55-902b-c4a060b723f9\") " pod="openshift-ovn-kubernetes/ovnkube-node-qs89h"
Jul 27 21:04:33.954906 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:04:33.952473    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/e84e95da-4c32-4514-9c49-aaa0f96abea2-etc-systemd\") pod \"tuned-tfsvr\" (UID: \"e84e95da-4c32-4514-9c49-aaa0f96abea2\") " pod="openshift-cluster-node-tuning-operator/tuned-tfsvr"
Jul 27 21:04:33.954906 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:04:33.952485    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/7f66b2c1-260e-4a55-902b-c4a060b723f9-run-ovn\") pod \"ovnkube-node-qs89h\" (UID: \"7f66b2c1-260e-4a55-902b-c4a060b723f9\") " pod="openshift-ovn-kubernetes/ovnkube-node-qs89h"
Jul 27 21:04:33.954906 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:04:33.952498    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e84e95da-4c32-4514-9c49-aaa0f96abea2-sys\") pod \"tuned-tfsvr\" (UID: \"e84e95da-4c32-4514-9c49-aaa0f96abea2\") " pod="openshift-cluster-node-tuning-operator/tuned-tfsvr"
Jul 27 21:04:33.955671 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:04:33.952541    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/7f66b2c1-260e-4a55-902b-c4a060b723f9-host-kubelet\") pod \"ovnkube-node-qs89h\" (UID: \"7f66b2c1-260e-4a55-902b-c4a060b723f9\") " pod="openshift-ovn-kubernetes/ovnkube-node-qs89h"
Jul 27 21:04:33.955671 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:04:33.952552    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/4275e4fb-a8e5-43a5-8b8c-ed4cc2094674-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-2pbtk\" (UID: \"4275e4fb-a8e5-43a5-8b8c-ed4cc2094674\") " pod="openshift-multus/multus-additional-cni-plugins-2pbtk"
Jul 27 21:04:33.955671 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:04:33.952548    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/7f66b2c1-260e-4a55-902b-c4a060b723f9-host-run-netns\") pod \"ovnkube-node-qs89h\" (UID: \"7f66b2c1-260e-4a55-902b-c4a060b723f9\") " pod="openshift-ovn-kubernetes/ovnkube-node-qs89h"
Jul 27 21:04:33.955671 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:04:33.952575    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/84a7d738-f486-43b0-ac43-3401891da8d1-hosts-file\") pod \"node-resolver-fvbmc\" (UID: \"84a7d738-f486-43b0-ac43-3401891da8d1\") " pod="openshift-dns/node-resolver-fvbmc"
Jul 27 21:04:33.955671 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:04:33.952677    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/c2d67a6e-7963-4c73-9983-40b775a8134a-iptables-alerter-script\") pod \"iptables-alerter-cd6w9\" (UID: \"c2d67a6e-7963-4c73-9983-40b775a8134a\") " pod="openshift-network-operator/iptables-alerter-cd6w9"
Jul 27 21:04:33.955671 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:04:33.953272    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/c2d67a6e-7963-4c73-9983-40b775a8134a-iptables-alerter-script\") pod \"iptables-alerter-cd6w9\" (UID: \"c2d67a6e-7963-4c73-9983-40b775a8134a\") " pod="openshift-network-operator/iptables-alerter-cd6w9"
Jul 27 21:04:33.955671 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:04:33.952709    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/e84e95da-4c32-4514-9c49-aaa0f96abea2-etc-sysctl-conf\") pod \"tuned-tfsvr\" (UID: \"e84e95da-4c32-4514-9c49-aaa0f96abea2\") " pod="openshift-cluster-node-tuning-operator/tuned-tfsvr"
Jul 27 21:04:33.955671 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:04:33.953345    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/3d80fd9e-83d7-4c96-b196-1c9afac0bad4-registration-dir\") pod \"aws-ebs-csi-driver-node-gs462\" (UID: \"3d80fd9e-83d7-4c96-b196-1c9afac0bad4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gs462"
Jul 27 21:04:33.955671 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:04:33.953406    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/4ff24566-f9aa-4eab-8795-a2808831cec1-konnectivity-ca\") pod \"konnectivity-agent-tfvtq\" (UID: \"4ff24566-f9aa-4eab-8795-a2808831cec1\") " pod="kube-system/konnectivity-agent-tfvtq"
Jul 27 21:04:33.955671 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:04:33.953505    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/3d80fd9e-83d7-4c96-b196-1c9afac0bad4-registration-dir\") pod \"aws-ebs-csi-driver-node-gs462\" (UID: \"3d80fd9e-83d7-4c96-b196-1c9afac0bad4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gs462"
Jul 27 21:04:33.955671 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:04:33.953640    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7f66b2c1-260e-4a55-902b-c4a060b723f9-var-lib-openvswitch\") pod \"ovnkube-node-qs89h\" (UID: \"7f66b2c1-260e-4a55-902b-c4a060b723f9\") " pod="openshift-ovn-kubernetes/ovnkube-node-qs89h"
Jul 27 21:04:33.955671 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:04:33.953938    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/7f66b2c1-260e-4a55-902b-c4a060b723f9-ovnkube-script-lib\") pod \"ovnkube-node-qs89h\" (UID: \"7f66b2c1-260e-4a55-902b-c4a060b723f9\") " pod="openshift-ovn-kubernetes/ovnkube-node-qs89h"
Jul 27 21:04:33.955671 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:04:33.953976    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/4ff24566-f9aa-4eab-8795-a2808831cec1-konnectivity-ca\") pod \"konnectivity-agent-tfvtq\" (UID: \"4ff24566-f9aa-4eab-8795-a2808831cec1\") " pod="kube-system/konnectivity-agent-tfvtq"
Jul 27 21:04:33.955671 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:04:33.953437    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7f66b2c1-260e-4a55-902b-c4a060b723f9-var-lib-openvswitch\") pod \"ovnkube-node-qs89h\" (UID: \"7f66b2c1-260e-4a55-902b-c4a060b723f9\") " pod="openshift-ovn-kubernetes/ovnkube-node-qs89h"
Jul 27 21:04:33.955671 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:04:33.954053    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/7f66b2c1-260e-4a55-902b-c4a060b723f9-host-cni-netd\") pod \"ovnkube-node-qs89h\" (UID: \"7f66b2c1-260e-4a55-902b-c4a060b723f9\") " pod="openshift-ovn-kubernetes/ovnkube-node-qs89h"
Jul 27 21:04:33.955671 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:04:33.954140    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/7f66b2c1-260e-4a55-902b-c4a060b723f9-host-cni-netd\") pod \"ovnkube-node-qs89h\" (UID: \"7f66b2c1-260e-4a55-902b-c4a060b723f9\") " pod="openshift-ovn-kubernetes/ovnkube-node-qs89h"
Jul 27 21:04:33.955671 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:04:33.954212    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/6b7af06d-65e7-45ca-80ef-e88bcee2282a-host-run-netns\") pod \"multus-dchr6\" (UID: \"6b7af06d-65e7-45ca-80ef-e88bcee2282a\") " pod="openshift-multus/multus-dchr6"
Jul 27 21:04:33.956636 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:04:33.954168    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/6b7af06d-65e7-45ca-80ef-e88bcee2282a-host-run-netns\") pod \"multus-dchr6\" (UID: \"6b7af06d-65e7-45ca-80ef-e88bcee2282a\") " pod="openshift-multus/multus-dchr6"
Jul 27 21:04:33.956636 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:04:33.954285    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j6h2z\" (UniqueName: \"kubernetes.io/projected/6b7af06d-65e7-45ca-80ef-e88bcee2282a-kube-api-access-j6h2z\") pod \"multus-dchr6\" (UID: \"6b7af06d-65e7-45ca-80ef-e88bcee2282a\") " pod="openshift-multus/multus-dchr6"
Jul 27 21:04:33.956636 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:04:33.954312    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mtwdg\" (UniqueName: \"kubernetes.io/projected/95b2123e-132a-42a8-8edb-33ce19c2df51-kube-api-access-mtwdg\") pod \"network-check-target-hmjv6\" (UID: \"95b2123e-132a-42a8-8edb-33ce19c2df51\") " pod="openshift-network-diagnostics/network-check-target-hmjv6"
Jul 27 21:04:33.956636 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:04:33.954730    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/00ef4d6d-0b8e-4ece-ba97-36a9e236ec53-kubelet-config\") pod \"global-pull-secret-syncer-rxl8w\" (UID: \"00ef4d6d-0b8e-4ece-ba97-36a9e236ec53\") " pod="kube-system/global-pull-secret-syncer-rxl8w"
Jul 27 21:04:33.956636 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:04:33.954821    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/00ef4d6d-0b8e-4ece-ba97-36a9e236ec53-kubelet-config\") pod \"global-pull-secret-syncer-rxl8w\" (UID: \"00ef4d6d-0b8e-4ece-ba97-36a9e236ec53\") " pod="kube-system/global-pull-secret-syncer-rxl8w"
Jul 27 21:04:33.956636 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:04:33.954900    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/00ef4d6d-0b8e-4ece-ba97-36a9e236ec53-dbus\") pod \"global-pull-secret-syncer-rxl8w\" (UID: \"00ef4d6d-0b8e-4ece-ba97-36a9e236ec53\") " pod="kube-system/global-pull-secret-syncer-rxl8w"
Jul 27 21:04:33.956636 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:04:33.954759    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/00ef4d6d-0b8e-4ece-ba97-36a9e236ec53-dbus\") pod \"global-pull-secret-syncer-rxl8w\" (UID: \"00ef4d6d-0b8e-4ece-ba97-36a9e236ec53\") " pod="kube-system/global-pull-secret-syncer-rxl8w"
Jul 27 21:04:33.956636 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:04:33.954962    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/4275e4fb-a8e5-43a5-8b8c-ed4cc2094674-cnibin\") pod \"multus-additional-cni-plugins-2pbtk\" (UID: \"4275e4fb-a8e5-43a5-8b8c-ed4cc2094674\") " pod="openshift-multus/multus-additional-cni-plugins-2pbtk"
Jul 27 21:04:33.956636 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:04:33.954988    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/3d80fd9e-83d7-4c96-b196-1c9afac0bad4-socket-dir\") pod \"aws-ebs-csi-driver-node-gs462\" (UID: \"3d80fd9e-83d7-4c96-b196-1c9afac0bad4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gs462"
Jul 27 21:04:33.956636 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:04:33.955049    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/4275e4fb-a8e5-43a5-8b8c-ed4cc2094674-cnibin\") pod \"multus-additional-cni-plugins-2pbtk\" (UID: \"4275e4fb-a8e5-43a5-8b8c-ed4cc2094674\") " pod="openshift-multus/multus-additional-cni-plugins-2pbtk"
Jul 27 21:04:33.956636 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:04:33.955161    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/3d80fd9e-83d7-4c96-b196-1c9afac0bad4-socket-dir\") pod \"aws-ebs-csi-driver-node-gs462\" (UID: \"3d80fd9e-83d7-4c96-b196-1c9afac0bad4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gs462"
Jul 27 21:04:33.956636 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:04:33.956467    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/7f66b2c1-260e-4a55-902b-c4a060b723f9-ovn-node-metrics-cert\") pod \"ovnkube-node-qs89h\" (UID: \"7f66b2c1-260e-4a55-902b-c4a060b723f9\") " pod="openshift-ovn-kubernetes/ovnkube-node-qs89h"
Jul 27 21:04:33.956636 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:04:33.956596    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/4ff24566-f9aa-4eab-8795-a2808831cec1-agent-certs\") pod \"konnectivity-agent-tfvtq\" (UID: \"4ff24566-f9aa-4eab-8795-a2808831cec1\") " pod="kube-system/konnectivity-agent-tfvtq"
Jul 27 21:04:33.959857 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:04:33.959539    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xcn66\" (UniqueName: \"kubernetes.io/projected/c2d67a6e-7963-4c73-9983-40b775a8134a-kube-api-access-xcn66\") pod \"iptables-alerter-cd6w9\" (UID: \"c2d67a6e-7963-4c73-9983-40b775a8134a\") " pod="openshift-network-operator/iptables-alerter-cd6w9"
Jul 27 21:04:33.959964 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:04:33.959869    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p2xrm\" (UniqueName: \"kubernetes.io/projected/7f66b2c1-260e-4a55-902b-c4a060b723f9-kube-api-access-p2xrm\") pod \"ovnkube-node-qs89h\" (UID: \"7f66b2c1-260e-4a55-902b-c4a060b723f9\") " pod="openshift-ovn-kubernetes/ovnkube-node-qs89h"
Jul 27 21:04:33.960493 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:04:33.960473    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wz54v\" (UniqueName: \"kubernetes.io/projected/3e4c5c3f-db85-4e88-a1d5-48e602b0018a-kube-api-access-wz54v\") pod \"node-ca-qxzfs\" (UID: \"3e4c5c3f-db85-4e88-a1d5-48e602b0018a\") " pod="openshift-image-registry/node-ca-qxzfs"
Jul 27 21:04:33.960713 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:04:33.960689    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-np6md\" (UniqueName: \"kubernetes.io/projected/3d80fd9e-83d7-4c96-b196-1c9afac0bad4-kube-api-access-np6md\") pod \"aws-ebs-csi-driver-node-gs462\" (UID: \"3d80fd9e-83d7-4c96-b196-1c9afac0bad4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gs462"
Jul 27 21:04:33.963013 ip-10-0-131-133 kubenswrapper[2559]: E0727 21:04:33.962542    2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 27 21:04:33.963013 ip-10-0-131-133 kubenswrapper[2559]: E0727 21:04:33.962575    2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 27 21:04:33.963013 ip-10-0-131-133 kubenswrapper[2559]: E0727 21:04:33.962593    2559 projected.go:196] Error preparing data for projected volume kube-api-access-mtwdg for pod openshift-network-diagnostics/network-check-target-hmjv6: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 27 21:04:33.963013 ip-10-0-131-133 kubenswrapper[2559]: E0727 21:04:33.962705    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/95b2123e-132a-42a8-8edb-33ce19c2df51-kube-api-access-mtwdg podName:95b2123e-132a-42a8-8edb-33ce19c2df51 nodeName:}" failed. No retries permitted until 2026-07-27 21:04:34.462684254 +0000 UTC m=+3.041868356 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-mtwdg" (UniqueName: "kubernetes.io/projected/95b2123e-132a-42a8-8edb-33ce19c2df51-kube-api-access-mtwdg") pod "network-check-target-hmjv6" (UID: "95b2123e-132a-42a8-8edb-33ce19c2df51") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 27 21:04:33.964067 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:04:33.964017    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ws2v2\" (UniqueName: \"kubernetes.io/projected/4275e4fb-a8e5-43a5-8b8c-ed4cc2094674-kube-api-access-ws2v2\") pod \"multus-additional-cni-plugins-2pbtk\" (UID: \"4275e4fb-a8e5-43a5-8b8c-ed4cc2094674\") " pod="openshift-multus/multus-additional-cni-plugins-2pbtk"
Jul 27 21:04:33.965179 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:04:33.965161    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j6h2z\" (UniqueName: \"kubernetes.io/projected/6b7af06d-65e7-45ca-80ef-e88bcee2282a-kube-api-access-j6h2z\") pod \"multus-dchr6\" (UID: \"6b7af06d-65e7-45ca-80ef-e88bcee2282a\") " pod="openshift-multus/multus-dchr6"
Jul 27 21:04:33.981728 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:04:33.981663    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-133.ec2.internal" event={"ID":"c500a9e213c47a39763b09a5e1d404dc","Type":"ContainerStarted","Data":"86d409f6ab56d6e9a39124b2073f8f8db7c835df23cf757ecebb359017b0b03c"}
Jul 27 21:04:33.983006 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:04:33.982966    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-133.ec2.internal" event={"ID":"47a35874b1255611552eed095cad3eb8","Type":"ContainerStarted","Data":"3260d4f77f1566540d2a7c1c25801ac373a519a743dcd4fadf847c5a33fab314"}
Jul 27 21:04:34.055695 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:04:34.055658    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/e84e95da-4c32-4514-9c49-aaa0f96abea2-etc-systemd\") pod \"tuned-tfsvr\" (UID: \"e84e95da-4c32-4514-9c49-aaa0f96abea2\") " pod="openshift-cluster-node-tuning-operator/tuned-tfsvr"
Jul 27 21:04:34.055695 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:04:34.055698    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e84e95da-4c32-4514-9c49-aaa0f96abea2-sys\") pod \"tuned-tfsvr\" (UID: \"e84e95da-4c32-4514-9c49-aaa0f96abea2\") " pod="openshift-cluster-node-tuning-operator/tuned-tfsvr"
Jul 27 21:04:34.056018 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:04:34.055723    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/84a7d738-f486-43b0-ac43-3401891da8d1-hosts-file\") pod \"node-resolver-fvbmc\" (UID: \"84a7d738-f486-43b0-ac43-3401891da8d1\") " pod="openshift-dns/node-resolver-fvbmc"
Jul 27 21:04:34.056018 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:04:34.055746    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/e84e95da-4c32-4514-9c49-aaa0f96abea2-etc-sysctl-conf\") pod \"tuned-tfsvr\" (UID: \"e84e95da-4c32-4514-9c49-aaa0f96abea2\") " pod="openshift-cluster-node-tuning-operator/tuned-tfsvr"
Jul 27 21:04:34.056018 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:04:34.055791    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/e84e95da-4c32-4514-9c49-aaa0f96abea2-etc-systemd\") pod \"tuned-tfsvr\" (UID: \"e84e95da-4c32-4514-9c49-aaa0f96abea2\") " pod="openshift-cluster-node-tuning-operator/tuned-tfsvr"
Jul 27 21:04:34.056018 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:04:34.055799    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e84e95da-4c32-4514-9c49-aaa0f96abea2-sys\") pod \"tuned-tfsvr\" (UID: \"e84e95da-4c32-4514-9c49-aaa0f96abea2\") " pod="openshift-cluster-node-tuning-operator/tuned-tfsvr"
Jul 27 21:04:34.056018 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:04:34.055828    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/84a7d738-f486-43b0-ac43-3401891da8d1-hosts-file\") pod \"node-resolver-fvbmc\" (UID: \"84a7d738-f486-43b0-ac43-3401891da8d1\") " pod="openshift-dns/node-resolver-fvbmc"
Jul 27 21:04:34.056018 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:04:34.055902    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e84e95da-4c32-4514-9c49-aaa0f96abea2-etc-kubernetes\") pod \"tuned-tfsvr\" (UID: \"e84e95da-4c32-4514-9c49-aaa0f96abea2\") " pod="openshift-cluster-node-tuning-operator/tuned-tfsvr"
Jul 27 21:04:34.056018 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:04:34.055912    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/e84e95da-4c32-4514-9c49-aaa0f96abea2-etc-sysctl-conf\") pod \"tuned-tfsvr\" (UID: \"e84e95da-4c32-4514-9c49-aaa0f96abea2\") " pod="openshift-cluster-node-tuning-operator/tuned-tfsvr"
Jul 27 21:04:34.056018 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:04:34.055948    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/e84e95da-4c32-4514-9c49-aaa0f96abea2-etc-tuned\") pod \"tuned-tfsvr\" (UID: \"e84e95da-4c32-4514-9c49-aaa0f96abea2\") " pod="openshift-cluster-node-tuning-operator/tuned-tfsvr"
Jul 27 21:04:34.056018 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:04:34.055960    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e84e95da-4c32-4514-9c49-aaa0f96abea2-etc-kubernetes\") pod \"tuned-tfsvr\" (UID: \"e84e95da-4c32-4514-9c49-aaa0f96abea2\") " pod="openshift-cluster-node-tuning-operator/tuned-tfsvr"
Jul 27 21:04:34.056018 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:04:34.055979    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/e84e95da-4c32-4514-9c49-aaa0f96abea2-run\") pod \"tuned-tfsvr\" (UID: \"e84e95da-4c32-4514-9c49-aaa0f96abea2\") " pod="openshift-cluster-node-tuning-operator/tuned-tfsvr"
Jul 27 21:04:34.056018 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:04:34.056004    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/e84e95da-4c32-4514-9c49-aaa0f96abea2-host\") pod \"tuned-tfsvr\" (UID: \"e84e95da-4c32-4514-9c49-aaa0f96abea2\") " pod="openshift-cluster-node-tuning-operator/tuned-tfsvr"
Jul 27 21:04:34.056613 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:04:34.056033    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g25kk\" (UniqueName: \"kubernetes.io/projected/84a7d738-f486-43b0-ac43-3401891da8d1-kube-api-access-g25kk\") pod \"node-resolver-fvbmc\" (UID: \"84a7d738-f486-43b0-ac43-3401891da8d1\") " pod="openshift-dns/node-resolver-fvbmc"
Jul 27 21:04:34.056613 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:04:34.056065    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/e84e95da-4c32-4514-9c49-aaa0f96abea2-etc-sysconfig\") pod \"tuned-tfsvr\" (UID: \"e84e95da-4c32-4514-9c49-aaa0f96abea2\") " pod="openshift-cluster-node-tuning-operator/tuned-tfsvr"
Jul 27 21:04:34.056613 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:04:34.056090    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rqj57\" (UniqueName: \"kubernetes.io/projected/e84e95da-4c32-4514-9c49-aaa0f96abea2-kube-api-access-rqj57\") pod \"tuned-tfsvr\" (UID: \"e84e95da-4c32-4514-9c49-aaa0f96abea2\") " pod="openshift-cluster-node-tuning-operator/tuned-tfsvr"
Jul 27 21:04:34.056613 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:04:34.056100    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/e84e95da-4c32-4514-9c49-aaa0f96abea2-host\") pod \"tuned-tfsvr\" (UID: \"e84e95da-4c32-4514-9c49-aaa0f96abea2\") " pod="openshift-cluster-node-tuning-operator/tuned-tfsvr"
Jul 27 21:04:34.056613 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:04:34.056124    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e84e95da-4c32-4514-9c49-aaa0f96abea2-tmp\") pod \"tuned-tfsvr\" (UID: \"e84e95da-4c32-4514-9c49-aaa0f96abea2\") " pod="openshift-cluster-node-tuning-operator/tuned-tfsvr"
Jul 27 21:04:34.056613 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:04:34.056124    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/e84e95da-4c32-4514-9c49-aaa0f96abea2-run\") pod \"tuned-tfsvr\" (UID: \"e84e95da-4c32-4514-9c49-aaa0f96abea2\") " pod="openshift-cluster-node-tuning-operator/tuned-tfsvr"
Jul 27 21:04:34.056613 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:04:34.056153    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5851de39-848d-4a52-bc17-fefe1bf5407a-metrics-certs\") pod \"network-metrics-daemon-pr4c2\" (UID: \"5851de39-848d-4a52-bc17-fefe1bf5407a\") " pod="openshift-multus/network-metrics-daemon-pr4c2"
Jul 27 21:04:34.056613 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:04:34.056162    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/e84e95da-4c32-4514-9c49-aaa0f96abea2-etc-sysconfig\") pod \"tuned-tfsvr\" (UID: \"e84e95da-4c32-4514-9c49-aaa0f96abea2\") " pod="openshift-cluster-node-tuning-operator/tuned-tfsvr"
Jul 27 21:04:34.056613 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:04:34.056184    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bpmt9\" (UniqueName: \"kubernetes.io/projected/5851de39-848d-4a52-bc17-fefe1bf5407a-kube-api-access-bpmt9\") pod \"network-metrics-daemon-pr4c2\" (UID: \"5851de39-848d-4a52-bc17-fefe1bf5407a\") " pod="openshift-multus/network-metrics-daemon-pr4c2"
Jul 27 21:04:34.056613 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:04:34.056216    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/e84e95da-4c32-4514-9c49-aaa0f96abea2-etc-sysctl-d\") pod \"tuned-tfsvr\" (UID: \"e84e95da-4c32-4514-9c49-aaa0f96abea2\") " pod="openshift-cluster-node-tuning-operator/tuned-tfsvr"
Jul 27 21:04:34.056613 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:04:34.056240    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e84e95da-4c32-4514-9c49-aaa0f96abea2-var-lib-kubelet\") pod \"tuned-tfsvr\" (UID: \"e84e95da-4c32-4514-9c49-aaa0f96abea2\") " pod="openshift-cluster-node-tuning-operator/tuned-tfsvr"
Jul 27 21:04:34.056613 ip-10-0-131-133 kubenswrapper[2559]: E0727 21:04:34.056280    2559 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 27 21:04:34.056613 ip-10-0-131-133 kubenswrapper[2559]: E0727 21:04:34.056353    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5851de39-848d-4a52-bc17-fefe1bf5407a-metrics-certs podName:5851de39-848d-4a52-bc17-fefe1bf5407a nodeName:}" failed. No retries permitted until 2026-07-27 21:04:34.556332888 +0000 UTC m=+3.135516882 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/5851de39-848d-4a52-bc17-fefe1bf5407a-metrics-certs") pod "network-metrics-daemon-pr4c2" (UID: "5851de39-848d-4a52-bc17-fefe1bf5407a") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 27 21:04:34.056613 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:04:34.056266    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/e84e95da-4c32-4514-9c49-aaa0f96abea2-lib-modules\") pod \"tuned-tfsvr\" (UID: \"e84e95da-4c32-4514-9c49-aaa0f96abea2\") " pod="openshift-cluster-node-tuning-operator/tuned-tfsvr"
Jul 27 21:04:34.056613 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:04:34.056416    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/e84e95da-4c32-4514-9c49-aaa0f96abea2-etc-sysctl-d\") pod \"tuned-tfsvr\" (UID: \"e84e95da-4c32-4514-9c49-aaa0f96abea2\") " pod="openshift-cluster-node-tuning-operator/tuned-tfsvr"
Jul 27 21:04:34.056613 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:04:34.056432    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/e84e95da-4c32-4514-9c49-aaa0f96abea2-lib-modules\") pod \"tuned-tfsvr\" (UID: \"e84e95da-4c32-4514-9c49-aaa0f96abea2\") " pod="openshift-cluster-node-tuning-operator/tuned-tfsvr"
Jul 27 21:04:34.056613 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:04:34.056430    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/84a7d738-f486-43b0-ac43-3401891da8d1-tmp-dir\") pod \"node-resolver-fvbmc\" (UID: \"84a7d738-f486-43b0-ac43-3401891da8d1\") " pod="openshift-dns/node-resolver-fvbmc"
Jul 27 21:04:34.057453 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:04:34.056485    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/e84e95da-4c32-4514-9c49-aaa0f96abea2-etc-modprobe-d\") pod \"tuned-tfsvr\" (UID: \"e84e95da-4c32-4514-9c49-aaa0f96abea2\") " pod="openshift-cluster-node-tuning-operator/tuned-tfsvr"
Jul 27 21:04:34.057453 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:04:34.056548    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e84e95da-4c32-4514-9c49-aaa0f96abea2-var-lib-kubelet\") pod \"tuned-tfsvr\" (UID: \"e84e95da-4c32-4514-9c49-aaa0f96abea2\") " pod="openshift-cluster-node-tuning-operator/tuned-tfsvr"
Jul 27 21:04:34.057453 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:04:34.056635    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/e84e95da-4c32-4514-9c49-aaa0f96abea2-etc-modprobe-d\") pod \"tuned-tfsvr\" (UID: \"e84e95da-4c32-4514-9c49-aaa0f96abea2\") " pod="openshift-cluster-node-tuning-operator/tuned-tfsvr"
Jul 27 21:04:34.057453 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:04:34.056709    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/84a7d738-f486-43b0-ac43-3401891da8d1-tmp-dir\") pod \"node-resolver-fvbmc\" (UID: \"84a7d738-f486-43b0-ac43-3401891da8d1\") " pod="openshift-dns/node-resolver-fvbmc"
Jul 27 21:04:34.058651 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:04:34.058626    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/e84e95da-4c32-4514-9c49-aaa0f96abea2-etc-tuned\") pod \"tuned-tfsvr\" (UID: \"e84e95da-4c32-4514-9c49-aaa0f96abea2\") " pod="openshift-cluster-node-tuning-operator/tuned-tfsvr"
Jul 27 21:04:34.058761 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:04:34.058716    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e84e95da-4c32-4514-9c49-aaa0f96abea2-tmp\") pod \"tuned-tfsvr\" (UID: \"e84e95da-4c32-4514-9c49-aaa0f96abea2\") " pod="openshift-cluster-node-tuning-operator/tuned-tfsvr"
Jul 27 21:04:34.066372 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:04:34.066294    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bpmt9\" (UniqueName: \"kubernetes.io/projected/5851de39-848d-4a52-bc17-fefe1bf5407a-kube-api-access-bpmt9\") pod \"network-metrics-daemon-pr4c2\" (UID: \"5851de39-848d-4a52-bc17-fefe1bf5407a\") " pod="openshift-multus/network-metrics-daemon-pr4c2"
Jul 27 21:04:34.066564 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:04:34.066540    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rqj57\" (UniqueName: \"kubernetes.io/projected/e84e95da-4c32-4514-9c49-aaa0f96abea2-kube-api-access-rqj57\") pod \"tuned-tfsvr\" (UID: \"e84e95da-4c32-4514-9c49-aaa0f96abea2\") " pod="openshift-cluster-node-tuning-operator/tuned-tfsvr"
Jul 27 21:04:34.066564 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:04:34.066557    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g25kk\" (UniqueName: \"kubernetes.io/projected/84a7d738-f486-43b0-ac43-3401891da8d1-kube-api-access-g25kk\") pod \"node-resolver-fvbmc\" (UID: \"84a7d738-f486-43b0-ac43-3401891da8d1\") " pod="openshift-dns/node-resolver-fvbmc"
Jul 27 21:04:34.143900 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:04:34.143863    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-cd6w9"
Jul 27 21:04:34.157663 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:04:34.157622    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-qs89h"
Jul 27 21:04:34.166756 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:04:34.166727    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-2pbtk"
Jul 27 21:04:34.175027 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:04:34.175001    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-dchr6"
Jul 27 21:04:34.182894 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:04:34.182858    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gs462"
Jul 27 21:04:34.190804 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:04:34.190776    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-tfvtq"
Jul 27 21:04:34.198781 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:04:34.198744    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-qxzfs"
Jul 27 21:04:34.206776 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:04:34.206751    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-tfsvr"
Jul 27 21:04:34.213638 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:04:34.213612    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-fvbmc"
Jul 27 21:04:34.460839 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:04:34.460741    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/00ef4d6d-0b8e-4ece-ba97-36a9e236ec53-original-pull-secret\") pod \"global-pull-secret-syncer-rxl8w\" (UID: \"00ef4d6d-0b8e-4ece-ba97-36a9e236ec53\") " pod="kube-system/global-pull-secret-syncer-rxl8w"
Jul 27 21:04:34.461004 ip-10-0-131-133 kubenswrapper[2559]: E0727 21:04:34.460893    2559 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 27 21:04:34.461004 ip-10-0-131-133 kubenswrapper[2559]: E0727 21:04:34.460978    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/00ef4d6d-0b8e-4ece-ba97-36a9e236ec53-original-pull-secret podName:00ef4d6d-0b8e-4ece-ba97-36a9e236ec53 nodeName:}" failed. No retries permitted until 2026-07-27 21:04:35.460957188 +0000 UTC m=+4.040141182 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/00ef4d6d-0b8e-4ece-ba97-36a9e236ec53-original-pull-secret") pod "global-pull-secret-syncer-rxl8w" (UID: "00ef4d6d-0b8e-4ece-ba97-36a9e236ec53") : object "kube-system"/"original-pull-secret" not registered
Jul 27 21:04:34.561135 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:04:34.561092    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5851de39-848d-4a52-bc17-fefe1bf5407a-metrics-certs\") pod \"network-metrics-daemon-pr4c2\" (UID: \"5851de39-848d-4a52-bc17-fefe1bf5407a\") " pod="openshift-multus/network-metrics-daemon-pr4c2"
Jul 27 21:04:34.561339 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:04:34.561164    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mtwdg\" (UniqueName: \"kubernetes.io/projected/95b2123e-132a-42a8-8edb-33ce19c2df51-kube-api-access-mtwdg\") pod \"network-check-target-hmjv6\" (UID: \"95b2123e-132a-42a8-8edb-33ce19c2df51\") " pod="openshift-network-diagnostics/network-check-target-hmjv6"
Jul 27 21:04:34.561339 ip-10-0-131-133 kubenswrapper[2559]: E0727 21:04:34.561299    2559 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 27 21:04:34.561339 ip-10-0-131-133 kubenswrapper[2559]: E0727 21:04:34.561316    2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 27 21:04:34.561339 ip-10-0-131-133 kubenswrapper[2559]: E0727 21:04:34.561332    2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 27 21:04:34.561339 ip-10-0-131-133 kubenswrapper[2559]: E0727 21:04:34.561343    2559 projected.go:196] Error preparing data for projected volume kube-api-access-mtwdg for pod openshift-network-diagnostics/network-check-target-hmjv6: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 27 21:04:34.561565 ip-10-0-131-133 kubenswrapper[2559]: E0727 21:04:34.561385    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5851de39-848d-4a52-bc17-fefe1bf5407a-metrics-certs podName:5851de39-848d-4a52-bc17-fefe1bf5407a nodeName:}" failed. No retries permitted until 2026-07-27 21:04:35.56136283 +0000 UTC m=+4.140546821 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/5851de39-848d-4a52-bc17-fefe1bf5407a-metrics-certs") pod "network-metrics-daemon-pr4c2" (UID: "5851de39-848d-4a52-bc17-fefe1bf5407a") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 27 21:04:34.561565 ip-10-0-131-133 kubenswrapper[2559]: E0727 21:04:34.561410    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/95b2123e-132a-42a8-8edb-33ce19c2df51-kube-api-access-mtwdg podName:95b2123e-132a-42a8-8edb-33ce19c2df51 nodeName:}" failed. No retries permitted until 2026-07-27 21:04:35.561399555 +0000 UTC m=+4.140583543 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-mtwdg" (UniqueName: "kubernetes.io/projected/95b2123e-132a-42a8-8edb-33ce19c2df51-kube-api-access-mtwdg") pod "network-check-target-hmjv6" (UID: "95b2123e-132a-42a8-8edb-33ce19c2df51") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 27 21:04:34.888936 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:04:34.888831    2559 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-26 20:59:34 +0000 UTC" deadline="2028-01-03 18:02:44.386562817 +0000 UTC"
Jul 27 21:04:34.888936 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:04:34.888882    2559 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12596h58m9.497683579s"
Jul 27 21:04:34.968909 ip-10-0-131-133 kubenswrapper[2559]: W0727 21:04:34.968883    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod84a7d738_f486_43b0_ac43_3401891da8d1.slice/crio-9d444a3f84c7c2ce5fcc4f2169802076e210b70e04aef7b7d2a8568ff6ed8558 WatchSource:0}: Error finding container 9d444a3f84c7c2ce5fcc4f2169802076e210b70e04aef7b7d2a8568ff6ed8558: Status 404 returned error can't find the container with id 9d444a3f84c7c2ce5fcc4f2169802076e210b70e04aef7b7d2a8568ff6ed8558
Jul 27 21:04:34.970847 ip-10-0-131-133 kubenswrapper[2559]: W0727 21:04:34.970814    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3d80fd9e_83d7_4c96_b196_1c9afac0bad4.slice/crio-d5adaa3a40134b81837942d1839aa40df2e67553821ada281624b02c71f60539 WatchSource:0}: Error finding container d5adaa3a40134b81837942d1839aa40df2e67553821ada281624b02c71f60539: Status 404 returned error can't find the container with id d5adaa3a40134b81837942d1839aa40df2e67553821ada281624b02c71f60539
Jul 27 21:04:34.972190 ip-10-0-131-133 kubenswrapper[2559]: W0727 21:04:34.972041    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode84e95da_4c32_4514_9c49_aaa0f96abea2.slice/crio-ca50c50031fc6c8aaae78c4aae62a97db6c7836728e1413be310cae5b1b1fb42 WatchSource:0}: Error finding container ca50c50031fc6c8aaae78c4aae62a97db6c7836728e1413be310cae5b1b1fb42: Status 404 returned error can't find the container with id ca50c50031fc6c8aaae78c4aae62a97db6c7836728e1413be310cae5b1b1fb42
Jul 27 21:04:34.972629 ip-10-0-131-133 kubenswrapper[2559]: W0727 21:04:34.972581    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc2d67a6e_7963_4c73_9983_40b775a8134a.slice/crio-6b4503997ae123f86bd1b205efbe604272868ef10b4c185bd3b0c4fafb79679c WatchSource:0}: Error finding container 6b4503997ae123f86bd1b205efbe604272868ef10b4c185bd3b0c4fafb79679c: Status 404 returned error can't find the container with id 6b4503997ae123f86bd1b205efbe604272868ef10b4c185bd3b0c4fafb79679c
Jul 27 21:04:34.976204 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:04:34.975846    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pr4c2"
Jul 27 21:04:34.976204 ip-10-0-131-133 kubenswrapper[2559]: E0727 21:04:34.975978    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pr4c2" podUID="5851de39-848d-4a52-bc17-fefe1bf5407a"
Jul 27 21:04:34.976204 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:04:34.976022    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rxl8w"
Jul 27 21:04:34.976204 ip-10-0-131-133 kubenswrapper[2559]: E0727 21:04:34.976099    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rxl8w" podUID="00ef4d6d-0b8e-4ece-ba97-36a9e236ec53"
Jul 27 21:04:34.976204 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:04:34.976139    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hmjv6"
Jul 27 21:04:34.976426 ip-10-0-131-133 kubenswrapper[2559]: E0727 21:04:34.976282    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hmjv6" podUID="95b2123e-132a-42a8-8edb-33ce19c2df51"
Jul 27 21:04:34.976426 ip-10-0-131-133 kubenswrapper[2559]: W0727 21:04:34.976380    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4275e4fb_a8e5_43a5_8b8c_ed4cc2094674.slice/crio-d05d411ddb1c20b82ca58e0289900f1a8f35217db2b7ac2a4457332fb6d7bbd0 WatchSource:0}: Error finding container d05d411ddb1c20b82ca58e0289900f1a8f35217db2b7ac2a4457332fb6d7bbd0: Status 404 returned error can't find the container with id d05d411ddb1c20b82ca58e0289900f1a8f35217db2b7ac2a4457332fb6d7bbd0
Jul 27 21:04:34.978411 ip-10-0-131-133 kubenswrapper[2559]: W0727 21:04:34.976987    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6b7af06d_65e7_45ca_80ef_e88bcee2282a.slice/crio-a1d11f5ef134239100581695d6e86493e788c82c19706f3d11b2355f9a42b060 WatchSource:0}: Error finding container a1d11f5ef134239100581695d6e86493e788c82c19706f3d11b2355f9a42b060: Status 404 returned error can't find the container with id a1d11f5ef134239100581695d6e86493e788c82c19706f3d11b2355f9a42b060
Jul 27 21:04:34.978411 ip-10-0-131-133 kubenswrapper[2559]: W0727 21:04:34.978332    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4ff24566_f9aa_4eab_8795_a2808831cec1.slice/crio-714f141758295fa8b30022111a695b72b90778a66b93832086dd0748f53aaddd WatchSource:0}: Error finding container 714f141758295fa8b30022111a695b72b90778a66b93832086dd0748f53aaddd: Status 404 returned error can't find the container with id 714f141758295fa8b30022111a695b72b90778a66b93832086dd0748f53aaddd
Jul 27 21:04:34.980760 ip-10-0-131-133 kubenswrapper[2559]: W0727 21:04:34.980730    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7f66b2c1_260e_4a55_902b_c4a060b723f9.slice/crio-6253cbf97cb28185a67f17e752e16937ec394ed9cdab1e51eca06589941641e6 WatchSource:0}: Error finding container 6253cbf97cb28185a67f17e752e16937ec394ed9cdab1e51eca06589941641e6: Status 404 returned error can't find the container with id 6253cbf97cb28185a67f17e752e16937ec394ed9cdab1e51eca06589941641e6
Jul 27 21:04:34.983590 ip-10-0-131-133 kubenswrapper[2559]: W0727 21:04:34.983562    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3e4c5c3f_db85_4e88_a1d5_48e602b0018a.slice/crio-b486c109d4bfa9f4f70093d80687a1a58466d6b2901a54ad5748bae99efc39e1 WatchSource:0}: Error finding container b486c109d4bfa9f4f70093d80687a1a58466d6b2901a54ad5748bae99efc39e1: Status 404 returned error can't find the container with id b486c109d4bfa9f4f70093d80687a1a58466d6b2901a54ad5748bae99efc39e1
Jul 27 21:04:34.988699 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:04:34.988677    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-dchr6" event={"ID":"6b7af06d-65e7-45ca-80ef-e88bcee2282a","Type":"ContainerStarted","Data":"a1d11f5ef134239100581695d6e86493e788c82c19706f3d11b2355f9a42b060"}
Jul 27 21:04:34.990464 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:04:34.990439    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-tfvtq" event={"ID":"4ff24566-f9aa-4eab-8795-a2808831cec1","Type":"ContainerStarted","Data":"714f141758295fa8b30022111a695b72b90778a66b93832086dd0748f53aaddd"}
Jul 27 21:04:34.991933 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:04:34.991893    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-cd6w9" event={"ID":"c2d67a6e-7963-4c73-9983-40b775a8134a","Type":"ContainerStarted","Data":"6b4503997ae123f86bd1b205efbe604272868ef10b4c185bd3b0c4fafb79679c"}
Jul 27 21:04:34.993304 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:04:34.993193    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-tfsvr" event={"ID":"e84e95da-4c32-4514-9c49-aaa0f96abea2","Type":"ContainerStarted","Data":"ca50c50031fc6c8aaae78c4aae62a97db6c7836728e1413be310cae5b1b1fb42"}
Jul 27 21:04:34.994560 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:04:34.994539    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-fvbmc" event={"ID":"84a7d738-f486-43b0-ac43-3401891da8d1","Type":"ContainerStarted","Data":"9d444a3f84c7c2ce5fcc4f2169802076e210b70e04aef7b7d2a8568ff6ed8558"}
Jul 27 21:04:34.995786 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:04:34.995762    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-qxzfs" event={"ID":"3e4c5c3f-db85-4e88-a1d5-48e602b0018a","Type":"ContainerStarted","Data":"b486c109d4bfa9f4f70093d80687a1a58466d6b2901a54ad5748bae99efc39e1"}
Jul 27 21:04:34.996973 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:04:34.996953    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-2pbtk" event={"ID":"4275e4fb-a8e5-43a5-8b8c-ed4cc2094674","Type":"ContainerStarted","Data":"d05d411ddb1c20b82ca58e0289900f1a8f35217db2b7ac2a4457332fb6d7bbd0"}
Jul 27 21:04:34.998221 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:04:34.998193    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gs462" event={"ID":"3d80fd9e-83d7-4c96-b196-1c9afac0bad4","Type":"ContainerStarted","Data":"d5adaa3a40134b81837942d1839aa40df2e67553821ada281624b02c71f60539"}
Jul 27 21:04:35.469281 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:04:35.468953    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/00ef4d6d-0b8e-4ece-ba97-36a9e236ec53-original-pull-secret\") pod \"global-pull-secret-syncer-rxl8w\" (UID: \"00ef4d6d-0b8e-4ece-ba97-36a9e236ec53\") " pod="kube-system/global-pull-secret-syncer-rxl8w"
Jul 27 21:04:35.469478 ip-10-0-131-133 kubenswrapper[2559]: E0727 21:04:35.469370    2559 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 27 21:04:35.469478 ip-10-0-131-133 kubenswrapper[2559]: E0727 21:04:35.469442    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/00ef4d6d-0b8e-4ece-ba97-36a9e236ec53-original-pull-secret podName:00ef4d6d-0b8e-4ece-ba97-36a9e236ec53 nodeName:}" failed. No retries permitted until 2026-07-27 21:04:37.469419351 +0000 UTC m=+6.048603342 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/00ef4d6d-0b8e-4ece-ba97-36a9e236ec53-original-pull-secret") pod "global-pull-secret-syncer-rxl8w" (UID: "00ef4d6d-0b8e-4ece-ba97-36a9e236ec53") : object "kube-system"/"original-pull-secret" not registered
Jul 27 21:04:35.570680 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:04:35.570290    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mtwdg\" (UniqueName: \"kubernetes.io/projected/95b2123e-132a-42a8-8edb-33ce19c2df51-kube-api-access-mtwdg\") pod \"network-check-target-hmjv6\" (UID: \"95b2123e-132a-42a8-8edb-33ce19c2df51\") " pod="openshift-network-diagnostics/network-check-target-hmjv6"
Jul 27 21:04:35.570680 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:04:35.570376    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5851de39-848d-4a52-bc17-fefe1bf5407a-metrics-certs\") pod \"network-metrics-daemon-pr4c2\" (UID: \"5851de39-848d-4a52-bc17-fefe1bf5407a\") " pod="openshift-multus/network-metrics-daemon-pr4c2"
Jul 27 21:04:35.570680 ip-10-0-131-133 kubenswrapper[2559]: E0727 21:04:35.570461    2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 27 21:04:35.570680 ip-10-0-131-133 kubenswrapper[2559]: E0727 21:04:35.570496    2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 27 21:04:35.570680 ip-10-0-131-133 kubenswrapper[2559]: E0727 21:04:35.570522    2559 projected.go:196] Error preparing data for projected volume kube-api-access-mtwdg for pod openshift-network-diagnostics/network-check-target-hmjv6: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 27 21:04:35.570680 ip-10-0-131-133 kubenswrapper[2559]: E0727 21:04:35.570578    2559 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 27 21:04:35.570680 ip-10-0-131-133 kubenswrapper[2559]: E0727 21:04:35.570590    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/95b2123e-132a-42a8-8edb-33ce19c2df51-kube-api-access-mtwdg podName:95b2123e-132a-42a8-8edb-33ce19c2df51 nodeName:}" failed. No retries permitted until 2026-07-27 21:04:37.570569486 +0000 UTC m=+6.149753476 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-mtwdg" (UniqueName: "kubernetes.io/projected/95b2123e-132a-42a8-8edb-33ce19c2df51-kube-api-access-mtwdg") pod "network-check-target-hmjv6" (UID: "95b2123e-132a-42a8-8edb-33ce19c2df51") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 27 21:04:35.570680 ip-10-0-131-133 kubenswrapper[2559]: E0727 21:04:35.570636    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5851de39-848d-4a52-bc17-fefe1bf5407a-metrics-certs podName:5851de39-848d-4a52-bc17-fefe1bf5407a nodeName:}" failed. No retries permitted until 2026-07-27 21:04:37.570619676 +0000 UTC m=+6.149803666 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/5851de39-848d-4a52-bc17-fefe1bf5407a-metrics-certs") pod "network-metrics-daemon-pr4c2" (UID: "5851de39-848d-4a52-bc17-fefe1bf5407a") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 27 21:04:35.889831 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:04:35.889786    2559 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-26 20:59:34 +0000 UTC" deadline="2028-02-08 06:19:37.728726429 +0000 UTC"
Jul 27 21:04:35.889831 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:04:35.889830    2559 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13449h15m1.838900868s"
Jul 27 21:04:36.011607 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:04:36.011474    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-qs89h" event={"ID":"7f66b2c1-260e-4a55-902b-c4a060b723f9","Type":"ContainerStarted","Data":"6253cbf97cb28185a67f17e752e16937ec394ed9cdab1e51eca06589941641e6"}
Jul 27 21:04:36.044581 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:04:36.043541    2559 generic.go:356] "Generic (PLEG): container finished" podID="47a35874b1255611552eed095cad3eb8" containerID="1ef7fc8c81fe60d168f208747525556cdd4ed89bf92e06c6dc948717cfe78492" exitCode=0
Jul 27 21:04:36.044581 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:04:36.043647    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-133.ec2.internal" event={"ID":"47a35874b1255611552eed095cad3eb8","Type":"ContainerDied","Data":"1ef7fc8c81fe60d168f208747525556cdd4ed89bf92e06c6dc948717cfe78492"}
Jul 27 21:04:36.100494 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:04:36.100455    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-133.ec2.internal" event={"ID":"c500a9e213c47a39763b09a5e1d404dc","Type":"ContainerStarted","Data":"b4b8ecf00013d6ff36f91ece886bc47c3977a84811814d4d52978a1ff47c06f4"}
Jul 27 21:04:36.118761 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:04:36.118680    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-133.ec2.internal" podStartSLOduration=3.118656277 podStartE2EDuration="3.118656277s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-27 21:04:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 21:04:36.117734972 +0000 UTC m=+4.696918985" watchObservedRunningTime="2026-07-27 21:04:36.118656277 +0000 UTC m=+4.697840289"
Jul 27 21:04:36.976370 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:04:36.976239    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rxl8w"
Jul 27 21:04:36.977146 ip-10-0-131-133 kubenswrapper[2559]: E0727 21:04:36.976384    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rxl8w" podUID="00ef4d6d-0b8e-4ece-ba97-36a9e236ec53"
Jul 27 21:04:36.977146 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:04:36.976847    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hmjv6"
Jul 27 21:04:36.977146 ip-10-0-131-133 kubenswrapper[2559]: E0727 21:04:36.976947    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hmjv6" podUID="95b2123e-132a-42a8-8edb-33ce19c2df51"
Jul 27 21:04:36.977146 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:04:36.977013    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pr4c2"
Jul 27 21:04:36.977146 ip-10-0-131-133 kubenswrapper[2559]: E0727 21:04:36.977113    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pr4c2" podUID="5851de39-848d-4a52-bc17-fefe1bf5407a"
Jul 27 21:04:37.144250 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:04:37.144040    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-133.ec2.internal" event={"ID":"47a35874b1255611552eed095cad3eb8","Type":"ContainerStarted","Data":"3778086dd5fd47d58462a330f5ee928fbee69c2a029a09aa41618b962b95d387"}
Jul 27 21:04:37.491005 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:04:37.490961    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/00ef4d6d-0b8e-4ece-ba97-36a9e236ec53-original-pull-secret\") pod \"global-pull-secret-syncer-rxl8w\" (UID: \"00ef4d6d-0b8e-4ece-ba97-36a9e236ec53\") " pod="kube-system/global-pull-secret-syncer-rxl8w"
Jul 27 21:04:37.491211 ip-10-0-131-133 kubenswrapper[2559]: E0727 21:04:37.491144    2559 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 27 21:04:37.491211 ip-10-0-131-133 kubenswrapper[2559]: E0727 21:04:37.491209    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/00ef4d6d-0b8e-4ece-ba97-36a9e236ec53-original-pull-secret podName:00ef4d6d-0b8e-4ece-ba97-36a9e236ec53 nodeName:}" failed. No retries permitted until 2026-07-27 21:04:41.491189985 +0000 UTC m=+10.070373977 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/00ef4d6d-0b8e-4ece-ba97-36a9e236ec53-original-pull-secret") pod "global-pull-secret-syncer-rxl8w" (UID: "00ef4d6d-0b8e-4ece-ba97-36a9e236ec53") : object "kube-system"/"original-pull-secret" not registered
Jul 27 21:04:37.591665 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:04:37.591629    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mtwdg\" (UniqueName: \"kubernetes.io/projected/95b2123e-132a-42a8-8edb-33ce19c2df51-kube-api-access-mtwdg\") pod \"network-check-target-hmjv6\" (UID: \"95b2123e-132a-42a8-8edb-33ce19c2df51\") " pod="openshift-network-diagnostics/network-check-target-hmjv6"
Jul 27 21:04:37.591868 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:04:37.591701    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5851de39-848d-4a52-bc17-fefe1bf5407a-metrics-certs\") pod \"network-metrics-daemon-pr4c2\" (UID: \"5851de39-848d-4a52-bc17-fefe1bf5407a\") " pod="openshift-multus/network-metrics-daemon-pr4c2"
Jul 27 21:04:37.591868 ip-10-0-131-133 kubenswrapper[2559]: E0727 21:04:37.591842    2559 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 27 21:04:37.591997 ip-10-0-131-133 kubenswrapper[2559]: E0727 21:04:37.591906    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5851de39-848d-4a52-bc17-fefe1bf5407a-metrics-certs podName:5851de39-848d-4a52-bc17-fefe1bf5407a nodeName:}" failed. No retries permitted until 2026-07-27 21:04:41.591885806 +0000 UTC m=+10.171069801 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/5851de39-848d-4a52-bc17-fefe1bf5407a-metrics-certs") pod "network-metrics-daemon-pr4c2" (UID: "5851de39-848d-4a52-bc17-fefe1bf5407a") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 27 21:04:37.592368 ip-10-0-131-133 kubenswrapper[2559]: E0727 21:04:37.592349    2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 27 21:04:37.592462 ip-10-0-131-133 kubenswrapper[2559]: E0727 21:04:37.592374    2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 27 21:04:37.592462 ip-10-0-131-133 kubenswrapper[2559]: E0727 21:04:37.592388    2559 projected.go:196] Error preparing data for projected volume kube-api-access-mtwdg for pod openshift-network-diagnostics/network-check-target-hmjv6: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 27 21:04:37.592462 ip-10-0-131-133 kubenswrapper[2559]: E0727 21:04:37.592437    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/95b2123e-132a-42a8-8edb-33ce19c2df51-kube-api-access-mtwdg podName:95b2123e-132a-42a8-8edb-33ce19c2df51 nodeName:}" failed. No retries permitted until 2026-07-27 21:04:41.592420695 +0000 UTC m=+10.171604688 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-mtwdg" (UniqueName: "kubernetes.io/projected/95b2123e-132a-42a8-8edb-33ce19c2df51-kube-api-access-mtwdg") pod "network-check-target-hmjv6" (UID: "95b2123e-132a-42a8-8edb-33ce19c2df51") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 27 21:04:38.976562 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:04:38.976524    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rxl8w"
Jul 27 21:04:38.977128 ip-10-0-131-133 kubenswrapper[2559]: E0727 21:04:38.976673    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rxl8w" podUID="00ef4d6d-0b8e-4ece-ba97-36a9e236ec53"
Jul 27 21:04:38.977128 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:04:38.977080    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hmjv6"
Jul 27 21:04:38.977309 ip-10-0-131-133 kubenswrapper[2559]: E0727 21:04:38.977177    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hmjv6" podUID="95b2123e-132a-42a8-8edb-33ce19c2df51"
Jul 27 21:04:38.977309 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:04:38.977233    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pr4c2"
Jul 27 21:04:38.977428 ip-10-0-131-133 kubenswrapper[2559]: E0727 21:04:38.977348    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pr4c2" podUID="5851de39-848d-4a52-bc17-fefe1bf5407a"
Jul 27 21:04:40.975968 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:04:40.975935    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rxl8w"
Jul 27 21:04:40.975968 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:04:40.975968    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hmjv6"
Jul 27 21:04:40.976427 ip-10-0-131-133 kubenswrapper[2559]: E0727 21:04:40.976071    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rxl8w" podUID="00ef4d6d-0b8e-4ece-ba97-36a9e236ec53"
Jul 27 21:04:40.976427 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:04:40.976135    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pr4c2"
Jul 27 21:04:40.976427 ip-10-0-131-133 kubenswrapper[2559]: E0727 21:04:40.976128    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hmjv6" podUID="95b2123e-132a-42a8-8edb-33ce19c2df51"
Jul 27 21:04:40.976427 ip-10-0-131-133 kubenswrapper[2559]: E0727 21:04:40.976189    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pr4c2" podUID="5851de39-848d-4a52-bc17-fefe1bf5407a"
Jul 27 21:04:41.527865 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:04:41.527328    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/00ef4d6d-0b8e-4ece-ba97-36a9e236ec53-original-pull-secret\") pod \"global-pull-secret-syncer-rxl8w\" (UID: \"00ef4d6d-0b8e-4ece-ba97-36a9e236ec53\") " pod="kube-system/global-pull-secret-syncer-rxl8w"
Jul 27 21:04:41.527865 ip-10-0-131-133 kubenswrapper[2559]: E0727 21:04:41.527471    2559 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 27 21:04:41.527865 ip-10-0-131-133 kubenswrapper[2559]: E0727 21:04:41.527534    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/00ef4d6d-0b8e-4ece-ba97-36a9e236ec53-original-pull-secret podName:00ef4d6d-0b8e-4ece-ba97-36a9e236ec53 nodeName:}" failed. No retries permitted until 2026-07-27 21:04:49.527515707 +0000 UTC m=+18.106699694 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/00ef4d6d-0b8e-4ece-ba97-36a9e236ec53-original-pull-secret") pod "global-pull-secret-syncer-rxl8w" (UID: "00ef4d6d-0b8e-4ece-ba97-36a9e236ec53") : object "kube-system"/"original-pull-secret" not registered
Jul 27 21:04:41.628137 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:04:41.628081    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5851de39-848d-4a52-bc17-fefe1bf5407a-metrics-certs\") pod \"network-metrics-daemon-pr4c2\" (UID: \"5851de39-848d-4a52-bc17-fefe1bf5407a\") " pod="openshift-multus/network-metrics-daemon-pr4c2"
Jul 27 21:04:41.628339 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:04:41.628184    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mtwdg\" (UniqueName: \"kubernetes.io/projected/95b2123e-132a-42a8-8edb-33ce19c2df51-kube-api-access-mtwdg\") pod \"network-check-target-hmjv6\" (UID: \"95b2123e-132a-42a8-8edb-33ce19c2df51\") " pod="openshift-network-diagnostics/network-check-target-hmjv6"
Jul 27 21:04:41.628399 ip-10-0-131-133 kubenswrapper[2559]: E0727 21:04:41.628359    2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 27 21:04:41.628399 ip-10-0-131-133 kubenswrapper[2559]: E0727 21:04:41.628390    2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 27 21:04:41.628489 ip-10-0-131-133 kubenswrapper[2559]: E0727 21:04:41.628404    2559 projected.go:196] Error preparing data for projected volume kube-api-access-mtwdg for pod openshift-network-diagnostics/network-check-target-hmjv6: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 27 21:04:41.628489 ip-10-0-131-133 kubenswrapper[2559]: E0727 21:04:41.628464    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/95b2123e-132a-42a8-8edb-33ce19c2df51-kube-api-access-mtwdg podName:95b2123e-132a-42a8-8edb-33ce19c2df51 nodeName:}" failed. No retries permitted until 2026-07-27 21:04:49.628444577 +0000 UTC m=+18.207628564 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-mtwdg" (UniqueName: "kubernetes.io/projected/95b2123e-132a-42a8-8edb-33ce19c2df51-kube-api-access-mtwdg") pod "network-check-target-hmjv6" (UID: "95b2123e-132a-42a8-8edb-33ce19c2df51") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 27 21:04:41.628832 ip-10-0-131-133 kubenswrapper[2559]: E0727 21:04:41.628717    2559 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 27 21:04:41.628832 ip-10-0-131-133 kubenswrapper[2559]: E0727 21:04:41.628784    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5851de39-848d-4a52-bc17-fefe1bf5407a-metrics-certs podName:5851de39-848d-4a52-bc17-fefe1bf5407a nodeName:}" failed. No retries permitted until 2026-07-27 21:04:49.628766466 +0000 UTC m=+18.207950466 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/5851de39-848d-4a52-bc17-fefe1bf5407a-metrics-certs") pod "network-metrics-daemon-pr4c2" (UID: "5851de39-848d-4a52-bc17-fefe1bf5407a") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 27 21:04:42.976307 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:04:42.976274    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rxl8w"
Jul 27 21:04:42.976760 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:04:42.976328    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pr4c2"
Jul 27 21:04:42.976760 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:04:42.976286    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hmjv6"
Jul 27 21:04:42.976760 ip-10-0-131-133 kubenswrapper[2559]: E0727 21:04:42.976419    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rxl8w" podUID="00ef4d6d-0b8e-4ece-ba97-36a9e236ec53"
Jul 27 21:04:42.976760 ip-10-0-131-133 kubenswrapper[2559]: E0727 21:04:42.976475    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hmjv6" podUID="95b2123e-132a-42a8-8edb-33ce19c2df51"
Jul 27 21:04:42.976760 ip-10-0-131-133 kubenswrapper[2559]: E0727 21:04:42.976569    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pr4c2" podUID="5851de39-848d-4a52-bc17-fefe1bf5407a"
Jul 27 21:04:44.975901 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:04:44.975864    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rxl8w"
Jul 27 21:04:44.976338 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:04:44.975864    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pr4c2"
Jul 27 21:04:44.976338 ip-10-0-131-133 kubenswrapper[2559]: E0727 21:04:44.975980    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rxl8w" podUID="00ef4d6d-0b8e-4ece-ba97-36a9e236ec53"
Jul 27 21:04:44.976338 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:04:44.975864    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hmjv6"
Jul 27 21:04:44.976338 ip-10-0-131-133 kubenswrapper[2559]: E0727 21:04:44.976061    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pr4c2" podUID="5851de39-848d-4a52-bc17-fefe1bf5407a"
Jul 27 21:04:44.976338 ip-10-0-131-133 kubenswrapper[2559]: E0727 21:04:44.976124    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hmjv6" podUID="95b2123e-132a-42a8-8edb-33ce19c2df51"
Jul 27 21:04:46.975893 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:04:46.975856    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pr4c2"
Jul 27 21:04:46.976340 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:04:46.975924    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hmjv6"
Jul 27 21:04:46.976340 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:04:46.975955    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rxl8w"
Jul 27 21:04:46.976340 ip-10-0-131-133 kubenswrapper[2559]: E0727 21:04:46.976081    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hmjv6" podUID="95b2123e-132a-42a8-8edb-33ce19c2df51"
Jul 27 21:04:46.976340 ip-10-0-131-133 kubenswrapper[2559]: E0727 21:04:46.976150    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rxl8w" podUID="00ef4d6d-0b8e-4ece-ba97-36a9e236ec53"
Jul 27 21:04:46.976340 ip-10-0-131-133 kubenswrapper[2559]: E0727 21:04:46.976230    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pr4c2" podUID="5851de39-848d-4a52-bc17-fefe1bf5407a"
Jul 27 21:04:48.976030 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:04:48.975987    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rxl8w"
Jul 27 21:04:48.976030 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:04:48.976020    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pr4c2"
Jul 27 21:04:48.976638 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:04:48.976020    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hmjv6"
Jul 27 21:04:48.976638 ip-10-0-131-133 kubenswrapper[2559]: E0727 21:04:48.976106    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rxl8w" podUID="00ef4d6d-0b8e-4ece-ba97-36a9e236ec53"
Jul 27 21:04:48.976638 ip-10-0-131-133 kubenswrapper[2559]: E0727 21:04:48.976208    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pr4c2" podUID="5851de39-848d-4a52-bc17-fefe1bf5407a"
Jul 27 21:04:48.976638 ip-10-0-131-133 kubenswrapper[2559]: E0727 21:04:48.976302    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hmjv6" podUID="95b2123e-132a-42a8-8edb-33ce19c2df51"
Jul 27 21:04:49.585518 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:04:49.585469    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/00ef4d6d-0b8e-4ece-ba97-36a9e236ec53-original-pull-secret\") pod \"global-pull-secret-syncer-rxl8w\" (UID: \"00ef4d6d-0b8e-4ece-ba97-36a9e236ec53\") " pod="kube-system/global-pull-secret-syncer-rxl8w"
Jul 27 21:04:49.585699 ip-10-0-131-133 kubenswrapper[2559]: E0727 21:04:49.585644    2559 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 27 21:04:49.585746 ip-10-0-131-133 kubenswrapper[2559]: E0727 21:04:49.585725    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/00ef4d6d-0b8e-4ece-ba97-36a9e236ec53-original-pull-secret podName:00ef4d6d-0b8e-4ece-ba97-36a9e236ec53 nodeName:}" failed. No retries permitted until 2026-07-27 21:05:05.585703508 +0000 UTC m=+34.164887495 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/00ef4d6d-0b8e-4ece-ba97-36a9e236ec53-original-pull-secret") pod "global-pull-secret-syncer-rxl8w" (UID: "00ef4d6d-0b8e-4ece-ba97-36a9e236ec53") : object "kube-system"/"original-pull-secret" not registered
Jul 27 21:04:49.686197 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:04:49.686149    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5851de39-848d-4a52-bc17-fefe1bf5407a-metrics-certs\") pod \"network-metrics-daemon-pr4c2\" (UID: \"5851de39-848d-4a52-bc17-fefe1bf5407a\") " pod="openshift-multus/network-metrics-daemon-pr4c2"
Jul 27 21:04:49.686364 ip-10-0-131-133 kubenswrapper[2559]: E0727 21:04:49.686316    2559 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 27 21:04:49.686440 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:04:49.686316    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mtwdg\" (UniqueName: \"kubernetes.io/projected/95b2123e-132a-42a8-8edb-33ce19c2df51-kube-api-access-mtwdg\") pod \"network-check-target-hmjv6\" (UID: \"95b2123e-132a-42a8-8edb-33ce19c2df51\") " pod="openshift-network-diagnostics/network-check-target-hmjv6"
Jul 27 21:04:49.686440 ip-10-0-131-133 kubenswrapper[2559]: E0727 21:04:49.686378    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5851de39-848d-4a52-bc17-fefe1bf5407a-metrics-certs podName:5851de39-848d-4a52-bc17-fefe1bf5407a nodeName:}" failed. No retries permitted until 2026-07-27 21:05:05.686362681 +0000 UTC m=+34.265546669 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/5851de39-848d-4a52-bc17-fefe1bf5407a-metrics-certs") pod "network-metrics-daemon-pr4c2" (UID: "5851de39-848d-4a52-bc17-fefe1bf5407a") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 27 21:04:49.686440 ip-10-0-131-133 kubenswrapper[2559]: E0727 21:04:49.686413    2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 27 21:04:49.686583 ip-10-0-131-133 kubenswrapper[2559]: E0727 21:04:49.686453    2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 27 21:04:49.686583 ip-10-0-131-133 kubenswrapper[2559]: E0727 21:04:49.686467    2559 projected.go:196] Error preparing data for projected volume kube-api-access-mtwdg for pod openshift-network-diagnostics/network-check-target-hmjv6: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 27 21:04:49.686583 ip-10-0-131-133 kubenswrapper[2559]: E0727 21:04:49.686522    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/95b2123e-132a-42a8-8edb-33ce19c2df51-kube-api-access-mtwdg podName:95b2123e-132a-42a8-8edb-33ce19c2df51 nodeName:}" failed. No retries permitted until 2026-07-27 21:05:05.686507547 +0000 UTC m=+34.265691549 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-mtwdg" (UniqueName: "kubernetes.io/projected/95b2123e-132a-42a8-8edb-33ce19c2df51-kube-api-access-mtwdg") pod "network-check-target-hmjv6" (UID: "95b2123e-132a-42a8-8edb-33ce19c2df51") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 27 21:04:50.976077 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:04:50.976035    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rxl8w"
Jul 27 21:04:50.976544 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:04:50.976092    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pr4c2"
Jul 27 21:04:50.976544 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:04:50.976053    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hmjv6"
Jul 27 21:04:50.976544 ip-10-0-131-133 kubenswrapper[2559]: E0727 21:04:50.976174    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rxl8w" podUID="00ef4d6d-0b8e-4ece-ba97-36a9e236ec53"
Jul 27 21:04:50.976544 ip-10-0-131-133 kubenswrapper[2559]: E0727 21:04:50.976287    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hmjv6" podUID="95b2123e-132a-42a8-8edb-33ce19c2df51"
Jul 27 21:04:50.976544 ip-10-0-131-133 kubenswrapper[2559]: E0727 21:04:50.976373    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pr4c2" podUID="5851de39-848d-4a52-bc17-fefe1bf5407a"
Jul 27 21:04:52.976588 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:04:52.976554    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rxl8w"
Jul 27 21:04:52.977050 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:04:52.976558    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hmjv6"
Jul 27 21:04:52.977050 ip-10-0-131-133 kubenswrapper[2559]: E0727 21:04:52.976667    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rxl8w" podUID="00ef4d6d-0b8e-4ece-ba97-36a9e236ec53"
Jul 27 21:04:52.977050 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:04:52.976692    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pr4c2"
Jul 27 21:04:52.977050 ip-10-0-131-133 kubenswrapper[2559]: E0727 21:04:52.976833    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hmjv6" podUID="95b2123e-132a-42a8-8edb-33ce19c2df51"
Jul 27 21:04:52.977050 ip-10-0-131-133 kubenswrapper[2559]: E0727 21:04:52.976957    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pr4c2" podUID="5851de39-848d-4a52-bc17-fefe1bf5407a"
Jul 27 21:04:54.181146 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:04:54.180969    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gs462" event={"ID":"3d80fd9e-83d7-4c96-b196-1c9afac0bad4","Type":"ContainerStarted","Data":"771ea64c2bc255e311fc6739e648986d080219725b8cd95af5928b5f3e6f787a"}
Jul 27 21:04:54.183259 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:04:54.183208    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-qs89h" event={"ID":"7f66b2c1-260e-4a55-902b-c4a060b723f9","Type":"ContainerStarted","Data":"b01e950dbc37db7f130bf2f4f464eaf3ecb1e01195960ac47aaa0b4ccbb47bf5"}
Jul 27 21:04:54.183368 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:04:54.183308    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-qs89h" event={"ID":"7f66b2c1-260e-4a55-902b-c4a060b723f9","Type":"ContainerStarted","Data":"7bd49e29756927bdbe6dc1d204c5824cf5de343631536bedebd047f516bcaa5b"}
Jul 27 21:04:54.183368 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:04:54.183339    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-qs89h" event={"ID":"7f66b2c1-260e-4a55-902b-c4a060b723f9","Type":"ContainerStarted","Data":"c3bccefb3728d9830b744da157996dc75003bbe931cb620e2dc54ac223639d66"}
Jul 27 21:04:54.183467 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:04:54.183375    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-qs89h" event={"ID":"7f66b2c1-260e-4a55-902b-c4a060b723f9","Type":"ContainerStarted","Data":"c9c7bad44fb7e40b1652909d20e5e7d69c066bc5828ccf3699c1e1aa04b249cb"}
Jul 27 21:04:54.184487 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:04:54.184452    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-dchr6" event={"ID":"6b7af06d-65e7-45ca-80ef-e88bcee2282a","Type":"ContainerStarted","Data":"c8f1f67b95307ca56135119ad774341563dfc5cfdc3b03f84ef6f559acd7547d"}
Jul 27 21:04:54.189721 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:04:54.189693    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-tfvtq" event={"ID":"4ff24566-f9aa-4eab-8795-a2808831cec1","Type":"ContainerStarted","Data":"d4cc44f6df10f7c3478e384b914a1f774a409c342466bd98b9eb547fcbff4df2"}
Jul 27 21:04:54.191012 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:04:54.190988    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-tfsvr" event={"ID":"e84e95da-4c32-4514-9c49-aaa0f96abea2","Type":"ContainerStarted","Data":"dd4bd4fa1b621ed0de8ab75db7fa7c6debb997e02c70392d40195fa16dba2f76"}
Jul 27 21:04:54.192400 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:04:54.192377    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-fvbmc" event={"ID":"84a7d738-f486-43b0-ac43-3401891da8d1","Type":"ContainerStarted","Data":"531a87617ae4322d54a4780e216df6555b343ecabc6f995c67ab749a5ecde550"}
Jul 27 21:04:54.193721 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:04:54.193691    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-qxzfs" event={"ID":"3e4c5c3f-db85-4e88-a1d5-48e602b0018a","Type":"ContainerStarted","Data":"2bd44c3b3bd6f2dc2b1602309a2edbae3391d1a31fb192de988f6ee25ae3bdb5"}
Jul 27 21:04:54.195060 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:04:54.195037    2559 generic.go:356] "Generic (PLEG): container finished" podID="4275e4fb-a8e5-43a5-8b8c-ed4cc2094674" containerID="07a768c4cb7d68217ea7ba79475a020c38627212a9e0baab79b0a13a2e585e55" exitCode=0
Jul 27 21:04:54.195138 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:04:54.195068    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-2pbtk" event={"ID":"4275e4fb-a8e5-43a5-8b8c-ed4cc2094674","Type":"ContainerDied","Data":"07a768c4cb7d68217ea7ba79475a020c38627212a9e0baab79b0a13a2e585e55"}
Jul 27 21:04:54.203268 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:04:54.203230    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-133.ec2.internal" podStartSLOduration=21.203183933 podStartE2EDuration="21.203183933s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-27 21:04:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 21:04:37.172024748 +0000 UTC m=+5.751208759" watchObservedRunningTime="2026-07-27 21:04:54.203183933 +0000 UTC m=+22.782367941"
Jul 27 21:04:54.203374 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:04:54.203358    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-dchr6" podStartSLOduration=3.448598998 podStartE2EDuration="22.203352866s" totalImagesPullingTime="18.754753868s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-27 21:04:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 21:04:54.202722575 +0000 UTC m=+22.781906581" watchObservedRunningTime="2026-07-27 21:04:54.203352866 +0000 UTC m=+22.782536873"
Jul 27 21:04:54.218944 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:04:54.218898    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-tfsvr" podStartSLOduration=3.495226691 podStartE2EDuration="22.218882711s" totalImagesPullingTime="18.72365602s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-27 21:04:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 21:04:54.218721507 +0000 UTC m=+22.797905510" watchObservedRunningTime="2026-07-27 21:04:54.218882711 +0000 UTC m=+22.798066718"
Jul 27 21:04:54.233236 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:04:54.233172    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-fvbmc" podStartSLOduration=3.529033432 podStartE2EDuration="22.233157779s" totalImagesPullingTime="18.704124347s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-27 21:04:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 21:04:54.233056196 +0000 UTC m=+22.812240204" watchObservedRunningTime="2026-07-27 21:04:54.233157779 +0000 UTC m=+22.812341783"
Jul 27 21:04:54.321082 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:04:54.321037    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-tfvtq" podStartSLOduration=7.556508689 podStartE2EDuration="22.321022113s" totalImagesPullingTime="14.764513424s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 21:04:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 21:04:54.297677627 +0000 UTC m=+22.876861633" watchObservedRunningTime="2026-07-27 21:04:54.321022113 +0000 UTC m=+22.900206119"
Jul 27 21:04:54.321251 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:04:54.321126    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-qxzfs" podStartSLOduration=3.627163431 podStartE2EDuration="22.321120257s" totalImagesPullingTime="18.693956826s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-27 21:04:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 21:04:54.320692918 +0000 UTC m=+22.899876922" watchObservedRunningTime="2026-07-27 21:04:54.321120257 +0000 UTC m=+22.900304266"
Jul 27 21:04:54.521057 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:04:54.520970    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-tfvtq"
Jul 27 21:04:54.976095 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:04:54.976057    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rxl8w"
Jul 27 21:04:54.976095 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:04:54.976094    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pr4c2"
Jul 27 21:04:54.976357 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:04:54.976094    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hmjv6"
Jul 27 21:04:54.976357 ip-10-0-131-133 kubenswrapper[2559]: E0727 21:04:54.976194    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rxl8w" podUID="00ef4d6d-0b8e-4ece-ba97-36a9e236ec53"
Jul 27 21:04:54.976357 ip-10-0-131-133 kubenswrapper[2559]: E0727 21:04:54.976336    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pr4c2" podUID="5851de39-848d-4a52-bc17-fefe1bf5407a"
Jul 27 21:04:54.976536 ip-10-0-131-133 kubenswrapper[2559]: E0727 21:04:54.976407    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hmjv6" podUID="95b2123e-132a-42a8-8edb-33ce19c2df51"
Jul 27 21:04:55.201296 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:04:55.201265    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-qs89h" event={"ID":"7f66b2c1-260e-4a55-902b-c4a060b723f9","Type":"ContainerStarted","Data":"4811fbd07ed899ec7052f39c4fa756947455b4a3a02af1f34e831d63eb87ce04"}
Jul 27 21:04:55.201729 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:04:55.201347    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-qs89h" event={"ID":"7f66b2c1-260e-4a55-902b-c4a060b723f9","Type":"ContainerStarted","Data":"87a88b2be05b0ae6d7306188c17eb1a98d82b789bf1b30a66a45b2b99b662895"}
Jul 27 21:04:55.202984 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:04:55.202958    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-cd6w9" event={"ID":"c2d67a6e-7963-4c73-9983-40b775a8134a","Type":"ContainerStarted","Data":"ebe9441bd0108cca1408baafbf7e437b1e70461cb0730170476c66a095170783"}
Jul 27 21:04:55.219259 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:04:55.219199    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-cd6w9" podStartSLOduration=4.517450285 podStartE2EDuration="23.21917879s" totalImagesPullingTime="18.701728505s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-27 21:04:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 21:04:55.218993777 +0000 UTC m=+23.798177785" watchObservedRunningTime="2026-07-27 21:04:55.21917879 +0000 UTC m=+23.798362797"
Jul 27 21:04:55.227887 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:04:55.227817    2559 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Jul 27 21:04:55.917263 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:04:55.917136    2559 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-27T21:04:55.22784004Z","UUID":"e2869f69-7b20-4a8d-9b0d-7c8042f883bb","Handler":null,"Name":"","Endpoint":""}
Jul 27 21:04:55.918858 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:04:55.918832    2559 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Jul 27 21:04:55.918998 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:04:55.918880    2559 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Jul 27 21:04:56.207539 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:04:56.207444    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gs462" event={"ID":"3d80fd9e-83d7-4c96-b196-1c9afac0bad4","Type":"ContainerStarted","Data":"8ad4fb9d4e2d916237c2e9fe9ea391e46eb1209a777ea7f68ebd42cbf24a43f4"}
Jul 27 21:04:56.575764 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:04:56.575739    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-tfvtq"
Jul 27 21:04:56.576536 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:04:56.576510    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-tfvtq"
Jul 27 21:04:56.976574 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:04:56.976370    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rxl8w"
Jul 27 21:04:56.976755 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:04:56.976376    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hmjv6"
Jul 27 21:04:56.976755 ip-10-0-131-133 kubenswrapper[2559]: E0727 21:04:56.976674    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rxl8w" podUID="00ef4d6d-0b8e-4ece-ba97-36a9e236ec53"
Jul 27 21:04:56.976863 ip-10-0-131-133 kubenswrapper[2559]: E0727 21:04:56.976765    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hmjv6" podUID="95b2123e-132a-42a8-8edb-33ce19c2df51"
Jul 27 21:04:56.976863 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:04:56.976427    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pr4c2"
Jul 27 21:04:56.976959 ip-10-0-131-133 kubenswrapper[2559]: E0727 21:04:56.976883    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pr4c2" podUID="5851de39-848d-4a52-bc17-fefe1bf5407a"
Jul 27 21:04:57.212536 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:04:57.212454    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gs462" event={"ID":"3d80fd9e-83d7-4c96-b196-1c9afac0bad4","Type":"ContainerStarted","Data":"92c881a84909375f1c89f6cece3343f57608e71e8a46b7ff30685ac5f9466ca9"}
Jul 27 21:04:57.216179 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:04:57.216149    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-qs89h" event={"ID":"7f66b2c1-260e-4a55-902b-c4a060b723f9","Type":"ContainerStarted","Data":"b250ed4ac47d648cfb9ff3fa715f41a878f1e7ba5972fcf9eead6a95ac5c8d73"}
Jul 27 21:04:57.216941 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:04:57.216918    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-tfvtq"
Jul 27 21:04:57.237651 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:04:57.237547    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gs462" podStartSLOduration=4.025260654 podStartE2EDuration="25.237526328s" totalImagesPullingTime="21.212265674s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-27 21:04:32 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 21:04:57.236695797 +0000 UTC m=+25.815879805" watchObservedRunningTime="2026-07-27 21:04:57.237526328 +0000 UTC m=+25.816710342"
Jul 27 21:04:58.975994 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:04:58.975801    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rxl8w"
Jul 27 21:04:58.976894 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:04:58.975859    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hmjv6"
Jul 27 21:04:58.976894 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:04:58.975923    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pr4c2"
Jul 27 21:04:58.976894 ip-10-0-131-133 kubenswrapper[2559]: E0727 21:04:58.976180    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hmjv6" podUID="95b2123e-132a-42a8-8edb-33ce19c2df51"
Jul 27 21:04:58.976894 ip-10-0-131-133 kubenswrapper[2559]: E0727 21:04:58.976132    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rxl8w" podUID="00ef4d6d-0b8e-4ece-ba97-36a9e236ec53"
Jul 27 21:04:58.976894 ip-10-0-131-133 kubenswrapper[2559]: E0727 21:04:58.976249    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pr4c2" podUID="5851de39-848d-4a52-bc17-fefe1bf5407a"
Jul 27 21:04:59.222612 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:04:59.222573    2559 generic.go:356] "Generic (PLEG): container finished" podID="4275e4fb-a8e5-43a5-8b8c-ed4cc2094674" containerID="16819325e8e2d69bc3ecc8e5084517bbe40ac0af8273d6bd57ba166a74b66f6c" exitCode=0
Jul 27 21:04:59.222797 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:04:59.222656    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-2pbtk" event={"ID":"4275e4fb-a8e5-43a5-8b8c-ed4cc2094674","Type":"ContainerDied","Data":"16819325e8e2d69bc3ecc8e5084517bbe40ac0af8273d6bd57ba166a74b66f6c"}
Jul 27 21:04:59.226433 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:04:59.226395    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-qs89h" event={"ID":"7f66b2c1-260e-4a55-902b-c4a060b723f9","Type":"ContainerStarted","Data":"fac53ed3df764593f72869bd5e0984731c860dad930ba68af4c7d57af529cb88"}
Jul 27 21:04:59.226752 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:04:59.226734    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-qs89h"
Jul 27 21:04:59.242615 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:04:59.242586    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-qs89h"
Jul 27 21:05:00.232268 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:05:00.232232    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-qs89h"
Jul 27 21:05:00.232775 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:05:00.232337    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-qs89h"
Jul 27 21:05:00.247749 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:05:00.247722    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-qs89h"
Jul 27 21:05:00.281267 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:05:00.281193    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-qs89h" podStartSLOduration=9.534456484 podStartE2EDuration="28.28117857s" totalImagesPullingTime="18.746722086s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-27 21:04:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 21:04:59.315814659 +0000 UTC m=+27.894998666" watchObservedRunningTime="2026-07-27 21:05:00.28117857 +0000 UTC m=+28.860362577"
Jul 27 21:05:00.976074 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:05:00.976037    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pr4c2"
Jul 27 21:05:00.976256 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:05:00.976079    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rxl8w"
Jul 27 21:05:00.976256 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:05:00.976079    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hmjv6"
Jul 27 21:05:00.976256 ip-10-0-131-133 kubenswrapper[2559]: E0727 21:05:00.976157    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pr4c2" podUID="5851de39-848d-4a52-bc17-fefe1bf5407a"
Jul 27 21:05:00.976362 ip-10-0-131-133 kubenswrapper[2559]: E0727 21:05:00.976273    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rxl8w" podUID="00ef4d6d-0b8e-4ece-ba97-36a9e236ec53"
Jul 27 21:05:00.976362 ip-10-0-131-133 kubenswrapper[2559]: E0727 21:05:00.976346    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hmjv6" podUID="95b2123e-132a-42a8-8edb-33ce19c2df51"
Jul 27 21:05:01.037509 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:05:01.037471    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-hmjv6"]
Jul 27 21:05:01.039300 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:05:01.039266    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-rxl8w"]
Jul 27 21:05:01.039821 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:05:01.039790    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-pr4c2"]
Jul 27 21:05:01.236074 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:05:01.235990    2559 generic.go:356] "Generic (PLEG): container finished" podID="4275e4fb-a8e5-43a5-8b8c-ed4cc2094674" containerID="af9a5b0c59ccb55ce0285d8d80ae50a9a4a79472c8c2934d4202189c47ef5d52" exitCode=0
Jul 27 21:05:01.236509 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:05:01.236076    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-2pbtk" event={"ID":"4275e4fb-a8e5-43a5-8b8c-ed4cc2094674","Type":"ContainerDied","Data":"af9a5b0c59ccb55ce0285d8d80ae50a9a4a79472c8c2934d4202189c47ef5d52"}
Jul 27 21:05:01.236509 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:05:01.236119    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hmjv6"
Jul 27 21:05:01.236509 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:05:01.236229    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rxl8w"
Jul 27 21:05:01.236509 ip-10-0-131-133 kubenswrapper[2559]: E0727 21:05:01.236332    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hmjv6" podUID="95b2123e-132a-42a8-8edb-33ce19c2df51"
Jul 27 21:05:01.236703 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:05:01.236670    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pr4c2"
Jul 27 21:05:01.236821 ip-10-0-131-133 kubenswrapper[2559]: E0727 21:05:01.236747    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rxl8w" podUID="00ef4d6d-0b8e-4ece-ba97-36a9e236ec53"
Jul 27 21:05:01.236874 ip-10-0-131-133 kubenswrapper[2559]: E0727 21:05:01.236853    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pr4c2" podUID="5851de39-848d-4a52-bc17-fefe1bf5407a"
Jul 27 21:05:02.976638 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:05:02.976423    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pr4c2"
Jul 27 21:05:02.977008 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:05:02.976446    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hmjv6"
Jul 27 21:05:02.977008 ip-10-0-131-133 kubenswrapper[2559]: E0727 21:05:02.976744    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pr4c2" podUID="5851de39-848d-4a52-bc17-fefe1bf5407a"
Jul 27 21:05:02.977008 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:05:02.976480    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rxl8w"
Jul 27 21:05:02.977008 ip-10-0-131-133 kubenswrapper[2559]: E0727 21:05:02.976787    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hmjv6" podUID="95b2123e-132a-42a8-8edb-33ce19c2df51"
Jul 27 21:05:02.977008 ip-10-0-131-133 kubenswrapper[2559]: E0727 21:05:02.976880    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rxl8w" podUID="00ef4d6d-0b8e-4ece-ba97-36a9e236ec53"
Jul 27 21:05:03.242568 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:05:03.242536    2559 generic.go:356] "Generic (PLEG): container finished" podID="4275e4fb-a8e5-43a5-8b8c-ed4cc2094674" containerID="d45e11aae197680ce80bb9c133cebefde3258f4d2ff3c10e6337022ef52d7e94" exitCode=0
Jul 27 21:05:03.242778 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:05:03.242597    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-2pbtk" event={"ID":"4275e4fb-a8e5-43a5-8b8c-ed4cc2094674","Type":"ContainerDied","Data":"d45e11aae197680ce80bb9c133cebefde3258f4d2ff3c10e6337022ef52d7e94"}
Jul 27 21:05:04.976524 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:05:04.976481    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rxl8w"
Jul 27 21:05:04.976524 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:05:04.976520    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hmjv6"
Jul 27 21:05:04.977178 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:05:04.976615    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pr4c2"
Jul 27 21:05:04.977178 ip-10-0-131-133 kubenswrapper[2559]: E0727 21:05:04.976629    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rxl8w" podUID="00ef4d6d-0b8e-4ece-ba97-36a9e236ec53"
Jul 27 21:05:04.977178 ip-10-0-131-133 kubenswrapper[2559]: E0727 21:05:04.976711    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hmjv6" podUID="95b2123e-132a-42a8-8edb-33ce19c2df51"
Jul 27 21:05:04.977178 ip-10-0-131-133 kubenswrapper[2559]: E0727 21:05:04.976765    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pr4c2" podUID="5851de39-848d-4a52-bc17-fefe1bf5407a"
Jul 27 21:05:05.602034 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:05:05.601999    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/00ef4d6d-0b8e-4ece-ba97-36a9e236ec53-original-pull-secret\") pod \"global-pull-secret-syncer-rxl8w\" (UID: \"00ef4d6d-0b8e-4ece-ba97-36a9e236ec53\") " pod="kube-system/global-pull-secret-syncer-rxl8w"
Jul 27 21:05:05.602227 ip-10-0-131-133 kubenswrapper[2559]: E0727 21:05:05.602142    2559 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 27 21:05:05.602227 ip-10-0-131-133 kubenswrapper[2559]: E0727 21:05:05.602207    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/00ef4d6d-0b8e-4ece-ba97-36a9e236ec53-original-pull-secret podName:00ef4d6d-0b8e-4ece-ba97-36a9e236ec53 nodeName:}" failed. No retries permitted until 2026-07-27 21:05:37.602186428 +0000 UTC m=+66.181370429 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/00ef4d6d-0b8e-4ece-ba97-36a9e236ec53-original-pull-secret") pod "global-pull-secret-syncer-rxl8w" (UID: "00ef4d6d-0b8e-4ece-ba97-36a9e236ec53") : object "kube-system"/"original-pull-secret" not registered
Jul 27 21:05:05.703022 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:05:05.702985    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mtwdg\" (UniqueName: \"kubernetes.io/projected/95b2123e-132a-42a8-8edb-33ce19c2df51-kube-api-access-mtwdg\") pod \"network-check-target-hmjv6\" (UID: \"95b2123e-132a-42a8-8edb-33ce19c2df51\") " pod="openshift-network-diagnostics/network-check-target-hmjv6"
Jul 27 21:05:05.703191 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:05:05.703036    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5851de39-848d-4a52-bc17-fefe1bf5407a-metrics-certs\") pod \"network-metrics-daemon-pr4c2\" (UID: \"5851de39-848d-4a52-bc17-fefe1bf5407a\") " pod="openshift-multus/network-metrics-daemon-pr4c2"
Jul 27 21:05:05.703191 ip-10-0-131-133 kubenswrapper[2559]: E0727 21:05:05.703128    2559 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 27 21:05:05.703191 ip-10-0-131-133 kubenswrapper[2559]: E0727 21:05:05.703154    2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 27 21:05:05.703191 ip-10-0-131-133 kubenswrapper[2559]: E0727 21:05:05.703176    2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 27 21:05:05.703191 ip-10-0-131-133 kubenswrapper[2559]: E0727 21:05:05.703181    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5851de39-848d-4a52-bc17-fefe1bf5407a-metrics-certs podName:5851de39-848d-4a52-bc17-fefe1bf5407a nodeName:}" failed. No retries permitted until 2026-07-27 21:05:37.703166484 +0000 UTC m=+66.282350468 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/5851de39-848d-4a52-bc17-fefe1bf5407a-metrics-certs") pod "network-metrics-daemon-pr4c2" (UID: "5851de39-848d-4a52-bc17-fefe1bf5407a") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 27 21:05:05.703191 ip-10-0-131-133 kubenswrapper[2559]: E0727 21:05:05.703189    2559 projected.go:196] Error preparing data for projected volume kube-api-access-mtwdg for pod openshift-network-diagnostics/network-check-target-hmjv6: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 27 21:05:05.703420 ip-10-0-131-133 kubenswrapper[2559]: E0727 21:05:05.703252    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/95b2123e-132a-42a8-8edb-33ce19c2df51-kube-api-access-mtwdg podName:95b2123e-132a-42a8-8edb-33ce19c2df51 nodeName:}" failed. No retries permitted until 2026-07-27 21:05:37.703240634 +0000 UTC m=+66.282424619 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-mtwdg" (UniqueName: "kubernetes.io/projected/95b2123e-132a-42a8-8edb-33ce19c2df51-kube-api-access-mtwdg") pod "network-check-target-hmjv6" (UID: "95b2123e-132a-42a8-8edb-33ce19c2df51") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 27 21:05:06.976140 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:05:06.976104    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rxl8w"
Jul 27 21:05:06.976140 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:05:06.976134    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pr4c2"
Jul 27 21:05:06.976140 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:05:06.976139    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hmjv6"
Jul 27 21:05:06.976892 ip-10-0-131-133 kubenswrapper[2559]: E0727 21:05:06.976268    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rxl8w" podUID="00ef4d6d-0b8e-4ece-ba97-36a9e236ec53"
Jul 27 21:05:06.976892 ip-10-0-131-133 kubenswrapper[2559]: E0727 21:05:06.976288    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hmjv6" podUID="95b2123e-132a-42a8-8edb-33ce19c2df51"
Jul 27 21:05:06.976892 ip-10-0-131-133 kubenswrapper[2559]: E0727 21:05:06.976377    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pr4c2" podUID="5851de39-848d-4a52-bc17-fefe1bf5407a"
Jul 27 21:05:07.703946 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:05:07.703917    2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-133.ec2.internal" event="NodeReady"
Jul 27 21:05:07.704118 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:05:07.704046    2559 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Jul 27 21:05:07.748675 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:05:07.748642    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-5b565b454b-2kr9d"]
Jul 27 21:05:07.750404 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:05:07.750389    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5b565b454b-2kr9d"
Jul 27 21:05:07.753664 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:05:07.753641    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Jul 27 21:05:07.753775 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:05:07.753641    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-kdvbm\""
Jul 27 21:05:07.753775 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:05:07.753763    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Jul 27 21:05:07.755731 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:05:07.755710    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Jul 27 21:05:07.761981 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:05:07.761953    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Jul 27 21:05:07.773065 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:05:07.773041    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-9kfdx"]
Jul 27 21:05:07.774967 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:05:07.774952    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-9kfdx"
Jul 27 21:05:07.776554 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:05:07.776531    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-78ljg"]
Jul 27 21:05:07.778439 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:05:07.778420    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Jul 27 21:05:07.778541 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:05:07.778443    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Jul 27 21:05:07.778541 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:05:07.778523    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-6vzp2\""
Jul 27 21:05:07.778886 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:05:07.778862    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-5b565b454b-2kr9d"]
Jul 27 21:05:07.778968 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:05:07.778948    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-78ljg"
Jul 27 21:05:07.783793 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:05:07.783770    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Jul 27 21:05:07.783885 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:05:07.783814    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-sfq8h\""
Jul 27 21:05:07.783885 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:05:07.783843    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Jul 27 21:05:07.784390 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:05:07.784371    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Jul 27 21:05:07.792265 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:05:07.792243    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-78ljg"]
Jul 27 21:05:07.797185 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:05:07.797165    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-9kfdx"]
Jul 27 21:05:07.918555 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:05:07.918511    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ac10d2c6-12bd-4f89-8d1e-c931206a2e92-metrics-tls\") pod \"dns-default-9kfdx\" (UID: \"ac10d2c6-12bd-4f89-8d1e-c931206a2e92\") " pod="openshift-dns/dns-default-9kfdx"
Jul 27 21:05:07.918555 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:05:07.918552    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/cf59a2e6-f3c6-4549-9158-f047bfb7e764-cert\") pod \"ingress-canary-78ljg\" (UID: \"cf59a2e6-f3c6-4549-9158-f047bfb7e764\") " pod="openshift-ingress-canary/ingress-canary-78ljg"
Jul 27 21:05:07.918788 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:05:07.918576    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/097262d9-cf78-4d89-98cc-cf92d23a4f04-trusted-ca\") pod \"image-registry-5b565b454b-2kr9d\" (UID: \"097262d9-cf78-4d89-98cc-cf92d23a4f04\") " pod="openshift-image-registry/image-registry-5b565b454b-2kr9d"
Jul 27 21:05:07.918788 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:05:07.918601    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sf2pq\" (UniqueName: \"kubernetes.io/projected/cf59a2e6-f3c6-4549-9158-f047bfb7e764-kube-api-access-sf2pq\") pod \"ingress-canary-78ljg\" (UID: \"cf59a2e6-f3c6-4549-9158-f047bfb7e764\") " pod="openshift-ingress-canary/ingress-canary-78ljg"
Jul 27 21:05:07.918788 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:05:07.918618    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/097262d9-cf78-4d89-98cc-cf92d23a4f04-installation-pull-secrets\") pod \"image-registry-5b565b454b-2kr9d\" (UID: \"097262d9-cf78-4d89-98cc-cf92d23a4f04\") " pod="openshift-image-registry/image-registry-5b565b454b-2kr9d"
Jul 27 21:05:07.918788 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:05:07.918707    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/097262d9-cf78-4d89-98cc-cf92d23a4f04-bound-sa-token\") pod \"image-registry-5b565b454b-2kr9d\" (UID: \"097262d9-cf78-4d89-98cc-cf92d23a4f04\") " pod="openshift-image-registry/image-registry-5b565b454b-2kr9d"
Jul 27 21:05:07.918788 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:05:07.918730    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/097262d9-cf78-4d89-98cc-cf92d23a4f04-registry-tls\") pod \"image-registry-5b565b454b-2kr9d\" (UID: \"097262d9-cf78-4d89-98cc-cf92d23a4f04\") " pod="openshift-image-registry/image-registry-5b565b454b-2kr9d"
Jul 27 21:05:07.918788 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:05:07.918769    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/097262d9-cf78-4d89-98cc-cf92d23a4f04-registry-certificates\") pod \"image-registry-5b565b454b-2kr9d\" (UID: \"097262d9-cf78-4d89-98cc-cf92d23a4f04\") " pod="openshift-image-registry/image-registry-5b565b454b-2kr9d"
Jul 27 21:05:07.919011 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:05:07.918808    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/097262d9-cf78-4d89-98cc-cf92d23a4f04-ca-trust-extracted\") pod \"image-registry-5b565b454b-2kr9d\" (UID: \"097262d9-cf78-4d89-98cc-cf92d23a4f04\") " pod="openshift-image-registry/image-registry-5b565b454b-2kr9d"
Jul 27 21:05:07.919011 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:05:07.918856    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7ccrk\" (UniqueName: \"kubernetes.io/projected/ac10d2c6-12bd-4f89-8d1e-c931206a2e92-kube-api-access-7ccrk\") pod \"dns-default-9kfdx\" (UID: \"ac10d2c6-12bd-4f89-8d1e-c931206a2e92\") " pod="openshift-dns/dns-default-9kfdx"
Jul 27 21:05:07.919011 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:05:07.918880    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cjjp7\" (UniqueName: \"kubernetes.io/projected/097262d9-cf78-4d89-98cc-cf92d23a4f04-kube-api-access-cjjp7\") pod \"image-registry-5b565b454b-2kr9d\" (UID: \"097262d9-cf78-4d89-98cc-cf92d23a4f04\") " pod="openshift-image-registry/image-registry-5b565b454b-2kr9d"
Jul 27 21:05:07.919011 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:05:07.918897    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/097262d9-cf78-4d89-98cc-cf92d23a4f04-image-registry-private-configuration\") pod \"image-registry-5b565b454b-2kr9d\" (UID: \"097262d9-cf78-4d89-98cc-cf92d23a4f04\") " pod="openshift-image-registry/image-registry-5b565b454b-2kr9d"
Jul 27 21:05:07.919011 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:05:07.918928    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ac10d2c6-12bd-4f89-8d1e-c931206a2e92-tmp-dir\") pod \"dns-default-9kfdx\" (UID: \"ac10d2c6-12bd-4f89-8d1e-c931206a2e92\") " pod="openshift-dns/dns-default-9kfdx"
Jul 27 21:05:07.919011 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:05:07.918947    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/ac10d2c6-12bd-4f89-8d1e-c931206a2e92-config-volume\") pod \"dns-default-9kfdx\" (UID: \"ac10d2c6-12bd-4f89-8d1e-c931206a2e92\") " pod="openshift-dns/dns-default-9kfdx"
Jul 27 21:05:08.019527 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:05:08.019427    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/097262d9-cf78-4d89-98cc-cf92d23a4f04-bound-sa-token\") pod \"image-registry-5b565b454b-2kr9d\" (UID: \"097262d9-cf78-4d89-98cc-cf92d23a4f04\") " pod="openshift-image-registry/image-registry-5b565b454b-2kr9d"
Jul 27 21:05:08.019527 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:05:08.019476    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/097262d9-cf78-4d89-98cc-cf92d23a4f04-registry-tls\") pod \"image-registry-5b565b454b-2kr9d\" (UID: \"097262d9-cf78-4d89-98cc-cf92d23a4f04\") " pod="openshift-image-registry/image-registry-5b565b454b-2kr9d"
Jul 27 21:05:08.019527 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:05:08.019495    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/097262d9-cf78-4d89-98cc-cf92d23a4f04-registry-certificates\") pod \"image-registry-5b565b454b-2kr9d\" (UID: \"097262d9-cf78-4d89-98cc-cf92d23a4f04\") " pod="openshift-image-registry/image-registry-5b565b454b-2kr9d"
Jul 27 21:05:08.020184 ip-10-0-131-133 kubenswrapper[2559]: E0727 21:05:08.019610    2559 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 27 21:05:08.020184 ip-10-0-131-133 kubenswrapper[2559]: E0727 21:05:08.019629    2559 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5b565b454b-2kr9d: secret "image-registry-tls" not found
Jul 27 21:05:08.020184 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:05:08.019673    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/097262d9-cf78-4d89-98cc-cf92d23a4f04-ca-trust-extracted\") pod \"image-registry-5b565b454b-2kr9d\" (UID: \"097262d9-cf78-4d89-98cc-cf92d23a4f04\") " pod="openshift-image-registry/image-registry-5b565b454b-2kr9d"
Jul 27 21:05:08.020184 ip-10-0-131-133 kubenswrapper[2559]: E0727 21:05:08.019692    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/097262d9-cf78-4d89-98cc-cf92d23a4f04-registry-tls podName:097262d9-cf78-4d89-98cc-cf92d23a4f04 nodeName:}" failed. No retries permitted until 2026-07-27 21:05:08.519670274 +0000 UTC m=+37.098854260 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/097262d9-cf78-4d89-98cc-cf92d23a4f04-registry-tls") pod "image-registry-5b565b454b-2kr9d" (UID: "097262d9-cf78-4d89-98cc-cf92d23a4f04") : secret "image-registry-tls" not found
Jul 27 21:05:08.020184 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:05:08.019732    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7ccrk\" (UniqueName: \"kubernetes.io/projected/ac10d2c6-12bd-4f89-8d1e-c931206a2e92-kube-api-access-7ccrk\") pod \"dns-default-9kfdx\" (UID: \"ac10d2c6-12bd-4f89-8d1e-c931206a2e92\") " pod="openshift-dns/dns-default-9kfdx"
Jul 27 21:05:08.020184 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:05:08.019770    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cjjp7\" (UniqueName: \"kubernetes.io/projected/097262d9-cf78-4d89-98cc-cf92d23a4f04-kube-api-access-cjjp7\") pod \"image-registry-5b565b454b-2kr9d\" (UID: \"097262d9-cf78-4d89-98cc-cf92d23a4f04\") " pod="openshift-image-registry/image-registry-5b565b454b-2kr9d"
Jul 27 21:05:08.020184 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:05:08.019802    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/097262d9-cf78-4d89-98cc-cf92d23a4f04-image-registry-private-configuration\") pod \"image-registry-5b565b454b-2kr9d\" (UID: \"097262d9-cf78-4d89-98cc-cf92d23a4f04\") " pod="openshift-image-registry/image-registry-5b565b454b-2kr9d"
Jul 27 21:05:08.020184 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:05:08.019847    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ac10d2c6-12bd-4f89-8d1e-c931206a2e92-tmp-dir\") pod \"dns-default-9kfdx\" (UID: \"ac10d2c6-12bd-4f89-8d1e-c931206a2e92\") " pod="openshift-dns/dns-default-9kfdx"
Jul 27 21:05:08.020184 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:05:08.019901    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/ac10d2c6-12bd-4f89-8d1e-c931206a2e92-config-volume\") pod \"dns-default-9kfdx\" (UID: \"ac10d2c6-12bd-4f89-8d1e-c931206a2e92\") " pod="openshift-dns/dns-default-9kfdx"
Jul 27 21:05:08.020184 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:05:08.019943    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ac10d2c6-12bd-4f89-8d1e-c931206a2e92-metrics-tls\") pod \"dns-default-9kfdx\" (UID: \"ac10d2c6-12bd-4f89-8d1e-c931206a2e92\") " pod="openshift-dns/dns-default-9kfdx"
Jul 27 21:05:08.020184 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:05:08.019970    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/cf59a2e6-f3c6-4549-9158-f047bfb7e764-cert\") pod \"ingress-canary-78ljg\" (UID: \"cf59a2e6-f3c6-4549-9158-f047bfb7e764\") " pod="openshift-ingress-canary/ingress-canary-78ljg"
Jul 27 21:05:08.020184 ip-10-0-131-133 kubenswrapper[2559]: E0727 21:05:08.020041    2559 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 27 21:05:08.020184 ip-10-0-131-133 kubenswrapper[2559]: E0727 21:05:08.020109    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ac10d2c6-12bd-4f89-8d1e-c931206a2e92-metrics-tls podName:ac10d2c6-12bd-4f89-8d1e-c931206a2e92 nodeName:}" failed. No retries permitted until 2026-07-27 21:05:08.520090331 +0000 UTC m=+37.099274332 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/ac10d2c6-12bd-4f89-8d1e-c931206a2e92-metrics-tls") pod "dns-default-9kfdx" (UID: "ac10d2c6-12bd-4f89-8d1e-c931206a2e92") : secret "dns-default-metrics-tls" not found
Jul 27 21:05:08.020184 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:05:08.020148    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/097262d9-cf78-4d89-98cc-cf92d23a4f04-trusted-ca\") pod \"image-registry-5b565b454b-2kr9d\" (UID: \"097262d9-cf78-4d89-98cc-cf92d23a4f04\") " pod="openshift-image-registry/image-registry-5b565b454b-2kr9d"
Jul 27 21:05:08.020184 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:05:08.020157    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/097262d9-cf78-4d89-98cc-cf92d23a4f04-ca-trust-extracted\") pod \"image-registry-5b565b454b-2kr9d\" (UID: \"097262d9-cf78-4d89-98cc-cf92d23a4f04\") " pod="openshift-image-registry/image-registry-5b565b454b-2kr9d"
Jul 27 21:05:08.020184 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:05:08.020163    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ac10d2c6-12bd-4f89-8d1e-c931206a2e92-tmp-dir\") pod \"dns-default-9kfdx\" (UID: \"ac10d2c6-12bd-4f89-8d1e-c931206a2e92\") " pod="openshift-dns/dns-default-9kfdx"
Jul 27 21:05:08.020184 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:05:08.020179    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sf2pq\" (UniqueName: \"kubernetes.io/projected/cf59a2e6-f3c6-4549-9158-f047bfb7e764-kube-api-access-sf2pq\") pod \"ingress-canary-78ljg\" (UID: \"cf59a2e6-f3c6-4549-9158-f047bfb7e764\") " pod="openshift-ingress-canary/ingress-canary-78ljg"
Jul 27 21:05:08.020837 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:05:08.020322    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/097262d9-cf78-4d89-98cc-cf92d23a4f04-installation-pull-secrets\") pod \"image-registry-5b565b454b-2kr9d\" (UID: \"097262d9-cf78-4d89-98cc-cf92d23a4f04\") " pod="openshift-image-registry/image-registry-5b565b454b-2kr9d"
Jul 27 21:05:08.020837 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:05:08.020420    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/097262d9-cf78-4d89-98cc-cf92d23a4f04-registry-certificates\") pod \"image-registry-5b565b454b-2kr9d\" (UID: \"097262d9-cf78-4d89-98cc-cf92d23a4f04\") " pod="openshift-image-registry/image-registry-5b565b454b-2kr9d"
Jul 27 21:05:08.020837 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:05:08.020517    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/ac10d2c6-12bd-4f89-8d1e-c931206a2e92-config-volume\") pod \"dns-default-9kfdx\" (UID: \"ac10d2c6-12bd-4f89-8d1e-c931206a2e92\") " pod="openshift-dns/dns-default-9kfdx"
Jul 27 21:05:08.021601 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:05:08.021579    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/097262d9-cf78-4d89-98cc-cf92d23a4f04-trusted-ca\") pod \"image-registry-5b565b454b-2kr9d\" (UID: \"097262d9-cf78-4d89-98cc-cf92d23a4f04\") " pod="openshift-image-registry/image-registry-5b565b454b-2kr9d"
Jul 27 21:05:08.024036 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:05:08.024013    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/097262d9-cf78-4d89-98cc-cf92d23a4f04-installation-pull-secrets\") pod \"image-registry-5b565b454b-2kr9d\" (UID: \"097262d9-cf78-4d89-98cc-cf92d23a4f04\") " pod="openshift-image-registry/image-registry-5b565b454b-2kr9d"
Jul 27 21:05:08.024125 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:05:08.024050    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/097262d9-cf78-4d89-98cc-cf92d23a4f04-image-registry-private-configuration\") pod \"image-registry-5b565b454b-2kr9d\" (UID: \"097262d9-cf78-4d89-98cc-cf92d23a4f04\") " pod="openshift-image-registry/image-registry-5b565b454b-2kr9d"
Jul 27 21:05:08.024125 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:05:08.024082    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/cf59a2e6-f3c6-4549-9158-f047bfb7e764-cert\") pod \"ingress-canary-78ljg\" (UID: \"cf59a2e6-f3c6-4549-9158-f047bfb7e764\") " pod="openshift-ingress-canary/ingress-canary-78ljg"
Jul 27 21:05:08.034336 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:05:08.034303    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/097262d9-cf78-4d89-98cc-cf92d23a4f04-bound-sa-token\") pod \"image-registry-5b565b454b-2kr9d\" (UID: \"097262d9-cf78-4d89-98cc-cf92d23a4f04\") " pod="openshift-image-registry/image-registry-5b565b454b-2kr9d"
Jul 27 21:05:08.034546 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:05:08.034523    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cjjp7\" (UniqueName: \"kubernetes.io/projected/097262d9-cf78-4d89-98cc-cf92d23a4f04-kube-api-access-cjjp7\") pod \"image-registry-5b565b454b-2kr9d\" (UID: \"097262d9-cf78-4d89-98cc-cf92d23a4f04\") " pod="openshift-image-registry/image-registry-5b565b454b-2kr9d"
Jul 27 21:05:08.034644 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:05:08.034630    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sf2pq\" (UniqueName: \"kubernetes.io/projected/cf59a2e6-f3c6-4549-9158-f047bfb7e764-kube-api-access-sf2pq\") pod \"ingress-canary-78ljg\" (UID: \"cf59a2e6-f3c6-4549-9158-f047bfb7e764\") " pod="openshift-ingress-canary/ingress-canary-78ljg"
Jul 27 21:05:08.035124 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:05:08.035109    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7ccrk\" (UniqueName: \"kubernetes.io/projected/ac10d2c6-12bd-4f89-8d1e-c931206a2e92-kube-api-access-7ccrk\") pod \"dns-default-9kfdx\" (UID: \"ac10d2c6-12bd-4f89-8d1e-c931206a2e92\") " pod="openshift-dns/dns-default-9kfdx"
Jul 27 21:05:08.091135 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:05:08.091097    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-78ljg"
Jul 27 21:05:08.224424 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:05:08.224229    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-78ljg"]
Jul 27 21:05:08.227773 ip-10-0-131-133 kubenswrapper[2559]: W0727 21:05:08.227744    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcf59a2e6_f3c6_4549_9158_f047bfb7e764.slice/crio-3eed0749896f7ba187e122f9ec90a6b0acf4e05d608a795d3a59b54447813cbf WatchSource:0}: Error finding container 3eed0749896f7ba187e122f9ec90a6b0acf4e05d608a795d3a59b54447813cbf: Status 404 returned error can't find the container with id 3eed0749896f7ba187e122f9ec90a6b0acf4e05d608a795d3a59b54447813cbf
Jul 27 21:05:08.255842 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:05:08.255814    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-78ljg" event={"ID":"cf59a2e6-f3c6-4549-9158-f047bfb7e764","Type":"ContainerStarted","Data":"3eed0749896f7ba187e122f9ec90a6b0acf4e05d608a795d3a59b54447813cbf"}
Jul 27 21:05:08.258603 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:05:08.258577    2559 generic.go:356] "Generic (PLEG): container finished" podID="4275e4fb-a8e5-43a5-8b8c-ed4cc2094674" containerID="90bedffbb339f1f40728c4e14482e2d3cd91b379ba918cc8f580bd75c222c603" exitCode=0
Jul 27 21:05:08.258731 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:05:08.258651    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-2pbtk" event={"ID":"4275e4fb-a8e5-43a5-8b8c-ed4cc2094674","Type":"ContainerDied","Data":"90bedffbb339f1f40728c4e14482e2d3cd91b379ba918cc8f580bd75c222c603"}
Jul 27 21:05:08.528631 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:05:08.526305    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ac10d2c6-12bd-4f89-8d1e-c931206a2e92-metrics-tls\") pod \"dns-default-9kfdx\" (UID: \"ac10d2c6-12bd-4f89-8d1e-c931206a2e92\") " pod="openshift-dns/dns-default-9kfdx"
Jul 27 21:05:08.528631 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:05:08.526399    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/097262d9-cf78-4d89-98cc-cf92d23a4f04-registry-tls\") pod \"image-registry-5b565b454b-2kr9d\" (UID: \"097262d9-cf78-4d89-98cc-cf92d23a4f04\") " pod="openshift-image-registry/image-registry-5b565b454b-2kr9d"
Jul 27 21:05:08.528631 ip-10-0-131-133 kubenswrapper[2559]: E0727 21:05:08.526544    2559 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 27 21:05:08.528631 ip-10-0-131-133 kubenswrapper[2559]: E0727 21:05:08.526561    2559 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5b565b454b-2kr9d: secret "image-registry-tls" not found
Jul 27 21:05:08.528631 ip-10-0-131-133 kubenswrapper[2559]: E0727 21:05:08.526630    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/097262d9-cf78-4d89-98cc-cf92d23a4f04-registry-tls podName:097262d9-cf78-4d89-98cc-cf92d23a4f04 nodeName:}" failed. No retries permitted until 2026-07-27 21:05:09.52661058 +0000 UTC m=+38.105794565 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/097262d9-cf78-4d89-98cc-cf92d23a4f04-registry-tls") pod "image-registry-5b565b454b-2kr9d" (UID: "097262d9-cf78-4d89-98cc-cf92d23a4f04") : secret "image-registry-tls" not found
Jul 27 21:05:08.528631 ip-10-0-131-133 kubenswrapper[2559]: E0727 21:05:08.527152    2559 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 27 21:05:08.528631 ip-10-0-131-133 kubenswrapper[2559]: E0727 21:05:08.527229    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ac10d2c6-12bd-4f89-8d1e-c931206a2e92-metrics-tls podName:ac10d2c6-12bd-4f89-8d1e-c931206a2e92 nodeName:}" failed. No retries permitted until 2026-07-27 21:05:09.527188361 +0000 UTC m=+38.106372361 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/ac10d2c6-12bd-4f89-8d1e-c931206a2e92-metrics-tls") pod "dns-default-9kfdx" (UID: "ac10d2c6-12bd-4f89-8d1e-c931206a2e92") : secret "dns-default-metrics-tls" not found
Jul 27 21:05:08.976255 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:05:08.976197    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pr4c2"
Jul 27 21:05:08.976255 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:05:08.976246    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hmjv6"
Jul 27 21:05:08.976513 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:05:08.976340    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rxl8w"
Jul 27 21:05:08.979864 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:05:08.979771    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 27 21:05:08.979864 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:05:08.979777    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 27 21:05:08.979864 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:05:08.979860    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 27 21:05:08.980446 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:05:08.980427    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 27 21:05:08.980559 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:05:08.980521    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-vs64j\""
Jul 27 21:05:08.980619 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:05:08.980431    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-7jdqc\""
Jul 27 21:05:09.265953 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:05:09.265867    2559 generic.go:356] "Generic (PLEG): container finished" podID="4275e4fb-a8e5-43a5-8b8c-ed4cc2094674" containerID="96d76df63f257a9d4703ed21065610ce4a6bf109420d4781fb5aedabb4276f0d" exitCode=0
Jul 27 21:05:09.266579 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:05:09.265956    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-2pbtk" event={"ID":"4275e4fb-a8e5-43a5-8b8c-ed4cc2094674","Type":"ContainerDied","Data":"96d76df63f257a9d4703ed21065610ce4a6bf109420d4781fb5aedabb4276f0d"}
Jul 27 21:05:09.534259 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:05:09.534151    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/097262d9-cf78-4d89-98cc-cf92d23a4f04-registry-tls\") pod \"image-registry-5b565b454b-2kr9d\" (UID: \"097262d9-cf78-4d89-98cc-cf92d23a4f04\") " pod="openshift-image-registry/image-registry-5b565b454b-2kr9d"
Jul 27 21:05:09.534426 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:05:09.534271    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ac10d2c6-12bd-4f89-8d1e-c931206a2e92-metrics-tls\") pod \"dns-default-9kfdx\" (UID: \"ac10d2c6-12bd-4f89-8d1e-c931206a2e92\") " pod="openshift-dns/dns-default-9kfdx"
Jul 27 21:05:09.534426 ip-10-0-131-133 kubenswrapper[2559]: E0727 21:05:09.534338    2559 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 27 21:05:09.534426 ip-10-0-131-133 kubenswrapper[2559]: E0727 21:05:09.534362    2559 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5b565b454b-2kr9d: secret "image-registry-tls" not found
Jul 27 21:05:09.534426 ip-10-0-131-133 kubenswrapper[2559]: E0727 21:05:09.534397    2559 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 27 21:05:09.534426 ip-10-0-131-133 kubenswrapper[2559]: E0727 21:05:09.534432    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/097262d9-cf78-4d89-98cc-cf92d23a4f04-registry-tls podName:097262d9-cf78-4d89-98cc-cf92d23a4f04 nodeName:}" failed. No retries permitted until 2026-07-27 21:05:11.534411578 +0000 UTC m=+40.113595578 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/097262d9-cf78-4d89-98cc-cf92d23a4f04-registry-tls") pod "image-registry-5b565b454b-2kr9d" (UID: "097262d9-cf78-4d89-98cc-cf92d23a4f04") : secret "image-registry-tls" not found
Jul 27 21:05:09.534696 ip-10-0-131-133 kubenswrapper[2559]: E0727 21:05:09.534460    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ac10d2c6-12bd-4f89-8d1e-c931206a2e92-metrics-tls podName:ac10d2c6-12bd-4f89-8d1e-c931206a2e92 nodeName:}" failed. No retries permitted until 2026-07-27 21:05:11.534441278 +0000 UTC m=+40.113625287 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/ac10d2c6-12bd-4f89-8d1e-c931206a2e92-metrics-tls") pod "dns-default-9kfdx" (UID: "ac10d2c6-12bd-4f89-8d1e-c931206a2e92") : secret "dns-default-metrics-tls" not found
Jul 27 21:05:10.271559 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:05:10.271358    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-2pbtk" event={"ID":"4275e4fb-a8e5-43a5-8b8c-ed4cc2094674","Type":"ContainerStarted","Data":"1604cc3d53ca157c0e4e4782d32b4b7868935dddf4952e7b0f62822cdd1606fc"}
Jul 27 21:05:10.305424 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:05:10.305368    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-2pbtk" podStartSLOduration=9.025231207 podStartE2EDuration="38.305351932s" totalImagesPullingTime="29.280120725s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-27 21:04:32 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 21:05:10.303564693 +0000 UTC m=+38.882748699" watchObservedRunningTime="2026-07-27 21:05:10.305351932 +0000 UTC m=+38.884535940"
Jul 27 21:05:11.275536 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:05:11.275444    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-78ljg" event={"ID":"cf59a2e6-f3c6-4549-9158-f047bfb7e764","Type":"ContainerStarted","Data":"5841396b7b14fd51eca12928650e18fd45ae536acb1cc56a987be8d3e53a08ae"}
Jul 27 21:05:11.301631 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:05:11.301578    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-78ljg" podStartSLOduration=2.216825456 podStartE2EDuration="4.301560883s" totalImagesPullingTime="2.084735427s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 21:05:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 21:05:11.301288635 +0000 UTC m=+39.880472643" watchObservedRunningTime="2026-07-27 21:05:11.301560883 +0000 UTC m=+39.880744890"
Jul 27 21:05:11.552225 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:05:11.552120    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ac10d2c6-12bd-4f89-8d1e-c931206a2e92-metrics-tls\") pod \"dns-default-9kfdx\" (UID: \"ac10d2c6-12bd-4f89-8d1e-c931206a2e92\") " pod="openshift-dns/dns-default-9kfdx"
Jul 27 21:05:11.552225 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:05:11.552179    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/097262d9-cf78-4d89-98cc-cf92d23a4f04-registry-tls\") pod \"image-registry-5b565b454b-2kr9d\" (UID: \"097262d9-cf78-4d89-98cc-cf92d23a4f04\") " pod="openshift-image-registry/image-registry-5b565b454b-2kr9d"
Jul 27 21:05:11.552415 ip-10-0-131-133 kubenswrapper[2559]: E0727 21:05:11.552305    2559 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 27 21:05:11.552415 ip-10-0-131-133 kubenswrapper[2559]: E0727 21:05:11.552318    2559 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5b565b454b-2kr9d: secret "image-registry-tls" not found
Jul 27 21:05:11.552415 ip-10-0-131-133 kubenswrapper[2559]: E0727 21:05:11.552321    2559 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 27 21:05:11.552415 ip-10-0-131-133 kubenswrapper[2559]: E0727 21:05:11.552370    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/097262d9-cf78-4d89-98cc-cf92d23a4f04-registry-tls podName:097262d9-cf78-4d89-98cc-cf92d23a4f04 nodeName:}" failed. No retries permitted until 2026-07-27 21:05:15.552356849 +0000 UTC m=+44.131540834 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/097262d9-cf78-4d89-98cc-cf92d23a4f04-registry-tls") pod "image-registry-5b565b454b-2kr9d" (UID: "097262d9-cf78-4d89-98cc-cf92d23a4f04") : secret "image-registry-tls" not found
Jul 27 21:05:11.552415 ip-10-0-131-133 kubenswrapper[2559]: E0727 21:05:11.552382    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ac10d2c6-12bd-4f89-8d1e-c931206a2e92-metrics-tls podName:ac10d2c6-12bd-4f89-8d1e-c931206a2e92 nodeName:}" failed. No retries permitted until 2026-07-27 21:05:15.552376564 +0000 UTC m=+44.131560549 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/ac10d2c6-12bd-4f89-8d1e-c931206a2e92-metrics-tls") pod "dns-default-9kfdx" (UID: "ac10d2c6-12bd-4f89-8d1e-c931206a2e92") : secret "dns-default-metrics-tls" not found
Jul 27 21:05:15.576431 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:05:15.576384    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ac10d2c6-12bd-4f89-8d1e-c931206a2e92-metrics-tls\") pod \"dns-default-9kfdx\" (UID: \"ac10d2c6-12bd-4f89-8d1e-c931206a2e92\") " pod="openshift-dns/dns-default-9kfdx"
Jul 27 21:05:15.576832 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:05:15.576447    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/097262d9-cf78-4d89-98cc-cf92d23a4f04-registry-tls\") pod \"image-registry-5b565b454b-2kr9d\" (UID: \"097262d9-cf78-4d89-98cc-cf92d23a4f04\") " pod="openshift-image-registry/image-registry-5b565b454b-2kr9d"
Jul 27 21:05:15.576832 ip-10-0-131-133 kubenswrapper[2559]: E0727 21:05:15.576552    2559 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 27 21:05:15.576832 ip-10-0-131-133 kubenswrapper[2559]: E0727 21:05:15.576636    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ac10d2c6-12bd-4f89-8d1e-c931206a2e92-metrics-tls podName:ac10d2c6-12bd-4f89-8d1e-c931206a2e92 nodeName:}" failed. No retries permitted until 2026-07-27 21:05:23.576620384 +0000 UTC m=+52.155804369 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/ac10d2c6-12bd-4f89-8d1e-c931206a2e92-metrics-tls") pod "dns-default-9kfdx" (UID: "ac10d2c6-12bd-4f89-8d1e-c931206a2e92") : secret "dns-default-metrics-tls" not found
Jul 27 21:05:15.576832 ip-10-0-131-133 kubenswrapper[2559]: E0727 21:05:15.576556    2559 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 27 21:05:15.576832 ip-10-0-131-133 kubenswrapper[2559]: E0727 21:05:15.576660    2559 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5b565b454b-2kr9d: secret "image-registry-tls" not found
Jul 27 21:05:15.576832 ip-10-0-131-133 kubenswrapper[2559]: E0727 21:05:15.576709    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/097262d9-cf78-4d89-98cc-cf92d23a4f04-registry-tls podName:097262d9-cf78-4d89-98cc-cf92d23a4f04 nodeName:}" failed. No retries permitted until 2026-07-27 21:05:23.576697583 +0000 UTC m=+52.155881568 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/097262d9-cf78-4d89-98cc-cf92d23a4f04-registry-tls") pod "image-registry-5b565b454b-2kr9d" (UID: "097262d9-cf78-4d89-98cc-cf92d23a4f04") : secret "image-registry-tls" not found
Jul 27 21:05:23.626202 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:05:23.626161    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ac10d2c6-12bd-4f89-8d1e-c931206a2e92-metrics-tls\") pod \"dns-default-9kfdx\" (UID: \"ac10d2c6-12bd-4f89-8d1e-c931206a2e92\") " pod="openshift-dns/dns-default-9kfdx"
Jul 27 21:05:23.626679 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:05:23.626236    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/097262d9-cf78-4d89-98cc-cf92d23a4f04-registry-tls\") pod \"image-registry-5b565b454b-2kr9d\" (UID: \"097262d9-cf78-4d89-98cc-cf92d23a4f04\") " pod="openshift-image-registry/image-registry-5b565b454b-2kr9d"
Jul 27 21:05:23.626679 ip-10-0-131-133 kubenswrapper[2559]: E0727 21:05:23.626325    2559 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 27 21:05:23.626679 ip-10-0-131-133 kubenswrapper[2559]: E0727 21:05:23.626339    2559 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5b565b454b-2kr9d: secret "image-registry-tls" not found
Jul 27 21:05:23.626679 ip-10-0-131-133 kubenswrapper[2559]: E0727 21:05:23.626384    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/097262d9-cf78-4d89-98cc-cf92d23a4f04-registry-tls podName:097262d9-cf78-4d89-98cc-cf92d23a4f04 nodeName:}" failed. No retries permitted until 2026-07-27 21:05:39.626370988 +0000 UTC m=+68.205554973 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/097262d9-cf78-4d89-98cc-cf92d23a4f04-registry-tls") pod "image-registry-5b565b454b-2kr9d" (UID: "097262d9-cf78-4d89-98cc-cf92d23a4f04") : secret "image-registry-tls" not found
Jul 27 21:05:23.626679 ip-10-0-131-133 kubenswrapper[2559]: E0727 21:05:23.626324    2559 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 27 21:05:23.626679 ip-10-0-131-133 kubenswrapper[2559]: E0727 21:05:23.626455    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ac10d2c6-12bd-4f89-8d1e-c931206a2e92-metrics-tls podName:ac10d2c6-12bd-4f89-8d1e-c931206a2e92 nodeName:}" failed. No retries permitted until 2026-07-27 21:05:39.626440708 +0000 UTC m=+68.205624706 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/ac10d2c6-12bd-4f89-8d1e-c931206a2e92-metrics-tls") pod "dns-default-9kfdx" (UID: "ac10d2c6-12bd-4f89-8d1e-c931206a2e92") : secret "dns-default-metrics-tls" not found
Jul 27 21:05:32.260517 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:05:32.260483    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-qs89h"
Jul 27 21:05:37.632739 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:05:37.632695    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/00ef4d6d-0b8e-4ece-ba97-36a9e236ec53-original-pull-secret\") pod \"global-pull-secret-syncer-rxl8w\" (UID: \"00ef4d6d-0b8e-4ece-ba97-36a9e236ec53\") " pod="kube-system/global-pull-secret-syncer-rxl8w"
Jul 27 21:05:37.637164 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:05:37.637144    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 27 21:05:37.646566 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:05:37.646536    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/00ef4d6d-0b8e-4ece-ba97-36a9e236ec53-original-pull-secret\") pod \"global-pull-secret-syncer-rxl8w\" (UID: \"00ef4d6d-0b8e-4ece-ba97-36a9e236ec53\") " pod="kube-system/global-pull-secret-syncer-rxl8w"
Jul 27 21:05:37.733915 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:05:37.733875    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mtwdg\" (UniqueName: \"kubernetes.io/projected/95b2123e-132a-42a8-8edb-33ce19c2df51-kube-api-access-mtwdg\") pod \"network-check-target-hmjv6\" (UID: \"95b2123e-132a-42a8-8edb-33ce19c2df51\") " pod="openshift-network-diagnostics/network-check-target-hmjv6"
Jul 27 21:05:37.734084 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:05:37.733928    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5851de39-848d-4a52-bc17-fefe1bf5407a-metrics-certs\") pod \"network-metrics-daemon-pr4c2\" (UID: \"5851de39-848d-4a52-bc17-fefe1bf5407a\") " pod="openshift-multus/network-metrics-daemon-pr4c2"
Jul 27 21:05:37.736505 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:05:37.736484    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 27 21:05:37.736560 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:05:37.736532    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 27 21:05:37.744432 ip-10-0-131-133 kubenswrapper[2559]: E0727 21:05:37.744401    2559 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Jul 27 21:05:37.744505 ip-10-0-131-133 kubenswrapper[2559]: E0727 21:05:37.744475    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5851de39-848d-4a52-bc17-fefe1bf5407a-metrics-certs podName:5851de39-848d-4a52-bc17-fefe1bf5407a nodeName:}" failed. No retries permitted until 2026-07-27 21:06:41.744457128 +0000 UTC m=+130.323641114 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/5851de39-848d-4a52-bc17-fefe1bf5407a-metrics-certs") pod "network-metrics-daemon-pr4c2" (UID: "5851de39-848d-4a52-bc17-fefe1bf5407a") : secret "metrics-daemon-secret" not found
Jul 27 21:05:37.746798 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:05:37.746782    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 27 21:05:37.757434 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:05:37.757401    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mtwdg\" (UniqueName: \"kubernetes.io/projected/95b2123e-132a-42a8-8edb-33ce19c2df51-kube-api-access-mtwdg\") pod \"network-check-target-hmjv6\" (UID: \"95b2123e-132a-42a8-8edb-33ce19c2df51\") " pod="openshift-network-diagnostics/network-check-target-hmjv6"
Jul 27 21:05:37.795320 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:05:37.795284    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rxl8w"
Jul 27 21:05:37.803589 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:05:37.803563    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-7jdqc\""
Jul 27 21:05:37.811584 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:05:37.811563    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hmjv6"
Jul 27 21:05:37.949905 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:05:37.949877    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-rxl8w"]
Jul 27 21:05:37.953421 ip-10-0-131-133 kubenswrapper[2559]: W0727 21:05:37.953393    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod00ef4d6d_0b8e_4ece_ba97_36a9e236ec53.slice/crio-468a45c5c51b2180103c6d9cca408240a4d3899182cc701d557e4b5202b3bccb WatchSource:0}: Error finding container 468a45c5c51b2180103c6d9cca408240a4d3899182cc701d557e4b5202b3bccb: Status 404 returned error can't find the container with id 468a45c5c51b2180103c6d9cca408240a4d3899182cc701d557e4b5202b3bccb
Jul 27 21:05:37.971446 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:05:37.971401    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-hmjv6"]
Jul 27 21:05:37.973725 ip-10-0-131-133 kubenswrapper[2559]: W0727 21:05:37.973702    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod95b2123e_132a_42a8_8edb_33ce19c2df51.slice/crio-ab708535f104363d328b37662834d4c803a9b78df6006abae423678318afaf36 WatchSource:0}: Error finding container ab708535f104363d328b37662834d4c803a9b78df6006abae423678318afaf36: Status 404 returned error can't find the container with id ab708535f104363d328b37662834d4c803a9b78df6006abae423678318afaf36
Jul 27 21:05:38.341015 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:05:38.340929    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-hmjv6" event={"ID":"95b2123e-132a-42a8-8edb-33ce19c2df51","Type":"ContainerStarted","Data":"ab708535f104363d328b37662834d4c803a9b78df6006abae423678318afaf36"}
Jul 27 21:05:38.341859 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:05:38.341838    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-rxl8w" event={"ID":"00ef4d6d-0b8e-4ece-ba97-36a9e236ec53","Type":"ContainerStarted","Data":"468a45c5c51b2180103c6d9cca408240a4d3899182cc701d557e4b5202b3bccb"}
Jul 27 21:05:39.650634 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:05:39.650586    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ac10d2c6-12bd-4f89-8d1e-c931206a2e92-metrics-tls\") pod \"dns-default-9kfdx\" (UID: \"ac10d2c6-12bd-4f89-8d1e-c931206a2e92\") " pod="openshift-dns/dns-default-9kfdx"
Jul 27 21:05:39.651125 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:05:39.650658    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/097262d9-cf78-4d89-98cc-cf92d23a4f04-registry-tls\") pod \"image-registry-5b565b454b-2kr9d\" (UID: \"097262d9-cf78-4d89-98cc-cf92d23a4f04\") " pod="openshift-image-registry/image-registry-5b565b454b-2kr9d"
Jul 27 21:05:39.651125 ip-10-0-131-133 kubenswrapper[2559]: E0727 21:05:39.650785    2559 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 27 21:05:39.651125 ip-10-0-131-133 kubenswrapper[2559]: E0727 21:05:39.650799    2559 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5b565b454b-2kr9d: secret "image-registry-tls" not found
Jul 27 21:05:39.651125 ip-10-0-131-133 kubenswrapper[2559]: E0727 21:05:39.650871    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/097262d9-cf78-4d89-98cc-cf92d23a4f04-registry-tls podName:097262d9-cf78-4d89-98cc-cf92d23a4f04 nodeName:}" failed. No retries permitted until 2026-07-27 21:06:11.65085136 +0000 UTC m=+100.230035356 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/097262d9-cf78-4d89-98cc-cf92d23a4f04-registry-tls") pod "image-registry-5b565b454b-2kr9d" (UID: "097262d9-cf78-4d89-98cc-cf92d23a4f04") : secret "image-registry-tls" not found
Jul 27 21:05:39.651125 ip-10-0-131-133 kubenswrapper[2559]: E0727 21:05:39.650785    2559 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 27 21:05:39.651125 ip-10-0-131-133 kubenswrapper[2559]: E0727 21:05:39.650938    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ac10d2c6-12bd-4f89-8d1e-c931206a2e92-metrics-tls podName:ac10d2c6-12bd-4f89-8d1e-c931206a2e92 nodeName:}" failed. No retries permitted until 2026-07-27 21:06:11.650923516 +0000 UTC m=+100.230107513 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/ac10d2c6-12bd-4f89-8d1e-c931206a2e92-metrics-tls") pod "dns-default-9kfdx" (UID: "ac10d2c6-12bd-4f89-8d1e-c931206a2e92") : secret "dns-default-metrics-tls" not found
Jul 27 21:05:42.021788 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:05:42.020791    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console-operator/console-operator-584dcb6b54-bjvbc"]
Jul 27 21:05:42.022960 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:05:42.022935    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-9d4zq"]
Jul 27 21:05:42.023117 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:05:42.023090    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-584dcb6b54-bjvbc"
Jul 27 21:05:42.024777 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:05:42.024757    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-9d4zq"
Jul 27 21:05:42.027417 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:05:42.027270    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\""
Jul 27 21:05:42.027417 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:05:42.027395    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\""
Jul 27 21:05:42.027569 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:05:42.027394    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\""
Jul 27 21:05:42.027886 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:05:42.027864    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\""
Jul 27 21:05:42.028050 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:05:42.027979    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\""
Jul 27 21:05:42.028241 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:05:42.028052    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-zvwwp\""
Jul 27 21:05:42.028241 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:05:42.028169    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\""
Jul 27 21:05:42.028241 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:05:42.028188    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\""
Jul 27 21:05:42.028241 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:05:42.028188    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-5jg6c\""
Jul 27 21:05:42.028489 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:05:42.028301    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\""
Jul 27 21:05:42.035076 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:05:42.035059    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\""
Jul 27 21:05:42.048910 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:05:42.048882    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-584dcb6b54-bjvbc"]
Jul 27 21:05:42.050893 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:05:42.050866    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-9d4zq"]
Jul 27 21:05:42.073073 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:05:42.073038    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/cba53c9a-64c0-4afa-8731-35521e196622-serving-cert\") pod \"console-operator-584dcb6b54-bjvbc\" (UID: \"cba53c9a-64c0-4afa-8731-35521e196622\") " pod="openshift-console-operator/console-operator-584dcb6b54-bjvbc"
Jul 27 21:05:42.073301 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:05:42.073126    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d7ld5\" (UniqueName: \"kubernetes.io/projected/9e560b08-69f7-4ee3-8236-22a5c398a476-kube-api-access-d7ld5\") pod \"kube-storage-version-migrator-operator-6f4d6fb6bc-9d4zq\" (UID: \"9e560b08-69f7-4ee3-8236-22a5c398a476\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-9d4zq"
Jul 27 21:05:42.073301 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:05:42.073157    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tg8rl\" (UniqueName: \"kubernetes.io/projected/cba53c9a-64c0-4afa-8731-35521e196622-kube-api-access-tg8rl\") pod \"console-operator-584dcb6b54-bjvbc\" (UID: \"cba53c9a-64c0-4afa-8731-35521e196622\") " pod="openshift-console-operator/console-operator-584dcb6b54-bjvbc"
Jul 27 21:05:42.073301 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:05:42.073230    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/cba53c9a-64c0-4afa-8731-35521e196622-trusted-ca\") pod \"console-operator-584dcb6b54-bjvbc\" (UID: \"cba53c9a-64c0-4afa-8731-35521e196622\") " pod="openshift-console-operator/console-operator-584dcb6b54-bjvbc"
Jul 27 21:05:42.073301 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:05:42.073295    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/9e560b08-69f7-4ee3-8236-22a5c398a476-config\") pod \"kube-storage-version-migrator-operator-6f4d6fb6bc-9d4zq\" (UID: \"9e560b08-69f7-4ee3-8236-22a5c398a476\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-9d4zq"
Jul 27 21:05:42.073528 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:05:42.073322    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/cba53c9a-64c0-4afa-8731-35521e196622-config\") pod \"console-operator-584dcb6b54-bjvbc\" (UID: \"cba53c9a-64c0-4afa-8731-35521e196622\") " pod="openshift-console-operator/console-operator-584dcb6b54-bjvbc"
Jul 27 21:05:42.073528 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:05:42.073362    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/9e560b08-69f7-4ee3-8236-22a5c398a476-serving-cert\") pod \"kube-storage-version-migrator-operator-6f4d6fb6bc-9d4zq\" (UID: \"9e560b08-69f7-4ee3-8236-22a5c398a476\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-9d4zq"
Jul 27 21:05:42.116615 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:05:42.116582    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-548b489cb8-dxktc"]
Jul 27 21:05:42.118542 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:05:42.118520    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-dxktc"
Jul 27 21:05:42.122305 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:05:42.122206    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Jul 27 21:05:42.122305 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:05:42.122281    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\""
Jul 27 21:05:42.122305 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:05:42.122293    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-6rdvw\""
Jul 27 21:05:42.122547 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:05:42.122383    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\""
Jul 27 21:05:42.122547 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:05:42.122494    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Jul 27 21:05:42.144450 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:05:42.144406    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-548b489cb8-dxktc"]
Jul 27 21:05:42.174655 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:05:42.174617    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/cba53c9a-64c0-4afa-8731-35521e196622-trusted-ca\") pod \"console-operator-584dcb6b54-bjvbc\" (UID: \"cba53c9a-64c0-4afa-8731-35521e196622\") " pod="openshift-console-operator/console-operator-584dcb6b54-bjvbc"
Jul 27 21:05:42.174835 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:05:42.174671    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7652t\" (UniqueName: \"kubernetes.io/projected/a05c7c31-10b9-4e3b-84a9-afe9cf929e07-kube-api-access-7652t\") pod \"cluster-monitoring-operator-548b489cb8-dxktc\" (UID: \"a05c7c31-10b9-4e3b-84a9-afe9cf929e07\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-dxktc"
Jul 27 21:05:42.174835 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:05:42.174704    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/9e560b08-69f7-4ee3-8236-22a5c398a476-config\") pod \"kube-storage-version-migrator-operator-6f4d6fb6bc-9d4zq\" (UID: \"9e560b08-69f7-4ee3-8236-22a5c398a476\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-9d4zq"
Jul 27 21:05:42.174835 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:05:42.174728    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/cba53c9a-64c0-4afa-8731-35521e196622-config\") pod \"console-operator-584dcb6b54-bjvbc\" (UID: \"cba53c9a-64c0-4afa-8731-35521e196622\") " pod="openshift-console-operator/console-operator-584dcb6b54-bjvbc"
Jul 27 21:05:42.174835 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:05:42.174772    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/9e560b08-69f7-4ee3-8236-22a5c398a476-serving-cert\") pod \"kube-storage-version-migrator-operator-6f4d6fb6bc-9d4zq\" (UID: \"9e560b08-69f7-4ee3-8236-22a5c398a476\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-9d4zq"
Jul 27 21:05:42.175040 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:05:42.174936    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/cba53c9a-64c0-4afa-8731-35521e196622-serving-cert\") pod \"console-operator-584dcb6b54-bjvbc\" (UID: \"cba53c9a-64c0-4afa-8731-35521e196622\") " pod="openshift-console-operator/console-operator-584dcb6b54-bjvbc"
Jul 27 21:05:42.175040 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:05:42.174973    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/a05c7c31-10b9-4e3b-84a9-afe9cf929e07-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-548b489cb8-dxktc\" (UID: \"a05c7c31-10b9-4e3b-84a9-afe9cf929e07\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-dxktc"
Jul 27 21:05:42.175040 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:05:42.175017    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/a05c7c31-10b9-4e3b-84a9-afe9cf929e07-telemetry-config\") pod \"cluster-monitoring-operator-548b489cb8-dxktc\" (UID: \"a05c7c31-10b9-4e3b-84a9-afe9cf929e07\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-dxktc"
Jul 27 21:05:42.175186 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:05:42.175078    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d7ld5\" (UniqueName: \"kubernetes.io/projected/9e560b08-69f7-4ee3-8236-22a5c398a476-kube-api-access-d7ld5\") pod \"kube-storage-version-migrator-operator-6f4d6fb6bc-9d4zq\" (UID: \"9e560b08-69f7-4ee3-8236-22a5c398a476\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-9d4zq"
Jul 27 21:05:42.175186 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:05:42.175103    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tg8rl\" (UniqueName: \"kubernetes.io/projected/cba53c9a-64c0-4afa-8731-35521e196622-kube-api-access-tg8rl\") pod \"console-operator-584dcb6b54-bjvbc\" (UID: \"cba53c9a-64c0-4afa-8731-35521e196622\") " pod="openshift-console-operator/console-operator-584dcb6b54-bjvbc"
Jul 27 21:05:42.175473 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:05:42.175427    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/9e560b08-69f7-4ee3-8236-22a5c398a476-config\") pod \"kube-storage-version-migrator-operator-6f4d6fb6bc-9d4zq\" (UID: \"9e560b08-69f7-4ee3-8236-22a5c398a476\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-9d4zq"
Jul 27 21:05:42.175702 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:05:42.175659    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/cba53c9a-64c0-4afa-8731-35521e196622-config\") pod \"console-operator-584dcb6b54-bjvbc\" (UID: \"cba53c9a-64c0-4afa-8731-35521e196622\") " pod="openshift-console-operator/console-operator-584dcb6b54-bjvbc"
Jul 27 21:05:42.175702 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:05:42.175679    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/cba53c9a-64c0-4afa-8731-35521e196622-trusted-ca\") pod \"console-operator-584dcb6b54-bjvbc\" (UID: \"cba53c9a-64c0-4afa-8731-35521e196622\") " pod="openshift-console-operator/console-operator-584dcb6b54-bjvbc"
Jul 27 21:05:42.177733 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:05:42.177709    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/9e560b08-69f7-4ee3-8236-22a5c398a476-serving-cert\") pod \"kube-storage-version-migrator-operator-6f4d6fb6bc-9d4zq\" (UID: \"9e560b08-69f7-4ee3-8236-22a5c398a476\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-9d4zq"
Jul 27 21:05:42.177878 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:05:42.177861    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/cba53c9a-64c0-4afa-8731-35521e196622-serving-cert\") pod \"console-operator-584dcb6b54-bjvbc\" (UID: \"cba53c9a-64c0-4afa-8731-35521e196622\") " pod="openshift-console-operator/console-operator-584dcb6b54-bjvbc"
Jul 27 21:05:42.184358 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:05:42.184332    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tg8rl\" (UniqueName: \"kubernetes.io/projected/cba53c9a-64c0-4afa-8731-35521e196622-kube-api-access-tg8rl\") pod \"console-operator-584dcb6b54-bjvbc\" (UID: \"cba53c9a-64c0-4afa-8731-35521e196622\") " pod="openshift-console-operator/console-operator-584dcb6b54-bjvbc"
Jul 27 21:05:42.184829 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:05:42.184807    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d7ld5\" (UniqueName: \"kubernetes.io/projected/9e560b08-69f7-4ee3-8236-22a5c398a476-kube-api-access-d7ld5\") pod \"kube-storage-version-migrator-operator-6f4d6fb6bc-9d4zq\" (UID: \"9e560b08-69f7-4ee3-8236-22a5c398a476\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-9d4zq"
Jul 27 21:05:42.275656 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:05:42.275571    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7652t\" (UniqueName: \"kubernetes.io/projected/a05c7c31-10b9-4e3b-84a9-afe9cf929e07-kube-api-access-7652t\") pod \"cluster-monitoring-operator-548b489cb8-dxktc\" (UID: \"a05c7c31-10b9-4e3b-84a9-afe9cf929e07\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-dxktc"
Jul 27 21:05:42.275656 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:05:42.275645    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/a05c7c31-10b9-4e3b-84a9-afe9cf929e07-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-548b489cb8-dxktc\" (UID: \"a05c7c31-10b9-4e3b-84a9-afe9cf929e07\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-dxktc"
Jul 27 21:05:42.275897 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:05:42.275672    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/a05c7c31-10b9-4e3b-84a9-afe9cf929e07-telemetry-config\") pod \"cluster-monitoring-operator-548b489cb8-dxktc\" (UID: \"a05c7c31-10b9-4e3b-84a9-afe9cf929e07\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-dxktc"
Jul 27 21:05:42.275897 ip-10-0-131-133 kubenswrapper[2559]: E0727 21:05:42.275794    2559 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Jul 27 21:05:42.275897 ip-10-0-131-133 kubenswrapper[2559]: E0727 21:05:42.275885    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a05c7c31-10b9-4e3b-84a9-afe9cf929e07-cluster-monitoring-operator-tls podName:a05c7c31-10b9-4e3b-84a9-afe9cf929e07 nodeName:}" failed. No retries permitted until 2026-07-27 21:05:42.775865844 +0000 UTC m=+71.355049829 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/a05c7c31-10b9-4e3b-84a9-afe9cf929e07-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-548b489cb8-dxktc" (UID: "a05c7c31-10b9-4e3b-84a9-afe9cf929e07") : secret "cluster-monitoring-operator-tls" not found
Jul 27 21:05:42.276455 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:05:42.276435    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/a05c7c31-10b9-4e3b-84a9-afe9cf929e07-telemetry-config\") pod \"cluster-monitoring-operator-548b489cb8-dxktc\" (UID: \"a05c7c31-10b9-4e3b-84a9-afe9cf929e07\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-dxktc"
Jul 27 21:05:42.305719 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:05:42.305689    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7652t\" (UniqueName: \"kubernetes.io/projected/a05c7c31-10b9-4e3b-84a9-afe9cf929e07-kube-api-access-7652t\") pod \"cluster-monitoring-operator-548b489cb8-dxktc\" (UID: \"a05c7c31-10b9-4e3b-84a9-afe9cf929e07\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-dxktc"
Jul 27 21:05:42.336585 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:05:42.336546    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-584dcb6b54-bjvbc"
Jul 27 21:05:42.342052 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:05:42.342026    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-9d4zq"
Jul 27 21:05:42.596527 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:05:42.596444    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-584dcb6b54-bjvbc"]
Jul 27 21:05:42.600302 ip-10-0-131-133 kubenswrapper[2559]: W0727 21:05:42.600275    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcba53c9a_64c0_4afa_8731_35521e196622.slice/crio-cdfa3a15fc08018b707dc835e2aa77e59806183b8f49e7b76855ae228350de4c WatchSource:0}: Error finding container cdfa3a15fc08018b707dc835e2aa77e59806183b8f49e7b76855ae228350de4c: Status 404 returned error can't find the container with id cdfa3a15fc08018b707dc835e2aa77e59806183b8f49e7b76855ae228350de4c
Jul 27 21:05:42.610765 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:05:42.610738    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-9d4zq"]
Jul 27 21:05:42.615660 ip-10-0-131-133 kubenswrapper[2559]: W0727 21:05:42.615615    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9e560b08_69f7_4ee3_8236_22a5c398a476.slice/crio-ef110e087facad42f0cc272cdbe86d591465341d4ea8ea637aa4646327ce3c1a WatchSource:0}: Error finding container ef110e087facad42f0cc272cdbe86d591465341d4ea8ea637aa4646327ce3c1a: Status 404 returned error can't find the container with id ef110e087facad42f0cc272cdbe86d591465341d4ea8ea637aa4646327ce3c1a
Jul 27 21:05:42.781488 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:05:42.781448    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/a05c7c31-10b9-4e3b-84a9-afe9cf929e07-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-548b489cb8-dxktc\" (UID: \"a05c7c31-10b9-4e3b-84a9-afe9cf929e07\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-dxktc"
Jul 27 21:05:42.781677 ip-10-0-131-133 kubenswrapper[2559]: E0727 21:05:42.781626    2559 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Jul 27 21:05:42.781724 ip-10-0-131-133 kubenswrapper[2559]: E0727 21:05:42.781705    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a05c7c31-10b9-4e3b-84a9-afe9cf929e07-cluster-monitoring-operator-tls podName:a05c7c31-10b9-4e3b-84a9-afe9cf929e07 nodeName:}" failed. No retries permitted until 2026-07-27 21:05:43.781688594 +0000 UTC m=+72.360872584 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/a05c7c31-10b9-4e3b-84a9-afe9cf929e07-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-548b489cb8-dxktc" (UID: "a05c7c31-10b9-4e3b-84a9-afe9cf929e07") : secret "cluster-monitoring-operator-tls" not found
Jul 27 21:05:43.357053 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:05:43.357019    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-584dcb6b54-bjvbc" event={"ID":"cba53c9a-64c0-4afa-8731-35521e196622","Type":"ContainerStarted","Data":"cdfa3a15fc08018b707dc835e2aa77e59806183b8f49e7b76855ae228350de4c"}
Jul 27 21:05:43.358823 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:05:43.358763    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-hmjv6" event={"ID":"95b2123e-132a-42a8-8edb-33ce19c2df51","Type":"ContainerStarted","Data":"f82dbb61d2e41007d613cc082ad1682836491691223b3241a2c5b525d7f1d52f"}
Jul 27 21:05:43.359008 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:05:43.358989    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-hmjv6"
Jul 27 21:05:43.360035 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:05:43.360008    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-9d4zq" event={"ID":"9e560b08-69f7-4ee3-8236-22a5c398a476","Type":"ContainerStarted","Data":"ef110e087facad42f0cc272cdbe86d591465341d4ea8ea637aa4646327ce3c1a"}
Jul 27 21:05:43.361720 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:05:43.361696    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-rxl8w" event={"ID":"00ef4d6d-0b8e-4ece-ba97-36a9e236ec53","Type":"ContainerStarted","Data":"7023faf4afd8a500d19d6562597f799051a91f711884c353fd53d9ab7798137a"}
Jul 27 21:05:43.376420 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:05:43.376130    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-hmjv6" podStartSLOduration=66.888650209 podStartE2EDuration="1m11.376112435s" totalImagesPullingTime="4.487462226s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 21:04:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 21:05:43.375055088 +0000 UTC m=+71.954239096" watchObservedRunningTime="2026-07-27 21:05:43.376112435 +0000 UTC m=+71.955296442"
Jul 27 21:05:43.391926 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:05:43.391865    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-rxl8w" podStartSLOduration=66.875531711 podStartE2EDuration="1m11.391844777s" totalImagesPullingTime="4.516313066s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-27 21:04:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 21:05:43.391798456 +0000 UTC m=+71.970982468" watchObservedRunningTime="2026-07-27 21:05:43.391844777 +0000 UTC m=+71.971028786"
Jul 27 21:05:43.791559 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:05:43.791508    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/a05c7c31-10b9-4e3b-84a9-afe9cf929e07-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-548b489cb8-dxktc\" (UID: \"a05c7c31-10b9-4e3b-84a9-afe9cf929e07\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-dxktc"
Jul 27 21:05:43.791764 ip-10-0-131-133 kubenswrapper[2559]: E0727 21:05:43.791723    2559 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Jul 27 21:05:43.791849 ip-10-0-131-133 kubenswrapper[2559]: E0727 21:05:43.791832    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a05c7c31-10b9-4e3b-84a9-afe9cf929e07-cluster-monitoring-operator-tls podName:a05c7c31-10b9-4e3b-84a9-afe9cf929e07 nodeName:}" failed. No retries permitted until 2026-07-27 21:05:45.791797847 +0000 UTC m=+74.370981846 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/a05c7c31-10b9-4e3b-84a9-afe9cf929e07-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-548b489cb8-dxktc" (UID: "a05c7c31-10b9-4e3b-84a9-afe9cf929e07") : secret "cluster-monitoring-operator-tls" not found
Jul 27 21:05:45.367861 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:05:45.367816    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-9d4zq" event={"ID":"9e560b08-69f7-4ee3-8236-22a5c398a476","Type":"ContainerStarted","Data":"497e36afe3a388209e358dbc69ab906fffe45bb0712b69ebf4dba141ec699ca9"}
Jul 27 21:05:45.369359 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:05:45.369342    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-bjvbc_cba53c9a-64c0-4afa-8731-35521e196622/console-operator/0.log"
Jul 27 21:05:45.369435 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:05:45.369376    2559 generic.go:356] "Generic (PLEG): container finished" podID="cba53c9a-64c0-4afa-8731-35521e196622" containerID="6ebdb4b7121557c7eb23623ca12ce13522c62c98c6c87381080e0a509b6832df" exitCode=255
Jul 27 21:05:45.369435 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:05:45.369423    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-584dcb6b54-bjvbc" event={"ID":"cba53c9a-64c0-4afa-8731-35521e196622","Type":"ContainerDied","Data":"6ebdb4b7121557c7eb23623ca12ce13522c62c98c6c87381080e0a509b6832df"}
Jul 27 21:05:45.369611 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:05:45.369598    2559 scope.go:117] "RemoveContainer" containerID="6ebdb4b7121557c7eb23623ca12ce13522c62c98c6c87381080e0a509b6832df"
Jul 27 21:05:45.386251 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:05:45.386183    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-9d4zq" podStartSLOduration=2.156639156 podStartE2EDuration="4.386166498s" totalImagesPullingTime="2.229527342s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 21:05:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 21:05:45.385809448 +0000 UTC m=+73.964993456" watchObservedRunningTime="2026-07-27 21:05:45.386166498 +0000 UTC m=+73.965350504"
Jul 27 21:05:45.760406 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:05:45.760371    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-dz5rp"]
Jul 27 21:05:45.762681 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:05:45.762664    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-dz5rp"
Jul 27 21:05:45.766905 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:05:45.766881    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Jul 27 21:05:45.767066 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:05:45.767045    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-tk2sn\""
Jul 27 21:05:45.767180 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:05:45.767090    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Jul 27 21:05:45.767900 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:05:45.767880    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Jul 27 21:05:45.767900 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:05:45.767899    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Jul 27 21:05:45.778565 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:05:45.778538    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-dz5rp"]
Jul 27 21:05:45.811337 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:05:45.811308    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/922d0edc-f817-477a-bc19-a40bacc930ba-crio-socket\") pod \"insights-runtime-extractor-dz5rp\" (UID: \"922d0edc-f817-477a-bc19-a40bacc930ba\") " pod="openshift-insights/insights-runtime-extractor-dz5rp"
Jul 27 21:05:45.811337 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:05:45.811340    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/922d0edc-f817-477a-bc19-a40bacc930ba-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-dz5rp\" (UID: \"922d0edc-f817-477a-bc19-a40bacc930ba\") " pod="openshift-insights/insights-runtime-extractor-dz5rp"
Jul 27 21:05:45.811540 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:05:45.811374    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ls844\" (UniqueName: \"kubernetes.io/projected/922d0edc-f817-477a-bc19-a40bacc930ba-kube-api-access-ls844\") pod \"insights-runtime-extractor-dz5rp\" (UID: \"922d0edc-f817-477a-bc19-a40bacc930ba\") " pod="openshift-insights/insights-runtime-extractor-dz5rp"
Jul 27 21:05:45.811540 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:05:45.811478    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/922d0edc-f817-477a-bc19-a40bacc930ba-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-dz5rp\" (UID: \"922d0edc-f817-477a-bc19-a40bacc930ba\") " pod="openshift-insights/insights-runtime-extractor-dz5rp"
Jul 27 21:05:45.811540 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:05:45.811522    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/a05c7c31-10b9-4e3b-84a9-afe9cf929e07-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-548b489cb8-dxktc\" (UID: \"a05c7c31-10b9-4e3b-84a9-afe9cf929e07\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-dxktc"
Jul 27 21:05:45.811637 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:05:45.811577    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/922d0edc-f817-477a-bc19-a40bacc930ba-data-volume\") pod \"insights-runtime-extractor-dz5rp\" (UID: \"922d0edc-f817-477a-bc19-a40bacc930ba\") " pod="openshift-insights/insights-runtime-extractor-dz5rp"
Jul 27 21:05:45.811637 ip-10-0-131-133 kubenswrapper[2559]: E0727 21:05:45.811627    2559 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Jul 27 21:05:45.811700 ip-10-0-131-133 kubenswrapper[2559]: E0727 21:05:45.811684    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a05c7c31-10b9-4e3b-84a9-afe9cf929e07-cluster-monitoring-operator-tls podName:a05c7c31-10b9-4e3b-84a9-afe9cf929e07 nodeName:}" failed. No retries permitted until 2026-07-27 21:05:49.81166805 +0000 UTC m=+78.390852036 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/a05c7c31-10b9-4e3b-84a9-afe9cf929e07-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-548b489cb8-dxktc" (UID: "a05c7c31-10b9-4e3b-84a9-afe9cf929e07") : secret "cluster-monitoring-operator-tls" not found
Jul 27 21:05:45.912739 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:05:45.912695    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/922d0edc-f817-477a-bc19-a40bacc930ba-crio-socket\") pod \"insights-runtime-extractor-dz5rp\" (UID: \"922d0edc-f817-477a-bc19-a40bacc930ba\") " pod="openshift-insights/insights-runtime-extractor-dz5rp"
Jul 27 21:05:45.912739 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:05:45.912740    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/922d0edc-f817-477a-bc19-a40bacc930ba-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-dz5rp\" (UID: \"922d0edc-f817-477a-bc19-a40bacc930ba\") " pod="openshift-insights/insights-runtime-extractor-dz5rp"
Jul 27 21:05:45.912938 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:05:45.912775    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ls844\" (UniqueName: \"kubernetes.io/projected/922d0edc-f817-477a-bc19-a40bacc930ba-kube-api-access-ls844\") pod \"insights-runtime-extractor-dz5rp\" (UID: \"922d0edc-f817-477a-bc19-a40bacc930ba\") " pod="openshift-insights/insights-runtime-extractor-dz5rp"
Jul 27 21:05:45.912938 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:05:45.912822    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/922d0edc-f817-477a-bc19-a40bacc930ba-crio-socket\") pod \"insights-runtime-extractor-dz5rp\" (UID: \"922d0edc-f817-477a-bc19-a40bacc930ba\") " pod="openshift-insights/insights-runtime-extractor-dz5rp"
Jul 27 21:05:45.912938 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:05:45.912849    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/922d0edc-f817-477a-bc19-a40bacc930ba-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-dz5rp\" (UID: \"922d0edc-f817-477a-bc19-a40bacc930ba\") " pod="openshift-insights/insights-runtime-extractor-dz5rp"
Jul 27 21:05:45.912938 ip-10-0-131-133 kubenswrapper[2559]: E0727 21:05:45.912849    2559 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 27 21:05:45.912938 ip-10-0-131-133 kubenswrapper[2559]: E0727 21:05:45.912922    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/922d0edc-f817-477a-bc19-a40bacc930ba-kube-rbac-proxy-cm podName:922d0edc-f817-477a-bc19-a40bacc930ba nodeName:}" failed. No retries permitted until 2026-07-27 21:05:46.412903264 +0000 UTC m=+74.992087252 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/922d0edc-f817-477a-bc19-a40bacc930ba-kube-rbac-proxy-cm") pod "insights-runtime-extractor-dz5rp" (UID: "922d0edc-f817-477a-bc19-a40bacc930ba") : configmap "kube-rbac-proxy" not found
Jul 27 21:05:45.913120 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:05:45.912983    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/922d0edc-f817-477a-bc19-a40bacc930ba-data-volume\") pod \"insights-runtime-extractor-dz5rp\" (UID: \"922d0edc-f817-477a-bc19-a40bacc930ba\") " pod="openshift-insights/insights-runtime-extractor-dz5rp"
Jul 27 21:05:45.913120 ip-10-0-131-133 kubenswrapper[2559]: E0727 21:05:45.913035    2559 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 27 21:05:45.913120 ip-10-0-131-133 kubenswrapper[2559]: E0727 21:05:45.913077    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/922d0edc-f817-477a-bc19-a40bacc930ba-insights-runtime-extractor-tls podName:922d0edc-f817-477a-bc19-a40bacc930ba nodeName:}" failed. No retries permitted until 2026-07-27 21:05:46.413064053 +0000 UTC m=+74.992248049 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/922d0edc-f817-477a-bc19-a40bacc930ba-insights-runtime-extractor-tls") pod "insights-runtime-extractor-dz5rp" (UID: "922d0edc-f817-477a-bc19-a40bacc930ba") : secret "insights-runtime-extractor-tls" not found
Jul 27 21:05:45.913324 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:05:45.913310    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/922d0edc-f817-477a-bc19-a40bacc930ba-data-volume\") pod \"insights-runtime-extractor-dz5rp\" (UID: \"922d0edc-f817-477a-bc19-a40bacc930ba\") " pod="openshift-insights/insights-runtime-extractor-dz5rp"
Jul 27 21:05:45.921373 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:05:45.921345    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ls844\" (UniqueName: \"kubernetes.io/projected/922d0edc-f817-477a-bc19-a40bacc930ba-kube-api-access-ls844\") pod \"insights-runtime-extractor-dz5rp\" (UID: \"922d0edc-f817-477a-bc19-a40bacc930ba\") " pod="openshift-insights/insights-runtime-extractor-dz5rp"
Jul 27 21:05:46.374387 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:05:46.374359    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-bjvbc_cba53c9a-64c0-4afa-8731-35521e196622/console-operator/1.log"
Jul 27 21:05:46.374802 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:05:46.374749    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-bjvbc_cba53c9a-64c0-4afa-8731-35521e196622/console-operator/0.log"
Jul 27 21:05:46.374802 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:05:46.374782    2559 generic.go:356] "Generic (PLEG): container finished" podID="cba53c9a-64c0-4afa-8731-35521e196622" containerID="6da6ab5bdc7f8baabce64058584a99a138af02fc6e1882f833f02c579d98d398" exitCode=255
Jul 27 21:05:46.374920 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:05:46.374874    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-584dcb6b54-bjvbc" event={"ID":"cba53c9a-64c0-4afa-8731-35521e196622","Type":"ContainerDied","Data":"6da6ab5bdc7f8baabce64058584a99a138af02fc6e1882f833f02c579d98d398"}
Jul 27 21:05:46.374954 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:05:46.374935    2559 scope.go:117] "RemoveContainer" containerID="6ebdb4b7121557c7eb23623ca12ce13522c62c98c6c87381080e0a509b6832df"
Jul 27 21:05:46.375174 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:05:46.375155    2559 scope.go:117] "RemoveContainer" containerID="6da6ab5bdc7f8baabce64058584a99a138af02fc6e1882f833f02c579d98d398"
Jul 27 21:05:46.375370 ip-10-0-131-133 kubenswrapper[2559]: E0727 21:05:46.375351    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-584dcb6b54-bjvbc_openshift-console-operator(cba53c9a-64c0-4afa-8731-35521e196622)\"" pod="openshift-console-operator/console-operator-584dcb6b54-bjvbc" podUID="cba53c9a-64c0-4afa-8731-35521e196622"
Jul 27 21:05:46.416261 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:05:46.416200    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/922d0edc-f817-477a-bc19-a40bacc930ba-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-dz5rp\" (UID: \"922d0edc-f817-477a-bc19-a40bacc930ba\") " pod="openshift-insights/insights-runtime-extractor-dz5rp"
Jul 27 21:05:46.416466 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:05:46.416339    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/922d0edc-f817-477a-bc19-a40bacc930ba-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-dz5rp\" (UID: \"922d0edc-f817-477a-bc19-a40bacc930ba\") " pod="openshift-insights/insights-runtime-extractor-dz5rp"
Jul 27 21:05:46.416466 ip-10-0-131-133 kubenswrapper[2559]: E0727 21:05:46.416418    2559 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 27 21:05:46.416466 ip-10-0-131-133 kubenswrapper[2559]: E0727 21:05:46.416426    2559 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 27 21:05:46.416622 ip-10-0-131-133 kubenswrapper[2559]: E0727 21:05:46.416483    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/922d0edc-f817-477a-bc19-a40bacc930ba-kube-rbac-proxy-cm podName:922d0edc-f817-477a-bc19-a40bacc930ba nodeName:}" failed. No retries permitted until 2026-07-27 21:05:47.41646359 +0000 UTC m=+75.995647600 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/922d0edc-f817-477a-bc19-a40bacc930ba-kube-rbac-proxy-cm") pod "insights-runtime-extractor-dz5rp" (UID: "922d0edc-f817-477a-bc19-a40bacc930ba") : configmap "kube-rbac-proxy" not found
Jul 27 21:05:46.416622 ip-10-0-131-133 kubenswrapper[2559]: E0727 21:05:46.416505    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/922d0edc-f817-477a-bc19-a40bacc930ba-insights-runtime-extractor-tls podName:922d0edc-f817-477a-bc19-a40bacc930ba nodeName:}" failed. No retries permitted until 2026-07-27 21:05:47.416495231 +0000 UTC m=+75.995679216 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/922d0edc-f817-477a-bc19-a40bacc930ba-insights-runtime-extractor-tls") pod "insights-runtime-extractor-dz5rp" (UID: "922d0edc-f817-477a-bc19-a40bacc930ba") : secret "insights-runtime-extractor-tls" not found
Jul 27 21:05:47.378446 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:05:47.378421    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-bjvbc_cba53c9a-64c0-4afa-8731-35521e196622/console-operator/1.log"
Jul 27 21:05:47.378821 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:05:47.378742    2559 scope.go:117] "RemoveContainer" containerID="6da6ab5bdc7f8baabce64058584a99a138af02fc6e1882f833f02c579d98d398"
Jul 27 21:05:47.378936 ip-10-0-131-133 kubenswrapper[2559]: E0727 21:05:47.378918    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-584dcb6b54-bjvbc_openshift-console-operator(cba53c9a-64c0-4afa-8731-35521e196622)\"" pod="openshift-console-operator/console-operator-584dcb6b54-bjvbc" podUID="cba53c9a-64c0-4afa-8731-35521e196622"
Jul 27 21:05:47.424206 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:05:47.424168    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/922d0edc-f817-477a-bc19-a40bacc930ba-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-dz5rp\" (UID: \"922d0edc-f817-477a-bc19-a40bacc930ba\") " pod="openshift-insights/insights-runtime-extractor-dz5rp"
Jul 27 21:05:47.424422 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:05:47.424299    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/922d0edc-f817-477a-bc19-a40bacc930ba-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-dz5rp\" (UID: \"922d0edc-f817-477a-bc19-a40bacc930ba\") " pod="openshift-insights/insights-runtime-extractor-dz5rp"
Jul 27 21:05:47.424422 ip-10-0-131-133 kubenswrapper[2559]: E0727 21:05:47.424350    2559 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 27 21:05:47.424422 ip-10-0-131-133 kubenswrapper[2559]: E0727 21:05:47.424418    2559 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 27 21:05:47.424584 ip-10-0-131-133 kubenswrapper[2559]: E0727 21:05:47.424424    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/922d0edc-f817-477a-bc19-a40bacc930ba-insights-runtime-extractor-tls podName:922d0edc-f817-477a-bc19-a40bacc930ba nodeName:}" failed. No retries permitted until 2026-07-27 21:05:49.424405162 +0000 UTC m=+78.003589152 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/922d0edc-f817-477a-bc19-a40bacc930ba-insights-runtime-extractor-tls") pod "insights-runtime-extractor-dz5rp" (UID: "922d0edc-f817-477a-bc19-a40bacc930ba") : secret "insights-runtime-extractor-tls" not found
Jul 27 21:05:47.424584 ip-10-0-131-133 kubenswrapper[2559]: E0727 21:05:47.424496    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/922d0edc-f817-477a-bc19-a40bacc930ba-kube-rbac-proxy-cm podName:922d0edc-f817-477a-bc19-a40bacc930ba nodeName:}" failed. No retries permitted until 2026-07-27 21:05:49.424479909 +0000 UTC m=+78.003663897 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/922d0edc-f817-477a-bc19-a40bacc930ba-kube-rbac-proxy-cm") pod "insights-runtime-extractor-dz5rp" (UID: "922d0edc-f817-477a-bc19-a40bacc930ba") : configmap "kube-rbac-proxy" not found
Jul 27 21:05:47.732549 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:05:47.732474    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-fvbmc_84a7d738-f486-43b0-ac43-3401891da8d1/dns-node-resolver/0.log"
Jul 27 21:05:48.533253 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:05:48.533208    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-qxzfs_3e4c5c3f-db85-4e88-a1d5-48e602b0018a/node-ca/0.log"
Jul 27 21:05:49.133563 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:05:49.133527    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-78ljg_cf59a2e6-f3c6-4549-9158-f047bfb7e764/serve-healthcheck-canary/0.log"
Jul 27 21:05:49.440782 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:05:49.440691    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/922d0edc-f817-477a-bc19-a40bacc930ba-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-dz5rp\" (UID: \"922d0edc-f817-477a-bc19-a40bacc930ba\") " pod="openshift-insights/insights-runtime-extractor-dz5rp"
Jul 27 21:05:49.440782 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:05:49.440765    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/922d0edc-f817-477a-bc19-a40bacc930ba-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-dz5rp\" (UID: \"922d0edc-f817-477a-bc19-a40bacc930ba\") " pod="openshift-insights/insights-runtime-extractor-dz5rp"
Jul 27 21:05:49.440972 ip-10-0-131-133 kubenswrapper[2559]: E0727 21:05:49.440821    2559 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 27 21:05:49.440972 ip-10-0-131-133 kubenswrapper[2559]: E0727 21:05:49.440869    2559 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 27 21:05:49.440972 ip-10-0-131-133 kubenswrapper[2559]: E0727 21:05:49.440889    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/922d0edc-f817-477a-bc19-a40bacc930ba-kube-rbac-proxy-cm podName:922d0edc-f817-477a-bc19-a40bacc930ba nodeName:}" failed. No retries permitted until 2026-07-27 21:05:53.440873915 +0000 UTC m=+82.020057900 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/922d0edc-f817-477a-bc19-a40bacc930ba-kube-rbac-proxy-cm") pod "insights-runtime-extractor-dz5rp" (UID: "922d0edc-f817-477a-bc19-a40bacc930ba") : configmap "kube-rbac-proxy" not found
Jul 27 21:05:49.440972 ip-10-0-131-133 kubenswrapper[2559]: E0727 21:05:49.440910    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/922d0edc-f817-477a-bc19-a40bacc930ba-insights-runtime-extractor-tls podName:922d0edc-f817-477a-bc19-a40bacc930ba nodeName:}" failed. No retries permitted until 2026-07-27 21:05:53.440897576 +0000 UTC m=+82.020081562 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/922d0edc-f817-477a-bc19-a40bacc930ba-insights-runtime-extractor-tls") pod "insights-runtime-extractor-dz5rp" (UID: "922d0edc-f817-477a-bc19-a40bacc930ba") : secret "insights-runtime-extractor-tls" not found
Jul 27 21:05:49.843762 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:05:49.843666    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/a05c7c31-10b9-4e3b-84a9-afe9cf929e07-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-548b489cb8-dxktc\" (UID: \"a05c7c31-10b9-4e3b-84a9-afe9cf929e07\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-dxktc"
Jul 27 21:05:49.844124 ip-10-0-131-133 kubenswrapper[2559]: E0727 21:05:49.843827    2559 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Jul 27 21:05:49.844124 ip-10-0-131-133 kubenswrapper[2559]: E0727 21:05:49.843891    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a05c7c31-10b9-4e3b-84a9-afe9cf929e07-cluster-monitoring-operator-tls podName:a05c7c31-10b9-4e3b-84a9-afe9cf929e07 nodeName:}" failed. No retries permitted until 2026-07-27 21:05:57.843877473 +0000 UTC m=+86.423061458 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/a05c7c31-10b9-4e3b-84a9-afe9cf929e07-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-548b489cb8-dxktc" (UID: "a05c7c31-10b9-4e3b-84a9-afe9cf929e07") : secret "cluster-monitoring-operator-tls" not found
Jul 27 21:05:52.337238 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:05:52.337181    2559 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-584dcb6b54-bjvbc"
Jul 27 21:05:52.337646 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:05:52.337262    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-584dcb6b54-bjvbc"
Jul 27 21:05:52.337646 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:05:52.337602    2559 scope.go:117] "RemoveContainer" containerID="6da6ab5bdc7f8baabce64058584a99a138af02fc6e1882f833f02c579d98d398"
Jul 27 21:05:52.337776 ip-10-0-131-133 kubenswrapper[2559]: E0727 21:05:52.337756    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-584dcb6b54-bjvbc_openshift-console-operator(cba53c9a-64c0-4afa-8731-35521e196622)\"" pod="openshift-console-operator/console-operator-584dcb6b54-bjvbc" podUID="cba53c9a-64c0-4afa-8731-35521e196622"
Jul 27 21:05:53.473026 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:05:53.472889    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/922d0edc-f817-477a-bc19-a40bacc930ba-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-dz5rp\" (UID: \"922d0edc-f817-477a-bc19-a40bacc930ba\") " pod="openshift-insights/insights-runtime-extractor-dz5rp"
Jul 27 21:05:53.473026 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:05:53.472983    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/922d0edc-f817-477a-bc19-a40bacc930ba-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-dz5rp\" (UID: \"922d0edc-f817-477a-bc19-a40bacc930ba\") " pod="openshift-insights/insights-runtime-extractor-dz5rp"
Jul 27 21:05:53.474183 ip-10-0-131-133 kubenswrapper[2559]: E0727 21:05:53.473566    2559 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 27 21:05:53.474183 ip-10-0-131-133 kubenswrapper[2559]: E0727 21:05:53.473780    2559 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 27 21:05:53.474183 ip-10-0-131-133 kubenswrapper[2559]: E0727 21:05:53.474014    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/922d0edc-f817-477a-bc19-a40bacc930ba-insights-runtime-extractor-tls podName:922d0edc-f817-477a-bc19-a40bacc930ba nodeName:}" failed. No retries permitted until 2026-07-27 21:06:01.473981658 +0000 UTC m=+90.053165643 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/922d0edc-f817-477a-bc19-a40bacc930ba-insights-runtime-extractor-tls") pod "insights-runtime-extractor-dz5rp" (UID: "922d0edc-f817-477a-bc19-a40bacc930ba") : secret "insights-runtime-extractor-tls" not found
Jul 27 21:05:53.476961 ip-10-0-131-133 kubenswrapper[2559]: E0727 21:05:53.474612    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/922d0edc-f817-477a-bc19-a40bacc930ba-kube-rbac-proxy-cm podName:922d0edc-f817-477a-bc19-a40bacc930ba nodeName:}" failed. No retries permitted until 2026-07-27 21:06:01.474567726 +0000 UTC m=+90.053751739 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/922d0edc-f817-477a-bc19-a40bacc930ba-kube-rbac-proxy-cm") pod "insights-runtime-extractor-dz5rp" (UID: "922d0edc-f817-477a-bc19-a40bacc930ba") : configmap "kube-rbac-proxy" not found
Jul 27 21:05:57.910502 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:05:57.910444    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/a05c7c31-10b9-4e3b-84a9-afe9cf929e07-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-548b489cb8-dxktc\" (UID: \"a05c7c31-10b9-4e3b-84a9-afe9cf929e07\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-dxktc"
Jul 27 21:05:57.911029 ip-10-0-131-133 kubenswrapper[2559]: E0727 21:05:57.910628    2559 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Jul 27 21:05:57.911029 ip-10-0-131-133 kubenswrapper[2559]: E0727 21:05:57.910718    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a05c7c31-10b9-4e3b-84a9-afe9cf929e07-cluster-monitoring-operator-tls podName:a05c7c31-10b9-4e3b-84a9-afe9cf929e07 nodeName:}" failed. No retries permitted until 2026-07-27 21:06:13.910696283 +0000 UTC m=+102.489880293 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/a05c7c31-10b9-4e3b-84a9-afe9cf929e07-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-548b489cb8-dxktc" (UID: "a05c7c31-10b9-4e3b-84a9-afe9cf929e07") : secret "cluster-monitoring-operator-tls" not found
Jul 27 21:06:01.541118 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:06:01.541083    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/922d0edc-f817-477a-bc19-a40bacc930ba-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-dz5rp\" (UID: \"922d0edc-f817-477a-bc19-a40bacc930ba\") " pod="openshift-insights/insights-runtime-extractor-dz5rp"
Jul 27 21:06:01.541527 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:06:01.541152    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/922d0edc-f817-477a-bc19-a40bacc930ba-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-dz5rp\" (UID: \"922d0edc-f817-477a-bc19-a40bacc930ba\") " pod="openshift-insights/insights-runtime-extractor-dz5rp"
Jul 27 21:06:01.541663 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:06:01.541646    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/922d0edc-f817-477a-bc19-a40bacc930ba-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-dz5rp\" (UID: \"922d0edc-f817-477a-bc19-a40bacc930ba\") " pod="openshift-insights/insights-runtime-extractor-dz5rp"
Jul 27 21:06:01.543543 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:06:01.543514    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/922d0edc-f817-477a-bc19-a40bacc930ba-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-dz5rp\" (UID: \"922d0edc-f817-477a-bc19-a40bacc930ba\") " pod="openshift-insights/insights-runtime-extractor-dz5rp"
Jul 27 21:06:01.671184 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:06:01.671141    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-dz5rp"
Jul 27 21:06:01.803578 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:06:01.803502    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-dz5rp"]
Jul 27 21:06:01.806692 ip-10-0-131-133 kubenswrapper[2559]: W0727 21:06:01.806656    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod922d0edc_f817_477a_bc19_a40bacc930ba.slice/crio-bcdfe64aab14d3deb2ff26bb07bdd129087c98ef1ed0702509085de443b2dd07 WatchSource:0}: Error finding container bcdfe64aab14d3deb2ff26bb07bdd129087c98ef1ed0702509085de443b2dd07: Status 404 returned error can't find the container with id bcdfe64aab14d3deb2ff26bb07bdd129087c98ef1ed0702509085de443b2dd07
Jul 27 21:06:02.418766 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:06:02.418703    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-dz5rp" event={"ID":"922d0edc-f817-477a-bc19-a40bacc930ba","Type":"ContainerStarted","Data":"bcdfe64aab14d3deb2ff26bb07bdd129087c98ef1ed0702509085de443b2dd07"}
Jul 27 21:06:04.977675 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:06:04.977644    2559 scope.go:117] "RemoveContainer" containerID="6da6ab5bdc7f8baabce64058584a99a138af02fc6e1882f833f02c579d98d398"
Jul 27 21:06:07.436776 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:06:07.436735    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-dz5rp" event={"ID":"922d0edc-f817-477a-bc19-a40bacc930ba","Type":"ContainerStarted","Data":"1f1d348d979310a7ec32b60525a4f14d0fe8f3ad0bd8dc3da77c86486650598a"}
Jul 27 21:06:07.439944 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:06:07.439915    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-bjvbc_cba53c9a-64c0-4afa-8731-35521e196622/console-operator/1.log"
Jul 27 21:06:07.440111 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:06:07.439978    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-584dcb6b54-bjvbc" event={"ID":"cba53c9a-64c0-4afa-8731-35521e196622","Type":"ContainerStarted","Data":"351e5a7f6df3f78c8d785496cf46b2ec0555c65134d96919ec9f064228f87d86"}
Jul 27 21:06:07.440372 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:06:07.440351    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-584dcb6b54-bjvbc"
Jul 27 21:06:07.446574 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:06:07.446533    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console-operator/console-operator-584dcb6b54-bjvbc"
Jul 27 21:06:07.506166 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:06:07.506107    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console-operator/console-operator-584dcb6b54-bjvbc" podStartSLOduration=24.260915909 podStartE2EDuration="26.506087622s" totalImagesPullingTime="2.245171713s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 21:05:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 21:06:07.472701946 +0000 UTC m=+96.051886002" watchObservedRunningTime="2026-07-27 21:06:07.506087622 +0000 UTC m=+96.085271631"
Jul 27 21:06:10.451998 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:06:10.451971    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-dz5rp" event={"ID":"922d0edc-f817-477a-bc19-a40bacc930ba","Type":"ContainerStarted","Data":"c7d1b0e99743d9f2e57566ecf68dfb715752ca016cf8dbd1ed4f56ef3e00197f"}
Jul 27 21:06:11.734111 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:06:11.734070    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ac10d2c6-12bd-4f89-8d1e-c931206a2e92-metrics-tls\") pod \"dns-default-9kfdx\" (UID: \"ac10d2c6-12bd-4f89-8d1e-c931206a2e92\") " pod="openshift-dns/dns-default-9kfdx"
Jul 27 21:06:11.734111 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:06:11.734125    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/097262d9-cf78-4d89-98cc-cf92d23a4f04-registry-tls\") pod \"image-registry-5b565b454b-2kr9d\" (UID: \"097262d9-cf78-4d89-98cc-cf92d23a4f04\") " pod="openshift-image-registry/image-registry-5b565b454b-2kr9d"
Jul 27 21:06:11.736757 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:06:11.736728    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ac10d2c6-12bd-4f89-8d1e-c931206a2e92-metrics-tls\") pod \"dns-default-9kfdx\" (UID: \"ac10d2c6-12bd-4f89-8d1e-c931206a2e92\") " pod="openshift-dns/dns-default-9kfdx"
Jul 27 21:06:11.736965 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:06:11.736939    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/097262d9-cf78-4d89-98cc-cf92d23a4f04-registry-tls\") pod \"image-registry-5b565b454b-2kr9d\" (UID: \"097262d9-cf78-4d89-98cc-cf92d23a4f04\") " pod="openshift-image-registry/image-registry-5b565b454b-2kr9d"
Jul 27 21:06:11.963188 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:06:11.963154    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-kdvbm\""
Jul 27 21:06:11.970977 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:06:11.970928    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5b565b454b-2kr9d"
Jul 27 21:06:11.989303 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:06:11.989229    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-6vzp2\""
Jul 27 21:06:11.997526 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:06:11.997497    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-9kfdx"
Jul 27 21:06:12.120558 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:06:12.120523    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-5b565b454b-2kr9d"]
Jul 27 21:06:13.766135 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:06:13.766091    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-57469ddc4-sbpnb"]
Jul 27 21:06:13.901481 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:06:13.901449    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-console/networking-console-plugin-74c9f884b7-bd6vk"]
Jul 27 21:06:13.901684 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:06:13.901546    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-57469ddc4-sbpnb"
Jul 27 21:06:13.903954 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:06:13.903931    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\""
Jul 27 21:06:13.904099 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:06:13.903933    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\""
Jul 27 21:06:13.904099 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:06:13.903963    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"work-manager-hub-kubeconfig\""
Jul 27 21:06:13.904292 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:06:13.904277    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\""
Jul 27 21:06:13.925855 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:06:13.925817    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-57469ddc4-sbpnb"]
Jul 27 21:06:13.925855 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:06:13.925857    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-74c9f884b7-bd6vk"]
Jul 27 21:06:13.926063 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:06:13.925940    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-7c97dc4555-7x89h"]
Jul 27 21:06:13.926063 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:06:13.925942    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-74c9f884b7-bd6vk"
Jul 27 21:06:13.929202 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:06:13.929179    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-ctln6\""
Jul 27 21:06:13.929558 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:06:13.929531    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\""
Jul 27 21:06:13.929634 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:06:13.929557    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\""
Jul 27 21:06:13.947898 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:06:13.947867    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-7c97dc4555-7x89h"]
Jul 27 21:06:13.948035 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:06:13.947964    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-7c97dc4555-7x89h"
Jul 27 21:06:13.950113 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:06:13.950089    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"default-dockercfg-hnwwj\""
Jul 27 21:06:13.950113 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:06:13.950113    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Jul 27 21:06:13.950429 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:06:13.950414    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Jul 27 21:06:13.955617 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:06:13.955589    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/a05c7c31-10b9-4e3b-84a9-afe9cf929e07-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-548b489cb8-dxktc\" (UID: \"a05c7c31-10b9-4e3b-84a9-afe9cf929e07\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-dxktc"
Jul 27 21:06:13.957995 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:06:13.957970    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/a05c7c31-10b9-4e3b-84a9-afe9cf929e07-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-548b489cb8-dxktc\" (UID: \"a05c7c31-10b9-4e3b-84a9-afe9cf929e07\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-dxktc"
Jul 27 21:06:13.960458 ip-10-0-131-133 kubenswrapper[2559]: W0727 21:06:13.960433    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod097262d9_cf78_4d89_98cc_cf92d23a4f04.slice/crio-b00495faa9d4af48eddb7c7598d8d1c16d98b4d05ebdef3c65b70e113cf88bc7 WatchSource:0}: Error finding container b00495faa9d4af48eddb7c7598d8d1c16d98b4d05ebdef3c65b70e113cf88bc7: Status 404 returned error can't find the container with id b00495faa9d4af48eddb7c7598d8d1c16d98b4d05ebdef3c65b70e113cf88bc7
Jul 27 21:06:14.056041 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:06:14.056017    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ndktz\" (UniqueName: \"kubernetes.io/projected/2fc45846-b5c2-44b8-b81e-7debda79f2f6-kube-api-access-ndktz\") pod \"klusterlet-addon-workmgr-57469ddc4-sbpnb\" (UID: \"2fc45846-b5c2-44b8-b81e-7debda79f2f6\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-57469ddc4-sbpnb"
Jul 27 21:06:14.056172 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:06:14.056050    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2fc45846-b5c2-44b8-b81e-7debda79f2f6-tmp\") pod \"klusterlet-addon-workmgr-57469ddc4-sbpnb\" (UID: \"2fc45846-b5c2-44b8-b81e-7debda79f2f6\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-57469ddc4-sbpnb"
Jul 27 21:06:14.056172 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:06:14.056090    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/2fc45846-b5c2-44b8-b81e-7debda79f2f6-klusterlet-config\") pod \"klusterlet-addon-workmgr-57469ddc4-sbpnb\" (UID: \"2fc45846-b5c2-44b8-b81e-7debda79f2f6\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-57469ddc4-sbpnb"
Jul 27 21:06:14.056302 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:06:14.056206    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/d8270118-51fb-45fb-99ed-6f3aa670345c-networking-console-plugin-cert\") pod \"networking-console-plugin-74c9f884b7-bd6vk\" (UID: \"d8270118-51fb-45fb-99ed-6f3aa670345c\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-bd6vk"
Jul 27 21:06:14.056302 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:06:14.056282    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b86xv\" (UniqueName: \"kubernetes.io/projected/2ded1152-f764-48fc-b45f-744519e49e62-kube-api-access-b86xv\") pod \"downloads-7c97dc4555-7x89h\" (UID: \"2ded1152-f764-48fc-b45f-744519e49e62\") " pod="openshift-console/downloads-7c97dc4555-7x89h"
Jul 27 21:06:14.056394 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:06:14.056320    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/d8270118-51fb-45fb-99ed-6f3aa670345c-nginx-conf\") pod \"networking-console-plugin-74c9f884b7-bd6vk\" (UID: \"d8270118-51fb-45fb-99ed-6f3aa670345c\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-bd6vk"
Jul 27 21:06:14.092665 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:06:14.092634    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-9kfdx"]
Jul 27 21:06:14.096063 ip-10-0-131-133 kubenswrapper[2559]: W0727 21:06:14.096029    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podac10d2c6_12bd_4f89_8d1e_c931206a2e92.slice/crio-c274950c9a17372f5d3ec01c415bd8785641451d170b8933e914155c08e67b6c WatchSource:0}: Error finding container c274950c9a17372f5d3ec01c415bd8785641451d170b8933e914155c08e67b6c: Status 404 returned error can't find the container with id c274950c9a17372f5d3ec01c415bd8785641451d170b8933e914155c08e67b6c
Jul 27 21:06:14.157470 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:06:14.157423    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/d8270118-51fb-45fb-99ed-6f3aa670345c-networking-console-plugin-cert\") pod \"networking-console-plugin-74c9f884b7-bd6vk\" (UID: \"d8270118-51fb-45fb-99ed-6f3aa670345c\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-bd6vk"
Jul 27 21:06:14.157470 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:06:14.157470    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b86xv\" (UniqueName: \"kubernetes.io/projected/2ded1152-f764-48fc-b45f-744519e49e62-kube-api-access-b86xv\") pod \"downloads-7c97dc4555-7x89h\" (UID: \"2ded1152-f764-48fc-b45f-744519e49e62\") " pod="openshift-console/downloads-7c97dc4555-7x89h"
Jul 27 21:06:14.157666 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:06:14.157492    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/d8270118-51fb-45fb-99ed-6f3aa670345c-nginx-conf\") pod \"networking-console-plugin-74c9f884b7-bd6vk\" (UID: \"d8270118-51fb-45fb-99ed-6f3aa670345c\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-bd6vk"
Jul 27 21:06:14.157666 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:06:14.157512    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ndktz\" (UniqueName: \"kubernetes.io/projected/2fc45846-b5c2-44b8-b81e-7debda79f2f6-kube-api-access-ndktz\") pod \"klusterlet-addon-workmgr-57469ddc4-sbpnb\" (UID: \"2fc45846-b5c2-44b8-b81e-7debda79f2f6\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-57469ddc4-sbpnb"
Jul 27 21:06:14.157666 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:06:14.157539    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2fc45846-b5c2-44b8-b81e-7debda79f2f6-tmp\") pod \"klusterlet-addon-workmgr-57469ddc4-sbpnb\" (UID: \"2fc45846-b5c2-44b8-b81e-7debda79f2f6\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-57469ddc4-sbpnb"
Jul 27 21:06:14.157666 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:06:14.157589    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/2fc45846-b5c2-44b8-b81e-7debda79f2f6-klusterlet-config\") pod \"klusterlet-addon-workmgr-57469ddc4-sbpnb\" (UID: \"2fc45846-b5c2-44b8-b81e-7debda79f2f6\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-57469ddc4-sbpnb"
Jul 27 21:06:14.158035 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:06:14.158004    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2fc45846-b5c2-44b8-b81e-7debda79f2f6-tmp\") pod \"klusterlet-addon-workmgr-57469ddc4-sbpnb\" (UID: \"2fc45846-b5c2-44b8-b81e-7debda79f2f6\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-57469ddc4-sbpnb"
Jul 27 21:06:14.158179 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:06:14.158155    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/d8270118-51fb-45fb-99ed-6f3aa670345c-nginx-conf\") pod \"networking-console-plugin-74c9f884b7-bd6vk\" (UID: \"d8270118-51fb-45fb-99ed-6f3aa670345c\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-bd6vk"
Jul 27 21:06:14.159929 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:06:14.159910    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/d8270118-51fb-45fb-99ed-6f3aa670345c-networking-console-plugin-cert\") pod \"networking-console-plugin-74c9f884b7-bd6vk\" (UID: \"d8270118-51fb-45fb-99ed-6f3aa670345c\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-bd6vk"
Jul 27 21:06:14.160011 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:06:14.159955    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/2fc45846-b5c2-44b8-b81e-7debda79f2f6-klusterlet-config\") pod \"klusterlet-addon-workmgr-57469ddc4-sbpnb\" (UID: \"2fc45846-b5c2-44b8-b81e-7debda79f2f6\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-57469ddc4-sbpnb"
Jul 27 21:06:14.168048 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:06:14.168023    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b86xv\" (UniqueName: \"kubernetes.io/projected/2ded1152-f764-48fc-b45f-744519e49e62-kube-api-access-b86xv\") pod \"downloads-7c97dc4555-7x89h\" (UID: \"2ded1152-f764-48fc-b45f-744519e49e62\") " pod="openshift-console/downloads-7c97dc4555-7x89h"
Jul 27 21:06:14.176961 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:06:14.176933    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ndktz\" (UniqueName: \"kubernetes.io/projected/2fc45846-b5c2-44b8-b81e-7debda79f2f6-kube-api-access-ndktz\") pod \"klusterlet-addon-workmgr-57469ddc4-sbpnb\" (UID: \"2fc45846-b5c2-44b8-b81e-7debda79f2f6\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-57469ddc4-sbpnb"
Jul 27 21:06:14.211818 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:06:14.211789    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-57469ddc4-sbpnb"
Jul 27 21:06:14.228344 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:06:14.228319    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-dxktc"
Jul 27 21:06:14.234474 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:06:14.234448    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-74c9f884b7-bd6vk"
Jul 27 21:06:14.255903 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:06:14.255881    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-7c97dc4555-7x89h"
Jul 27 21:06:14.370645 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:06:14.368899    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-hmjv6"
Jul 27 21:06:14.469740 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:06:14.469686    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-dz5rp" event={"ID":"922d0edc-f817-477a-bc19-a40bacc930ba","Type":"ContainerStarted","Data":"95031083d2915f0eadc7abc799dba62b81b3802308a8e63194e83ec10ea9e33c"}
Jul 27 21:06:14.471353 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:06:14.471292    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-548b489cb8-dxktc"]
Jul 27 21:06:14.471946 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:06:14.471712    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5b565b454b-2kr9d" event={"ID":"097262d9-cf78-4d89-98cc-cf92d23a4f04","Type":"ContainerStarted","Data":"8b1d854e5710d1523faad73121e9b3b88250fc3dabcfc04c09cb2d56430fd0ed"}
Jul 27 21:06:14.471946 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:06:14.471761    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5b565b454b-2kr9d" event={"ID":"097262d9-cf78-4d89-98cc-cf92d23a4f04","Type":"ContainerStarted","Data":"b00495faa9d4af48eddb7c7598d8d1c16d98b4d05ebdef3c65b70e113cf88bc7"}
Jul 27 21:06:14.471946 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:06:14.471826    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-5b565b454b-2kr9d"
Jul 27 21:06:14.473322 ip-10-0-131-133 kubenswrapper[2559]: W0727 21:06:14.473286    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda05c7c31_10b9_4e3b_84a9_afe9cf929e07.slice/crio-6d8e544d2ebe3a2f3179d456d84e0b9bd4e3e15a2a6a75991b7ad78ea7ac4cc2 WatchSource:0}: Error finding container 6d8e544d2ebe3a2f3179d456d84e0b9bd4e3e15a2a6a75991b7ad78ea7ac4cc2: Status 404 returned error can't find the container with id 6d8e544d2ebe3a2f3179d456d84e0b9bd4e3e15a2a6a75991b7ad78ea7ac4cc2
Jul 27 21:06:14.473833 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:06:14.473806    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-9kfdx" event={"ID":"ac10d2c6-12bd-4f89-8d1e-c931206a2e92","Type":"ContainerStarted","Data":"c274950c9a17372f5d3ec01c415bd8785641451d170b8933e914155c08e67b6c"}
Jul 27 21:06:14.499078 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:06:14.499029    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-dz5rp" podStartSLOduration=17.142491968 podStartE2EDuration="29.499012661s" totalImagesPullingTime="12.356520693s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-27 21:05:45 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 21:06:14.498571293 +0000 UTC m=+103.077755300" watchObservedRunningTime="2026-07-27 21:06:14.499012661 +0000 UTC m=+103.078196658"
Jul 27 21:06:14.510146 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:06:14.510106    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-74c9f884b7-bd6vk"]
Jul 27 21:06:14.513581 ip-10-0-131-133 kubenswrapper[2559]: W0727 21:06:14.513558    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd8270118_51fb_45fb_99ed_6f3aa670345c.slice/crio-b215431cf7e1349bd550a141ab0f784d7eae1fa63ef21d9607fe15b6df59b2b5 WatchSource:0}: Error finding container b215431cf7e1349bd550a141ab0f784d7eae1fa63ef21d9607fe15b6df59b2b5: Status 404 returned error can't find the container with id b215431cf7e1349bd550a141ab0f784d7eae1fa63ef21d9607fe15b6df59b2b5
Jul 27 21:06:14.548772 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:06:14.548679    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-5b565b454b-2kr9d" podStartSLOduration=101.548660494 podStartE2EDuration="1m41.548660494s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 21:04:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 21:06:14.546797581 +0000 UTC m=+103.125981586" watchObservedRunningTime="2026-07-27 21:06:14.548660494 +0000 UTC m=+103.127844501"
Jul 27 21:06:14.686546 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:06:14.686502    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-7c97dc4555-7x89h"]
Jul 27 21:06:14.687962 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:06:14.687935    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-57469ddc4-sbpnb"]
Jul 27 21:06:14.690234 ip-10-0-131-133 kubenswrapper[2559]: W0727 21:06:14.690184    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2ded1152_f764_48fc_b45f_744519e49e62.slice/crio-d4e1a32bdb38c38595242af1dd4bb8dee93f16de41186b055aebe5ab8a68351c WatchSource:0}: Error finding container d4e1a32bdb38c38595242af1dd4bb8dee93f16de41186b055aebe5ab8a68351c: Status 404 returned error can't find the container with id d4e1a32bdb38c38595242af1dd4bb8dee93f16de41186b055aebe5ab8a68351c
Jul 27 21:06:14.691133 ip-10-0-131-133 kubenswrapper[2559]: W0727 21:06:14.691106    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2fc45846_b5c2_44b8_b81e_7debda79f2f6.slice/crio-3a63188fee4ee7501a38418e2eff2c8184b3a43bb4ee394e3bbfd6c93909e35d WatchSource:0}: Error finding container 3a63188fee4ee7501a38418e2eff2c8184b3a43bb4ee394e3bbfd6c93909e35d: Status 404 returned error can't find the container with id 3a63188fee4ee7501a38418e2eff2c8184b3a43bb4ee394e3bbfd6c93909e35d
Jul 27 21:06:15.481294 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:06:15.481148    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-7c97dc4555-7x89h" event={"ID":"2ded1152-f764-48fc-b45f-744519e49e62","Type":"ContainerStarted","Data":"d4e1a32bdb38c38595242af1dd4bb8dee93f16de41186b055aebe5ab8a68351c"}
Jul 27 21:06:15.483100 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:06:15.483029    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-dxktc" event={"ID":"a05c7c31-10b9-4e3b-84a9-afe9cf929e07","Type":"ContainerStarted","Data":"6d8e544d2ebe3a2f3179d456d84e0b9bd4e3e15a2a6a75991b7ad78ea7ac4cc2"}
Jul 27 21:06:15.484516 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:06:15.484449    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-57469ddc4-sbpnb" event={"ID":"2fc45846-b5c2-44b8-b81e-7debda79f2f6","Type":"ContainerStarted","Data":"3a63188fee4ee7501a38418e2eff2c8184b3a43bb4ee394e3bbfd6c93909e35d"}
Jul 27 21:06:15.487015 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:06:15.486521    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-74c9f884b7-bd6vk" event={"ID":"d8270118-51fb-45fb-99ed-6f3aa670345c","Type":"ContainerStarted","Data":"b215431cf7e1349bd550a141ab0f784d7eae1fa63ef21d9607fe15b6df59b2b5"}
Jul 27 21:06:18.324121 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:06:18.324067    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-669984f9db-74b9w"]
Jul 27 21:06:18.347782 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:06:18.347748    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-669984f9db-74b9w"]
Jul 27 21:06:18.347958 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:06:18.347874    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-669984f9db-74b9w"
Jul 27 21:06:18.352474 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:06:18.352032    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Jul 27 21:06:18.352474 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:06:18.352303    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Jul 27 21:06:18.352474 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:06:18.352348    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-pzcm6\""
Jul 27 21:06:18.352710 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:06:18.352593    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Jul 27 21:06:18.353137 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:06:18.352861    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Jul 27 21:06:18.353137 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:06:18.353078    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Jul 27 21:06:18.400103 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:06:18.400011    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/f93aded2-05e8-4d10-8485-630a97a87fcf-console-config\") pod \"console-669984f9db-74b9w\" (UID: \"f93aded2-05e8-4d10-8485-630a97a87fcf\") " pod="openshift-console/console-669984f9db-74b9w"
Jul 27 21:06:18.400103 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:06:18.400063    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/f93aded2-05e8-4d10-8485-630a97a87fcf-console-oauth-config\") pod \"console-669984f9db-74b9w\" (UID: \"f93aded2-05e8-4d10-8485-630a97a87fcf\") " pod="openshift-console/console-669984f9db-74b9w"
Jul 27 21:06:18.400103 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:06:18.400095    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4lh67\" (UniqueName: \"kubernetes.io/projected/f93aded2-05e8-4d10-8485-630a97a87fcf-kube-api-access-4lh67\") pod \"console-669984f9db-74b9w\" (UID: \"f93aded2-05e8-4d10-8485-630a97a87fcf\") " pod="openshift-console/console-669984f9db-74b9w"
Jul 27 21:06:18.400425 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:06:18.400146    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/f93aded2-05e8-4d10-8485-630a97a87fcf-console-serving-cert\") pod \"console-669984f9db-74b9w\" (UID: \"f93aded2-05e8-4d10-8485-630a97a87fcf\") " pod="openshift-console/console-669984f9db-74b9w"
Jul 27 21:06:18.400425 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:06:18.400177    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/f93aded2-05e8-4d10-8485-630a97a87fcf-oauth-serving-cert\") pod \"console-669984f9db-74b9w\" (UID: \"f93aded2-05e8-4d10-8485-630a97a87fcf\") " pod="openshift-console/console-669984f9db-74b9w"
Jul 27 21:06:18.400425 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:06:18.400207    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/f93aded2-05e8-4d10-8485-630a97a87fcf-service-ca\") pod \"console-669984f9db-74b9w\" (UID: \"f93aded2-05e8-4d10-8485-630a97a87fcf\") " pod="openshift-console/console-669984f9db-74b9w"
Jul 27 21:06:18.500668 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:06:18.500632    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/f93aded2-05e8-4d10-8485-630a97a87fcf-console-config\") pod \"console-669984f9db-74b9w\" (UID: \"f93aded2-05e8-4d10-8485-630a97a87fcf\") " pod="openshift-console/console-669984f9db-74b9w"
Jul 27 21:06:18.501764 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:06:18.500674    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/f93aded2-05e8-4d10-8485-630a97a87fcf-console-oauth-config\") pod \"console-669984f9db-74b9w\" (UID: \"f93aded2-05e8-4d10-8485-630a97a87fcf\") " pod="openshift-console/console-669984f9db-74b9w"
Jul 27 21:06:18.501764 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:06:18.500704    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4lh67\" (UniqueName: \"kubernetes.io/projected/f93aded2-05e8-4d10-8485-630a97a87fcf-kube-api-access-4lh67\") pod \"console-669984f9db-74b9w\" (UID: \"f93aded2-05e8-4d10-8485-630a97a87fcf\") " pod="openshift-console/console-669984f9db-74b9w"
Jul 27 21:06:18.501764 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:06:18.500752    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/f93aded2-05e8-4d10-8485-630a97a87fcf-console-serving-cert\") pod \"console-669984f9db-74b9w\" (UID: \"f93aded2-05e8-4d10-8485-630a97a87fcf\") " pod="openshift-console/console-669984f9db-74b9w"
Jul 27 21:06:18.501764 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:06:18.500786    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/f93aded2-05e8-4d10-8485-630a97a87fcf-oauth-serving-cert\") pod \"console-669984f9db-74b9w\" (UID: \"f93aded2-05e8-4d10-8485-630a97a87fcf\") " pod="openshift-console/console-669984f9db-74b9w"
Jul 27 21:06:18.501764 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:06:18.500814    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/f93aded2-05e8-4d10-8485-630a97a87fcf-service-ca\") pod \"console-669984f9db-74b9w\" (UID: \"f93aded2-05e8-4d10-8485-630a97a87fcf\") " pod="openshift-console/console-669984f9db-74b9w"
Jul 27 21:06:18.501955 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:06:18.501830    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/f93aded2-05e8-4d10-8485-630a97a87fcf-service-ca\") pod \"console-669984f9db-74b9w\" (UID: \"f93aded2-05e8-4d10-8485-630a97a87fcf\") " pod="openshift-console/console-669984f9db-74b9w"
Jul 27 21:06:18.502188 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:06:18.502094    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-9kfdx" event={"ID":"ac10d2c6-12bd-4f89-8d1e-c931206a2e92","Type":"ContainerStarted","Data":"a3a234d798d8060d1cc5e7b811ee8d90c2f859c1cb43bce13f56cf4b6cd8390c"}
Jul 27 21:06:18.502188 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:06:18.502155    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-9kfdx" event={"ID":"ac10d2c6-12bd-4f89-8d1e-c931206a2e92","Type":"ContainerStarted","Data":"a8d5fc1ccd52a419106a1dfb225e027c758491d3e149798e9418979b428087f6"}
Jul 27 21:06:18.502399 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:06:18.502356    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/f93aded2-05e8-4d10-8485-630a97a87fcf-oauth-serving-cert\") pod \"console-669984f9db-74b9w\" (UID: \"f93aded2-05e8-4d10-8485-630a97a87fcf\") " pod="openshift-console/console-669984f9db-74b9w"
Jul 27 21:06:18.502457 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:06:18.502431    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-9kfdx"
Jul 27 21:06:18.502820 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:06:18.502783    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/f93aded2-05e8-4d10-8485-630a97a87fcf-console-config\") pod \"console-669984f9db-74b9w\" (UID: \"f93aded2-05e8-4d10-8485-630a97a87fcf\") " pod="openshift-console/console-669984f9db-74b9w"
Jul 27 21:06:18.505279 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:06:18.504436    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-74c9f884b7-bd6vk" event={"ID":"d8270118-51fb-45fb-99ed-6f3aa670345c","Type":"ContainerStarted","Data":"b9cbf977e124ca308df588339c24810a43a5bccf359bd79516df9d410a0d4182"}
Jul 27 21:06:18.509956 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:06:18.509930    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/f93aded2-05e8-4d10-8485-630a97a87fcf-console-serving-cert\") pod \"console-669984f9db-74b9w\" (UID: \"f93aded2-05e8-4d10-8485-630a97a87fcf\") " pod="openshift-console/console-669984f9db-74b9w"
Jul 27 21:06:18.515855 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:06:18.515819    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/f93aded2-05e8-4d10-8485-630a97a87fcf-console-oauth-config\") pod \"console-669984f9db-74b9w\" (UID: \"f93aded2-05e8-4d10-8485-630a97a87fcf\") " pod="openshift-console/console-669984f9db-74b9w"
Jul 27 21:06:18.517046 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:06:18.516778    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-dxktc" event={"ID":"a05c7c31-10b9-4e3b-84a9-afe9cf929e07","Type":"ContainerStarted","Data":"3d1593fa3ead8e680f099702394563e7e10aef90cbbe34298e164d985a00d8af"}
Jul 27 21:06:18.518442 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:06:18.518415    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4lh67\" (UniqueName: \"kubernetes.io/projected/f93aded2-05e8-4d10-8485-630a97a87fcf-kube-api-access-4lh67\") pod \"console-669984f9db-74b9w\" (UID: \"f93aded2-05e8-4d10-8485-630a97a87fcf\") " pod="openshift-console/console-669984f9db-74b9w"
Jul 27 21:06:18.527781 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:06:18.527584    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-9kfdx" podStartSLOduration=67.797084858 podStartE2EDuration="1m11.527562064s" totalImagesPullingTime="3.730477206s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 21:05:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 21:06:18.526299198 +0000 UTC m=+107.105483244" watchObservedRunningTime="2026-07-27 21:06:18.527562064 +0000 UTC m=+107.106746085"
Jul 27 21:06:18.543314 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:06:18.543262    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-dxktc" podStartSLOduration=33.180532322 podStartE2EDuration="36.543246581s" totalImagesPullingTime="3.362714259s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 21:05:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 21:06:18.541881022 +0000 UTC m=+107.121065031" watchObservedRunningTime="2026-07-27 21:06:18.543246581 +0000 UTC m=+107.122430590"
Jul 27 21:06:18.663182 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:06:18.662800    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-669984f9db-74b9w"
Jul 27 21:06:18.854241 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:06:18.853899    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-console/networking-console-plugin-74c9f884b7-bd6vk" podStartSLOduration=2.538766373 podStartE2EDuration="5.853871881s" totalImagesPullingTime="3.315105508s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 21:06:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 21:06:18.584675892 +0000 UTC m=+107.163859900" watchObservedRunningTime="2026-07-27 21:06:18.853871881 +0000 UTC m=+107.433055891"
Jul 27 21:06:18.855051 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:06:18.854919    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-669984f9db-74b9w"]
Jul 27 21:06:20.947735 ip-10-0-131-133 kubenswrapper[2559]: W0727 21:06:20.947696    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf93aded2_05e8_4d10_8485_630a97a87fcf.slice/crio-fbc6de46ab01d17d480bbfe1babd1ff6ad4d7b924a5ac3ea722709a189b04a0a WatchSource:0}: Error finding container fbc6de46ab01d17d480bbfe1babd1ff6ad4d7b924a5ac3ea722709a189b04a0a: Status 404 returned error can't find the container with id fbc6de46ab01d17d480bbfe1babd1ff6ad4d7b924a5ac3ea722709a189b04a0a
Jul 27 21:06:21.530209 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:06:21.530121    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-57469ddc4-sbpnb" event={"ID":"2fc45846-b5c2-44b8-b81e-7debda79f2f6","Type":"ContainerStarted","Data":"902542435b93c9fea8c1b88d46050e69a0a73ab64e774f0f225889c927f2a55a"}
Jul 27 21:06:21.530410 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:06:21.530396    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-57469ddc4-sbpnb"
Jul 27 21:06:21.531541 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:06:21.531508    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-669984f9db-74b9w" event={"ID":"f93aded2-05e8-4d10-8485-630a97a87fcf","Type":"ContainerStarted","Data":"fbc6de46ab01d17d480bbfe1babd1ff6ad4d7b924a5ac3ea722709a189b04a0a"}
Jul 27 21:06:21.532425 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:06:21.532406    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-57469ddc4-sbpnb"
Jul 27 21:06:21.547127 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:06:21.547069    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-57469ddc4-sbpnb" podStartSLOduration=2.246160044 podStartE2EDuration="8.547052192s" totalImagesPullingTime="6.300892148s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 21:06:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 21:06:21.545877976 +0000 UTC m=+110.125061984" watchObservedRunningTime="2026-07-27 21:06:21.547052192 +0000 UTC m=+110.126236198"
Jul 27 21:06:24.544735 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:06:24.544633    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-669984f9db-74b9w" event={"ID":"f93aded2-05e8-4d10-8485-630a97a87fcf","Type":"ContainerStarted","Data":"321705e386693335094cbd1c936a6103cd8fa771f828c51d967d4a0672ed76b5"}
Jul 27 21:06:24.571622 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:06:24.571557    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-669984f9db-74b9w" podStartSLOduration=3.238896476 podStartE2EDuration="6.571537397s" totalImagesPullingTime="3.332640921s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 21:06:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 21:06:24.569955406 +0000 UTC m=+113.149139414" watchObservedRunningTime="2026-07-27 21:06:24.571537397 +0000 UTC m=+113.150721404"
Jul 27 21:06:27.172559 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:06:27.172525    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-56d9f944b4-dbv9r"]
Jul 27 21:06:27.176285 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:06:27.176259    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-56d9f944b4-dbv9r"
Jul 27 21:06:27.176829 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:06:27.176807    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-6ngt5"]
Jul 27 21:06:27.178873 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:06:27.178844    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\""
Jul 27 21:06:27.178997 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:06:27.178925    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\""
Jul 27 21:06:27.180060 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:06:27.180036    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\""
Jul 27 21:06:27.180167 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:06:27.180124    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-6ngt5"
Jul 27 21:06:27.180404 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:06:27.180382    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-hqrsg\""
Jul 27 21:06:27.184487 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:06:27.184402    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Jul 27 21:06:27.192591 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:06:27.192570    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Jul 27 21:06:27.192887 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:06:27.192866    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Jul 27 21:06:27.193018 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:06:27.192978    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Jul 27 21:06:27.193138 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:06:27.193121    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-6qqmv\""
Jul 27 21:06:27.227353 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:06:27.227306    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-56d9f944b4-dbv9r"]
Jul 27 21:06:27.276816 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:06:27.276788    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/1b8f587c-fd7c-44dd-903f-0eced954b117-root\") pod \"node-exporter-6ngt5\" (UID: \"1b8f587c-fd7c-44dd-903f-0eced954b117\") " pod="openshift-monitoring/node-exporter-6ngt5"
Jul 27 21:06:27.277021 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:06:27.276821    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/7854d704-78c9-468b-9973-8799da0fe0eb-kube-state-metrics-tls\") pod \"kube-state-metrics-56d9f944b4-dbv9r\" (UID: \"7854d704-78c9-468b-9973-8799da0fe0eb\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-dbv9r"
Jul 27 21:06:27.277021 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:06:27.276856    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/1b8f587c-fd7c-44dd-903f-0eced954b117-sys\") pod \"node-exporter-6ngt5\" (UID: \"1b8f587c-fd7c-44dd-903f-0eced954b117\") " pod="openshift-monitoring/node-exporter-6ngt5"
Jul 27 21:06:27.277021 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:06:27.276879    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/1b8f587c-fd7c-44dd-903f-0eced954b117-node-exporter-accelerators-collector-config\") pod \"node-exporter-6ngt5\" (UID: \"1b8f587c-fd7c-44dd-903f-0eced954b117\") " pod="openshift-monitoring/node-exporter-6ngt5"
Jul 27 21:06:27.277021 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:06:27.276927    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jdr5q\" (UniqueName: \"kubernetes.io/projected/7854d704-78c9-468b-9973-8799da0fe0eb-kube-api-access-jdr5q\") pod \"kube-state-metrics-56d9f944b4-dbv9r\" (UID: \"7854d704-78c9-468b-9973-8799da0fe0eb\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-dbv9r"
Jul 27 21:06:27.277021 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:06:27.276993    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7854d704-78c9-468b-9973-8799da0fe0eb-metrics-client-ca\") pod \"kube-state-metrics-56d9f944b4-dbv9r\" (UID: \"7854d704-78c9-468b-9973-8799da0fe0eb\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-dbv9r"
Jul 27 21:06:27.277272 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:06:27.277034    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/7854d704-78c9-468b-9973-8799da0fe0eb-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-56d9f944b4-dbv9r\" (UID: \"7854d704-78c9-468b-9973-8799da0fe0eb\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-dbv9r"
Jul 27 21:06:27.277272 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:06:27.277084    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/7854d704-78c9-468b-9973-8799da0fe0eb-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-56d9f944b4-dbv9r\" (UID: \"7854d704-78c9-468b-9973-8799da0fe0eb\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-dbv9r"
Jul 27 21:06:27.277272 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:06:27.277108    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/1b8f587c-fd7c-44dd-903f-0eced954b117-node-exporter-tls\") pod \"node-exporter-6ngt5\" (UID: \"1b8f587c-fd7c-44dd-903f-0eced954b117\") " pod="openshift-monitoring/node-exporter-6ngt5"
Jul 27 21:06:27.277272 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:06:27.277123    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1b8f587c-fd7c-44dd-903f-0eced954b117-metrics-client-ca\") pod \"node-exporter-6ngt5\" (UID: \"1b8f587c-fd7c-44dd-903f-0eced954b117\") " pod="openshift-monitoring/node-exporter-6ngt5"
Jul 27 21:06:27.277272 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:06:27.277147    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/1b8f587c-fd7c-44dd-903f-0eced954b117-node-exporter-textfile\") pod \"node-exporter-6ngt5\" (UID: \"1b8f587c-fd7c-44dd-903f-0eced954b117\") " pod="openshift-monitoring/node-exporter-6ngt5"
Jul 27 21:06:27.277272 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:06:27.277232    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/1b8f587c-fd7c-44dd-903f-0eced954b117-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-6ngt5\" (UID: \"1b8f587c-fd7c-44dd-903f-0eced954b117\") " pod="openshift-monitoring/node-exporter-6ngt5"
Jul 27 21:06:27.277272 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:06:27.277256    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2cghc\" (UniqueName: \"kubernetes.io/projected/1b8f587c-fd7c-44dd-903f-0eced954b117-kube-api-access-2cghc\") pod \"node-exporter-6ngt5\" (UID: \"1b8f587c-fd7c-44dd-903f-0eced954b117\") " pod="openshift-monitoring/node-exporter-6ngt5"
Jul 27 21:06:27.277568 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:06:27.277288    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/1b8f587c-fd7c-44dd-903f-0eced954b117-node-exporter-wtmp\") pod \"node-exporter-6ngt5\" (UID: \"1b8f587c-fd7c-44dd-903f-0eced954b117\") " pod="openshift-monitoring/node-exporter-6ngt5"
Jul 27 21:06:27.277568 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:06:27.277317    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/7854d704-78c9-468b-9973-8799da0fe0eb-volume-directive-shadow\") pod \"kube-state-metrics-56d9f944b4-dbv9r\" (UID: \"7854d704-78c9-468b-9973-8799da0fe0eb\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-dbv9r"
Jul 27 21:06:27.378136 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:06:27.378097    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/1b8f587c-fd7c-44dd-903f-0eced954b117-node-exporter-textfile\") pod \"node-exporter-6ngt5\" (UID: \"1b8f587c-fd7c-44dd-903f-0eced954b117\") " pod="openshift-monitoring/node-exporter-6ngt5"
Jul 27 21:06:27.378136 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:06:27.378147    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/1b8f587c-fd7c-44dd-903f-0eced954b117-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-6ngt5\" (UID: \"1b8f587c-fd7c-44dd-903f-0eced954b117\") " pod="openshift-monitoring/node-exporter-6ngt5"
Jul 27 21:06:27.378396 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:06:27.378169    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2cghc\" (UniqueName: \"kubernetes.io/projected/1b8f587c-fd7c-44dd-903f-0eced954b117-kube-api-access-2cghc\") pod \"node-exporter-6ngt5\" (UID: \"1b8f587c-fd7c-44dd-903f-0eced954b117\") " pod="openshift-monitoring/node-exporter-6ngt5"
Jul 27 21:06:27.378396 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:06:27.378199    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/1b8f587c-fd7c-44dd-903f-0eced954b117-node-exporter-wtmp\") pod \"node-exporter-6ngt5\" (UID: \"1b8f587c-fd7c-44dd-903f-0eced954b117\") " pod="openshift-monitoring/node-exporter-6ngt5"
Jul 27 21:06:27.378396 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:06:27.378237    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/7854d704-78c9-468b-9973-8799da0fe0eb-volume-directive-shadow\") pod \"kube-state-metrics-56d9f944b4-dbv9r\" (UID: \"7854d704-78c9-468b-9973-8799da0fe0eb\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-dbv9r"
Jul 27 21:06:27.378396 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:06:27.378266    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/1b8f587c-fd7c-44dd-903f-0eced954b117-root\") pod \"node-exporter-6ngt5\" (UID: \"1b8f587c-fd7c-44dd-903f-0eced954b117\") " pod="openshift-monitoring/node-exporter-6ngt5"
Jul 27 21:06:27.378396 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:06:27.378291    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/7854d704-78c9-468b-9973-8799da0fe0eb-kube-state-metrics-tls\") pod \"kube-state-metrics-56d9f944b4-dbv9r\" (UID: \"7854d704-78c9-468b-9973-8799da0fe0eb\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-dbv9r"
Jul 27 21:06:27.378396 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:06:27.378324    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/1b8f587c-fd7c-44dd-903f-0eced954b117-sys\") pod \"node-exporter-6ngt5\" (UID: \"1b8f587c-fd7c-44dd-903f-0eced954b117\") " pod="openshift-monitoring/node-exporter-6ngt5"
Jul 27 21:06:27.378396 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:06:27.378346    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/1b8f587c-fd7c-44dd-903f-0eced954b117-node-exporter-accelerators-collector-config\") pod \"node-exporter-6ngt5\" (UID: \"1b8f587c-fd7c-44dd-903f-0eced954b117\") " pod="openshift-monitoring/node-exporter-6ngt5"
Jul 27 21:06:27.378396 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:06:27.378374    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jdr5q\" (UniqueName: \"kubernetes.io/projected/7854d704-78c9-468b-9973-8799da0fe0eb-kube-api-access-jdr5q\") pod \"kube-state-metrics-56d9f944b4-dbv9r\" (UID: \"7854d704-78c9-468b-9973-8799da0fe0eb\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-dbv9r"
Jul 27 21:06:27.378867 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:06:27.378401    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/1b8f587c-fd7c-44dd-903f-0eced954b117-node-exporter-wtmp\") pod \"node-exporter-6ngt5\" (UID: \"1b8f587c-fd7c-44dd-903f-0eced954b117\") " pod="openshift-monitoring/node-exporter-6ngt5"
Jul 27 21:06:27.378867 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:06:27.378406    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7854d704-78c9-468b-9973-8799da0fe0eb-metrics-client-ca\") pod \"kube-state-metrics-56d9f944b4-dbv9r\" (UID: \"7854d704-78c9-468b-9973-8799da0fe0eb\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-dbv9r"
Jul 27 21:06:27.378867 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:06:27.378462    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/7854d704-78c9-468b-9973-8799da0fe0eb-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-56d9f944b4-dbv9r\" (UID: \"7854d704-78c9-468b-9973-8799da0fe0eb\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-dbv9r"
Jul 27 21:06:27.378867 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:06:27.378513    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/7854d704-78c9-468b-9973-8799da0fe0eb-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-56d9f944b4-dbv9r\" (UID: \"7854d704-78c9-468b-9973-8799da0fe0eb\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-dbv9r"
Jul 27 21:06:27.378867 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:06:27.378520    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/1b8f587c-fd7c-44dd-903f-0eced954b117-sys\") pod \"node-exporter-6ngt5\" (UID: \"1b8f587c-fd7c-44dd-903f-0eced954b117\") " pod="openshift-monitoring/node-exporter-6ngt5"
Jul 27 21:06:27.378867 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:06:27.378545    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/1b8f587c-fd7c-44dd-903f-0eced954b117-node-exporter-tls\") pod \"node-exporter-6ngt5\" (UID: \"1b8f587c-fd7c-44dd-903f-0eced954b117\") " pod="openshift-monitoring/node-exporter-6ngt5"
Jul 27 21:06:27.378867 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:06:27.378572    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1b8f587c-fd7c-44dd-903f-0eced954b117-metrics-client-ca\") pod \"node-exporter-6ngt5\" (UID: \"1b8f587c-fd7c-44dd-903f-0eced954b117\") " pod="openshift-monitoring/node-exporter-6ngt5"
Jul 27 21:06:27.378867 ip-10-0-131-133 kubenswrapper[2559]: E0727 21:06:27.378627    2559 secret.go:189] Couldn't get secret openshift-monitoring/kube-state-metrics-tls: secret "kube-state-metrics-tls" not found
Jul 27 21:06:27.378867 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:06:27.378667    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/7854d704-78c9-468b-9973-8799da0fe0eb-volume-directive-shadow\") pod \"kube-state-metrics-56d9f944b4-dbv9r\" (UID: \"7854d704-78c9-468b-9973-8799da0fe0eb\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-dbv9r"
Jul 27 21:06:27.378867 ip-10-0-131-133 kubenswrapper[2559]: E0727 21:06:27.378700    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7854d704-78c9-468b-9973-8799da0fe0eb-kube-state-metrics-tls podName:7854d704-78c9-468b-9973-8799da0fe0eb nodeName:}" failed. No retries permitted until 2026-07-27 21:06:27.878680852 +0000 UTC m=+116.457864843 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-state-metrics-tls" (UniqueName: "kubernetes.io/secret/7854d704-78c9-468b-9973-8799da0fe0eb-kube-state-metrics-tls") pod "kube-state-metrics-56d9f944b4-dbv9r" (UID: "7854d704-78c9-468b-9973-8799da0fe0eb") : secret "kube-state-metrics-tls" not found
Jul 27 21:06:27.379421 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:06:27.378541    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/1b8f587c-fd7c-44dd-903f-0eced954b117-node-exporter-textfile\") pod \"node-exporter-6ngt5\" (UID: \"1b8f587c-fd7c-44dd-903f-0eced954b117\") " pod="openshift-monitoring/node-exporter-6ngt5"
Jul 27 21:06:27.379421 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:06:27.379026    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/1b8f587c-fd7c-44dd-903f-0eced954b117-root\") pod \"node-exporter-6ngt5\" (UID: \"1b8f587c-fd7c-44dd-903f-0eced954b117\") " pod="openshift-monitoring/node-exporter-6ngt5"
Jul 27 21:06:27.379421 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:06:27.379056    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7854d704-78c9-468b-9973-8799da0fe0eb-metrics-client-ca\") pod \"kube-state-metrics-56d9f944b4-dbv9r\" (UID: \"7854d704-78c9-468b-9973-8799da0fe0eb\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-dbv9r"
Jul 27 21:06:27.379421 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:06:27.379085    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/1b8f587c-fd7c-44dd-903f-0eced954b117-node-exporter-accelerators-collector-config\") pod \"node-exporter-6ngt5\" (UID: \"1b8f587c-fd7c-44dd-903f-0eced954b117\") " pod="openshift-monitoring/node-exporter-6ngt5"
Jul 27 21:06:27.379628 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:06:27.379435    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/7854d704-78c9-468b-9973-8799da0fe0eb-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-56d9f944b4-dbv9r\" (UID: \"7854d704-78c9-468b-9973-8799da0fe0eb\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-dbv9r"
Jul 27 21:06:27.379628 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:06:27.379495    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1b8f587c-fd7c-44dd-903f-0eced954b117-metrics-client-ca\") pod \"node-exporter-6ngt5\" (UID: \"1b8f587c-fd7c-44dd-903f-0eced954b117\") " pod="openshift-monitoring/node-exporter-6ngt5"
Jul 27 21:06:27.381249 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:06:27.381207    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/1b8f587c-fd7c-44dd-903f-0eced954b117-node-exporter-tls\") pod \"node-exporter-6ngt5\" (UID: \"1b8f587c-fd7c-44dd-903f-0eced954b117\") " pod="openshift-monitoring/node-exporter-6ngt5"
Jul 27 21:06:27.381249 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:06:27.381239    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/7854d704-78c9-468b-9973-8799da0fe0eb-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-56d9f944b4-dbv9r\" (UID: \"7854d704-78c9-468b-9973-8799da0fe0eb\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-dbv9r"
Jul 27 21:06:27.381419 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:06:27.381299    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/1b8f587c-fd7c-44dd-903f-0eced954b117-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-6ngt5\" (UID: \"1b8f587c-fd7c-44dd-903f-0eced954b117\") " pod="openshift-monitoring/node-exporter-6ngt5"
Jul 27 21:06:27.389242 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:06:27.389199    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2cghc\" (UniqueName: \"kubernetes.io/projected/1b8f587c-fd7c-44dd-903f-0eced954b117-kube-api-access-2cghc\") pod \"node-exporter-6ngt5\" (UID: \"1b8f587c-fd7c-44dd-903f-0eced954b117\") " pod="openshift-monitoring/node-exporter-6ngt5"
Jul 27 21:06:27.389496 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:06:27.389475    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jdr5q\" (UniqueName: \"kubernetes.io/projected/7854d704-78c9-468b-9973-8799da0fe0eb-kube-api-access-jdr5q\") pod \"kube-state-metrics-56d9f944b4-dbv9r\" (UID: \"7854d704-78c9-468b-9973-8799da0fe0eb\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-dbv9r"
Jul 27 21:06:27.494354 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:06:27.494257    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-6ngt5"
Jul 27 21:06:27.883593 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:06:27.883544    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/7854d704-78c9-468b-9973-8799da0fe0eb-kube-state-metrics-tls\") pod \"kube-state-metrics-56d9f944b4-dbv9r\" (UID: \"7854d704-78c9-468b-9973-8799da0fe0eb\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-dbv9r"
Jul 27 21:06:27.886524 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:06:27.886490    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/7854d704-78c9-468b-9973-8799da0fe0eb-kube-state-metrics-tls\") pod \"kube-state-metrics-56d9f944b4-dbv9r\" (UID: \"7854d704-78c9-468b-9973-8799da0fe0eb\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-dbv9r"
Jul 27 21:06:28.088534 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:06:28.088487    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-56d9f944b4-dbv9r"
Jul 27 21:06:28.523718 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:06:28.523689    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-9kfdx"
Jul 27 21:06:28.663632 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:06:28.663581    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-669984f9db-74b9w"
Jul 27 21:06:28.663813 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:06:28.663666    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-669984f9db-74b9w"
Jul 27 21:06:28.665187 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:06:28.665156    2559 patch_prober.go:28] interesting pod/console-669984f9db-74b9w container/console namespace/openshift-console: Startup probe status=failure output="Get \"https://10.132.0.16:8443/health\": dial tcp 10.132.0.16:8443: connect: connection refused" start-of-body=
Jul 27 21:06:28.665337 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:06:28.665241    2559 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-console/console-669984f9db-74b9w" podUID="f93aded2-05e8-4d10-8485-630a97a87fcf" containerName="console" probeResult="failure" output="Get \"https://10.132.0.16:8443/health\": dial tcp 10.132.0.16:8443: connect: connection refused"
Jul 27 21:06:30.142437 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:06:30.142402    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-64dfb7c55b-blsbx"]
Jul 27 21:06:30.273580 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:06:30.273538    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-64dfb7c55b-blsbx"]
Jul 27 21:06:30.273789 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:06:30.273768    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-64dfb7c55b-blsbx"
Jul 27 21:06:30.276457 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:06:30.276428    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\""
Jul 27 21:06:30.276621 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:06:30.276601    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\""
Jul 27 21:06:30.276695 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:06:30.276617    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\""
Jul 27 21:06:30.276695 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:06:30.276669    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-n8749\""
Jul 27 21:06:30.276810 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:06:30.276700    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\""
Jul 27 21:06:30.276810 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:06:30.276729    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\""
Jul 27 21:06:30.276810 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:06:30.276750    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-4vslb04g5i4d5\""
Jul 27 21:06:30.306679 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:06:30.306641    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8mtdh\" (UniqueName: \"kubernetes.io/projected/3ba37884-dc16-419a-b9ca-ba49c6171c9c-kube-api-access-8mtdh\") pod \"thanos-querier-64dfb7c55b-blsbx\" (UID: \"3ba37884-dc16-419a-b9ca-ba49c6171c9c\") " pod="openshift-monitoring/thanos-querier-64dfb7c55b-blsbx"
Jul 27 21:06:30.306863 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:06:30.306735    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/3ba37884-dc16-419a-b9ca-ba49c6171c9c-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-64dfb7c55b-blsbx\" (UID: \"3ba37884-dc16-419a-b9ca-ba49c6171c9c\") " pod="openshift-monitoring/thanos-querier-64dfb7c55b-blsbx"
Jul 27 21:06:30.306863 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:06:30.306780    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/3ba37884-dc16-419a-b9ca-ba49c6171c9c-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-64dfb7c55b-blsbx\" (UID: \"3ba37884-dc16-419a-b9ca-ba49c6171c9c\") " pod="openshift-monitoring/thanos-querier-64dfb7c55b-blsbx"
Jul 27 21:06:30.306863 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:06:30.306826    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/3ba37884-dc16-419a-b9ca-ba49c6171c9c-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-64dfb7c55b-blsbx\" (UID: \"3ba37884-dc16-419a-b9ca-ba49c6171c9c\") " pod="openshift-monitoring/thanos-querier-64dfb7c55b-blsbx"
Jul 27 21:06:30.306975 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:06:30.306883    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/3ba37884-dc16-419a-b9ca-ba49c6171c9c-secret-thanos-querier-tls\") pod \"thanos-querier-64dfb7c55b-blsbx\" (UID: \"3ba37884-dc16-419a-b9ca-ba49c6171c9c\") " pod="openshift-monitoring/thanos-querier-64dfb7c55b-blsbx"
Jul 27 21:06:30.306975 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:06:30.306940    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3ba37884-dc16-419a-b9ca-ba49c6171c9c-metrics-client-ca\") pod \"thanos-querier-64dfb7c55b-blsbx\" (UID: \"3ba37884-dc16-419a-b9ca-ba49c6171c9c\") " pod="openshift-monitoring/thanos-querier-64dfb7c55b-blsbx"
Jul 27 21:06:30.307040 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:06:30.306971    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/3ba37884-dc16-419a-b9ca-ba49c6171c9c-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-64dfb7c55b-blsbx\" (UID: \"3ba37884-dc16-419a-b9ca-ba49c6171c9c\") " pod="openshift-monitoring/thanos-querier-64dfb7c55b-blsbx"
Jul 27 21:06:30.307040 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:06:30.307026    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/3ba37884-dc16-419a-b9ca-ba49c6171c9c-secret-grpc-tls\") pod \"thanos-querier-64dfb7c55b-blsbx\" (UID: \"3ba37884-dc16-419a-b9ca-ba49c6171c9c\") " pod="openshift-monitoring/thanos-querier-64dfb7c55b-blsbx"
Jul 27 21:06:30.408168 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:06:30.408079    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/3ba37884-dc16-419a-b9ca-ba49c6171c9c-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-64dfb7c55b-blsbx\" (UID: \"3ba37884-dc16-419a-b9ca-ba49c6171c9c\") " pod="openshift-monitoring/thanos-querier-64dfb7c55b-blsbx"
Jul 27 21:06:30.408168 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:06:30.408153    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/3ba37884-dc16-419a-b9ca-ba49c6171c9c-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-64dfb7c55b-blsbx\" (UID: \"3ba37884-dc16-419a-b9ca-ba49c6171c9c\") " pod="openshift-monitoring/thanos-querier-64dfb7c55b-blsbx"
Jul 27 21:06:30.408409 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:06:30.408192    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/3ba37884-dc16-419a-b9ca-ba49c6171c9c-secret-thanos-querier-tls\") pod \"thanos-querier-64dfb7c55b-blsbx\" (UID: \"3ba37884-dc16-419a-b9ca-ba49c6171c9c\") " pod="openshift-monitoring/thanos-querier-64dfb7c55b-blsbx"
Jul 27 21:06:30.408409 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:06:30.408245    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3ba37884-dc16-419a-b9ca-ba49c6171c9c-metrics-client-ca\") pod \"thanos-querier-64dfb7c55b-blsbx\" (UID: \"3ba37884-dc16-419a-b9ca-ba49c6171c9c\") " pod="openshift-monitoring/thanos-querier-64dfb7c55b-blsbx"
Jul 27 21:06:30.408409 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:06:30.408278    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/3ba37884-dc16-419a-b9ca-ba49c6171c9c-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-64dfb7c55b-blsbx\" (UID: \"3ba37884-dc16-419a-b9ca-ba49c6171c9c\") " pod="openshift-monitoring/thanos-querier-64dfb7c55b-blsbx"
Jul 27 21:06:30.408409 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:06:30.408323    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/3ba37884-dc16-419a-b9ca-ba49c6171c9c-secret-grpc-tls\") pod \"thanos-querier-64dfb7c55b-blsbx\" (UID: \"3ba37884-dc16-419a-b9ca-ba49c6171c9c\") " pod="openshift-monitoring/thanos-querier-64dfb7c55b-blsbx"
Jul 27 21:06:30.408409 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:06:30.408344    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8mtdh\" (UniqueName: \"kubernetes.io/projected/3ba37884-dc16-419a-b9ca-ba49c6171c9c-kube-api-access-8mtdh\") pod \"thanos-querier-64dfb7c55b-blsbx\" (UID: \"3ba37884-dc16-419a-b9ca-ba49c6171c9c\") " pod="openshift-monitoring/thanos-querier-64dfb7c55b-blsbx"
Jul 27 21:06:30.408409 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:06:30.408370    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/3ba37884-dc16-419a-b9ca-ba49c6171c9c-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-64dfb7c55b-blsbx\" (UID: \"3ba37884-dc16-419a-b9ca-ba49c6171c9c\") " pod="openshift-monitoring/thanos-querier-64dfb7c55b-blsbx"
Jul 27 21:06:30.409473 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:06:30.409394    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3ba37884-dc16-419a-b9ca-ba49c6171c9c-metrics-client-ca\") pod \"thanos-querier-64dfb7c55b-blsbx\" (UID: \"3ba37884-dc16-419a-b9ca-ba49c6171c9c\") " pod="openshift-monitoring/thanos-querier-64dfb7c55b-blsbx"
Jul 27 21:06:30.411401 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:06:30.411365    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/3ba37884-dc16-419a-b9ca-ba49c6171c9c-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-64dfb7c55b-blsbx\" (UID: \"3ba37884-dc16-419a-b9ca-ba49c6171c9c\") " pod="openshift-monitoring/thanos-querier-64dfb7c55b-blsbx"
Jul 27 21:06:30.411524 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:06:30.411494    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/3ba37884-dc16-419a-b9ca-ba49c6171c9c-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-64dfb7c55b-blsbx\" (UID: \"3ba37884-dc16-419a-b9ca-ba49c6171c9c\") " pod="openshift-monitoring/thanos-querier-64dfb7c55b-blsbx"
Jul 27 21:06:30.411642 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:06:30.411606    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/3ba37884-dc16-419a-b9ca-ba49c6171c9c-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-64dfb7c55b-blsbx\" (UID: \"3ba37884-dc16-419a-b9ca-ba49c6171c9c\") " pod="openshift-monitoring/thanos-querier-64dfb7c55b-blsbx"
Jul 27 21:06:30.412046 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:06:30.412009    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/3ba37884-dc16-419a-b9ca-ba49c6171c9c-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-64dfb7c55b-blsbx\" (UID: \"3ba37884-dc16-419a-b9ca-ba49c6171c9c\") " pod="openshift-monitoring/thanos-querier-64dfb7c55b-blsbx"
Jul 27 21:06:30.412046 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:06:30.412029    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/3ba37884-dc16-419a-b9ca-ba49c6171c9c-secret-grpc-tls\") pod \"thanos-querier-64dfb7c55b-blsbx\" (UID: \"3ba37884-dc16-419a-b9ca-ba49c6171c9c\") " pod="openshift-monitoring/thanos-querier-64dfb7c55b-blsbx"
Jul 27 21:06:30.412844 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:06:30.412821    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/3ba37884-dc16-419a-b9ca-ba49c6171c9c-secret-thanos-querier-tls\") pod \"thanos-querier-64dfb7c55b-blsbx\" (UID: \"3ba37884-dc16-419a-b9ca-ba49c6171c9c\") " pod="openshift-monitoring/thanos-querier-64dfb7c55b-blsbx"
Jul 27 21:06:30.417556 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:06:30.417533    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8mtdh\" (UniqueName: \"kubernetes.io/projected/3ba37884-dc16-419a-b9ca-ba49c6171c9c-kube-api-access-8mtdh\") pod \"thanos-querier-64dfb7c55b-blsbx\" (UID: \"3ba37884-dc16-419a-b9ca-ba49c6171c9c\") " pod="openshift-monitoring/thanos-querier-64dfb7c55b-blsbx"
Jul 27 21:06:30.586176 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:06:30.586124    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-64dfb7c55b-blsbx"
Jul 27 21:06:31.657328 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:06:31.657297    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-686dbf5f69-zb2mj"]
Jul 27 21:06:31.697571 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:06:31.697528    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-686dbf5f69-zb2mj"]
Jul 27 21:06:31.697741 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:06:31.697639    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-686dbf5f69-zb2mj"
Jul 27 21:06:31.700668 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:06:31.700639    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-2e4vplc04kn84\""
Jul 27 21:06:31.701330 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:06:31.701309    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\""
Jul 27 21:06:31.701558 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:06:31.701538    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\""
Jul 27 21:06:31.702601 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:06:31.702577    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-8wxmn\""
Jul 27 21:06:31.702714 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:06:31.702691    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Jul 27 21:06:31.702792 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:06:31.702777    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\""
Jul 27 21:06:31.760646 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:06:31.760606    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-68b7fc9574-gmwds"]
Jul 27 21:06:31.783723 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:06:31.783691    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-68b7fc9574-gmwds"]
Jul 27 21:06:31.783900 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:06:31.783829    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-68b7fc9574-gmwds"
Jul 27 21:06:31.786958 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:06:31.786929    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\""
Jul 27 21:06:31.791658 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:06:31.791026    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-d49lq\""
Jul 27 21:06:31.822189 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:06:31.822130    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/45e10f06-0e63-4eac-b3c6-42287beb9a75-secret-metrics-server-tls\") pod \"metrics-server-686dbf5f69-zb2mj\" (UID: \"45e10f06-0e63-4eac-b3c6-42287beb9a75\") " pod="openshift-monitoring/metrics-server-686dbf5f69-zb2mj"
Jul 27 21:06:31.822189 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:06:31.822174    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/45e10f06-0e63-4eac-b3c6-42287beb9a75-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-686dbf5f69-zb2mj\" (UID: \"45e10f06-0e63-4eac-b3c6-42287beb9a75\") " pod="openshift-monitoring/metrics-server-686dbf5f69-zb2mj"
Jul 27 21:06:31.822434 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:06:31.822307    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/45e10f06-0e63-4eac-b3c6-42287beb9a75-metrics-server-audit-profiles\") pod \"metrics-server-686dbf5f69-zb2mj\" (UID: \"45e10f06-0e63-4eac-b3c6-42287beb9a75\") " pod="openshift-monitoring/metrics-server-686dbf5f69-zb2mj"
Jul 27 21:06:31.822434 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:06:31.822384    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pspbt\" (UniqueName: \"kubernetes.io/projected/45e10f06-0e63-4eac-b3c6-42287beb9a75-kube-api-access-pspbt\") pod \"metrics-server-686dbf5f69-zb2mj\" (UID: \"45e10f06-0e63-4eac-b3c6-42287beb9a75\") " pod="openshift-monitoring/metrics-server-686dbf5f69-zb2mj"
Jul 27 21:06:31.822434 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:06:31.822425    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/45e10f06-0e63-4eac-b3c6-42287beb9a75-secret-metrics-server-client-certs\") pod \"metrics-server-686dbf5f69-zb2mj\" (UID: \"45e10f06-0e63-4eac-b3c6-42287beb9a75\") " pod="openshift-monitoring/metrics-server-686dbf5f69-zb2mj"
Jul 27 21:06:31.822585 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:06:31.822441    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/45e10f06-0e63-4eac-b3c6-42287beb9a75-client-ca-bundle\") pod \"metrics-server-686dbf5f69-zb2mj\" (UID: \"45e10f06-0e63-4eac-b3c6-42287beb9a75\") " pod="openshift-monitoring/metrics-server-686dbf5f69-zb2mj"
Jul 27 21:06:31.822585 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:06:31.822475    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/fbf8db18-7974-4032-b919-6be505ffe0e1-monitoring-plugin-cert\") pod \"monitoring-plugin-68b7fc9574-gmwds\" (UID: \"fbf8db18-7974-4032-b919-6be505ffe0e1\") " pod="openshift-monitoring/monitoring-plugin-68b7fc9574-gmwds"
Jul 27 21:06:31.822585 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:06:31.822503    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/45e10f06-0e63-4eac-b3c6-42287beb9a75-audit-log\") pod \"metrics-server-686dbf5f69-zb2mj\" (UID: \"45e10f06-0e63-4eac-b3c6-42287beb9a75\") " pod="openshift-monitoring/metrics-server-686dbf5f69-zb2mj"
Jul 27 21:06:31.822585 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:06:31.822528    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-88mfg\" (UniqueName: \"kubernetes.io/projected/fbf8db18-7974-4032-b919-6be505ffe0e1-kube-api-access-88mfg\") pod \"monitoring-plugin-68b7fc9574-gmwds\" (UID: \"fbf8db18-7974-4032-b919-6be505ffe0e1\") " pod="openshift-monitoring/monitoring-plugin-68b7fc9574-gmwds"
Jul 27 21:06:31.923671 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:06:31.923588    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-88mfg\" (UniqueName: \"kubernetes.io/projected/fbf8db18-7974-4032-b919-6be505ffe0e1-kube-api-access-88mfg\") pod \"monitoring-plugin-68b7fc9574-gmwds\" (UID: \"fbf8db18-7974-4032-b919-6be505ffe0e1\") " pod="openshift-monitoring/monitoring-plugin-68b7fc9574-gmwds"
Jul 27 21:06:31.923671 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:06:31.923656    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/45e10f06-0e63-4eac-b3c6-42287beb9a75-secret-metrics-server-tls\") pod \"metrics-server-686dbf5f69-zb2mj\" (UID: \"45e10f06-0e63-4eac-b3c6-42287beb9a75\") " pod="openshift-monitoring/metrics-server-686dbf5f69-zb2mj"
Jul 27 21:06:31.923906 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:06:31.923732    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/45e10f06-0e63-4eac-b3c6-42287beb9a75-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-686dbf5f69-zb2mj\" (UID: \"45e10f06-0e63-4eac-b3c6-42287beb9a75\") " pod="openshift-monitoring/metrics-server-686dbf5f69-zb2mj"
Jul 27 21:06:31.923906 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:06:31.923821    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/45e10f06-0e63-4eac-b3c6-42287beb9a75-metrics-server-audit-profiles\") pod \"metrics-server-686dbf5f69-zb2mj\" (UID: \"45e10f06-0e63-4eac-b3c6-42287beb9a75\") " pod="openshift-monitoring/metrics-server-686dbf5f69-zb2mj"
Jul 27 21:06:31.923906 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:06:31.923868    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pspbt\" (UniqueName: \"kubernetes.io/projected/45e10f06-0e63-4eac-b3c6-42287beb9a75-kube-api-access-pspbt\") pod \"metrics-server-686dbf5f69-zb2mj\" (UID: \"45e10f06-0e63-4eac-b3c6-42287beb9a75\") " pod="openshift-monitoring/metrics-server-686dbf5f69-zb2mj"
Jul 27 21:06:31.924070 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:06:31.923922    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/45e10f06-0e63-4eac-b3c6-42287beb9a75-secret-metrics-server-client-certs\") pod \"metrics-server-686dbf5f69-zb2mj\" (UID: \"45e10f06-0e63-4eac-b3c6-42287beb9a75\") " pod="openshift-monitoring/metrics-server-686dbf5f69-zb2mj"
Jul 27 21:06:31.924070 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:06:31.923954    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/45e10f06-0e63-4eac-b3c6-42287beb9a75-client-ca-bundle\") pod \"metrics-server-686dbf5f69-zb2mj\" (UID: \"45e10f06-0e63-4eac-b3c6-42287beb9a75\") " pod="openshift-monitoring/metrics-server-686dbf5f69-zb2mj"
Jul 27 21:06:31.924070 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:06:31.924004    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/fbf8db18-7974-4032-b919-6be505ffe0e1-monitoring-plugin-cert\") pod \"monitoring-plugin-68b7fc9574-gmwds\" (UID: \"fbf8db18-7974-4032-b919-6be505ffe0e1\") " pod="openshift-monitoring/monitoring-plugin-68b7fc9574-gmwds"
Jul 27 21:06:31.924070 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:06:31.924036    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/45e10f06-0e63-4eac-b3c6-42287beb9a75-audit-log\") pod \"metrics-server-686dbf5f69-zb2mj\" (UID: \"45e10f06-0e63-4eac-b3c6-42287beb9a75\") " pod="openshift-monitoring/metrics-server-686dbf5f69-zb2mj"
Jul 27 21:06:31.924506 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:06:31.924478    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/45e10f06-0e63-4eac-b3c6-42287beb9a75-audit-log\") pod \"metrics-server-686dbf5f69-zb2mj\" (UID: \"45e10f06-0e63-4eac-b3c6-42287beb9a75\") " pod="openshift-monitoring/metrics-server-686dbf5f69-zb2mj"
Jul 27 21:06:31.926313 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:06:31.926256    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\""
Jul 27 21:06:31.926313 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:06:31.926265    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\""
Jul 27 21:06:31.926667 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:06:31.926647    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\""
Jul 27 21:06:31.926765 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:06:31.926707    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Jul 27 21:06:31.927067 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:06:31.927030    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-2e4vplc04kn84\""
Jul 27 21:06:31.927067 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:06:31.927054    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\""
Jul 27 21:06:31.934337 ip-10-0-131-133 kubenswrapper[2559]: E0727 21:06:31.934308    2559 secret.go:189] Couldn't get secret openshift-monitoring/monitoring-plugin-cert: secret "monitoring-plugin-cert" not found
Jul 27 21:06:31.934484 ip-10-0-131-133 kubenswrapper[2559]: E0727 21:06:31.934372    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fbf8db18-7974-4032-b919-6be505ffe0e1-monitoring-plugin-cert podName:fbf8db18-7974-4032-b919-6be505ffe0e1 nodeName:}" failed. No retries permitted until 2026-07-27 21:06:32.434354058 +0000 UTC m=+121.013538048 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "monitoring-plugin-cert" (UniqueName: "kubernetes.io/secret/fbf8db18-7974-4032-b919-6be505ffe0e1-monitoring-plugin-cert") pod "monitoring-plugin-68b7fc9574-gmwds" (UID: "fbf8db18-7974-4032-b919-6be505ffe0e1") : secret "monitoring-plugin-cert" not found
Jul 27 21:06:31.934484 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:06:31.934418    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pspbt\" (UniqueName: \"kubernetes.io/projected/45e10f06-0e63-4eac-b3c6-42287beb9a75-kube-api-access-pspbt\") pod \"metrics-server-686dbf5f69-zb2mj\" (UID: \"45e10f06-0e63-4eac-b3c6-42287beb9a75\") " pod="openshift-monitoring/metrics-server-686dbf5f69-zb2mj"
Jul 27 21:06:31.935862 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:06:31.934961    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-88mfg\" (UniqueName: \"kubernetes.io/projected/fbf8db18-7974-4032-b919-6be505ffe0e1-kube-api-access-88mfg\") pod \"monitoring-plugin-68b7fc9574-gmwds\" (UID: \"fbf8db18-7974-4032-b919-6be505ffe0e1\") " pod="openshift-monitoring/monitoring-plugin-68b7fc9574-gmwds"
Jul 27 21:06:31.935862 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:06:31.935011    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/45e10f06-0e63-4eac-b3c6-42287beb9a75-metrics-server-audit-profiles\") pod \"metrics-server-686dbf5f69-zb2mj\" (UID: \"45e10f06-0e63-4eac-b3c6-42287beb9a75\") " pod="openshift-monitoring/metrics-server-686dbf5f69-zb2mj"
Jul 27 21:06:31.935862 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:06:31.935174    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/45e10f06-0e63-4eac-b3c6-42287beb9a75-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-686dbf5f69-zb2mj\" (UID: \"45e10f06-0e63-4eac-b3c6-42287beb9a75\") " pod="openshift-monitoring/metrics-server-686dbf5f69-zb2mj"
Jul 27 21:06:31.937732 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:06:31.937704    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/45e10f06-0e63-4eac-b3c6-42287beb9a75-secret-metrics-server-tls\") pod \"metrics-server-686dbf5f69-zb2mj\" (UID: \"45e10f06-0e63-4eac-b3c6-42287beb9a75\") " pod="openshift-monitoring/metrics-server-686dbf5f69-zb2mj"
Jul 27 21:06:31.938164 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:06:31.938140    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/45e10f06-0e63-4eac-b3c6-42287beb9a75-client-ca-bundle\") pod \"metrics-server-686dbf5f69-zb2mj\" (UID: \"45e10f06-0e63-4eac-b3c6-42287beb9a75\") " pod="openshift-monitoring/metrics-server-686dbf5f69-zb2mj"
Jul 27 21:06:31.938291 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:06:31.938190    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/45e10f06-0e63-4eac-b3c6-42287beb9a75-secret-metrics-server-client-certs\") pod \"metrics-server-686dbf5f69-zb2mj\" (UID: \"45e10f06-0e63-4eac-b3c6-42287beb9a75\") " pod="openshift-monitoring/metrics-server-686dbf5f69-zb2mj"
Jul 27 21:06:31.975957 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:06:31.975916    2559 patch_prober.go:28] interesting pod/image-registry-5b565b454b-2kr9d container/registry namespace/openshift-image-registry: Liveness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]}
Jul 27 21:06:31.976149 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:06:31.975988    2559 prober.go:120] "Probe failed" probeType="Liveness" pod="openshift-image-registry/image-registry-5b565b454b-2kr9d" podUID="097262d9-cf78-4d89-98cc-cf92d23a4f04" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 27 21:06:32.014690 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:06:32.014660    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-8wxmn\""
Jul 27 21:06:32.022757 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:06:32.022719    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-686dbf5f69-zb2mj"
Jul 27 21:06:32.529225 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:06:32.529180    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/fbf8db18-7974-4032-b919-6be505ffe0e1-monitoring-plugin-cert\") pod \"monitoring-plugin-68b7fc9574-gmwds\" (UID: \"fbf8db18-7974-4032-b919-6be505ffe0e1\") " pod="openshift-monitoring/monitoring-plugin-68b7fc9574-gmwds"
Jul 27 21:06:32.532011 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:06:32.531985    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/fbf8db18-7974-4032-b919-6be505ffe0e1-monitoring-plugin-cert\") pod \"monitoring-plugin-68b7fc9574-gmwds\" (UID: \"fbf8db18-7974-4032-b919-6be505ffe0e1\") " pod="openshift-monitoring/monitoring-plugin-68b7fc9574-gmwds"
Jul 27 21:06:32.696876 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:06:32.696845    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-d49lq\""
Jul 27 21:06:32.704840 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:06:32.704809    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-68b7fc9574-gmwds"
Jul 27 21:06:32.810346 ip-10-0-131-133 kubenswrapper[2559]: W0727 21:06:32.810099    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1b8f587c_fd7c_44dd_903f_0eced954b117.slice/crio-c8976e7749ac1273c379023e4ad2f4790e900ce0162abb6c339726477674bfe0 WatchSource:0}: Error finding container c8976e7749ac1273c379023e4ad2f4790e900ce0162abb6c339726477674bfe0: Status 404 returned error can't find the container with id c8976e7749ac1273c379023e4ad2f4790e900ce0162abb6c339726477674bfe0
Jul 27 21:06:32.984773 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:06:32.984732    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-64dfb7c55b-blsbx"]
Jul 27 21:06:32.993099 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:06:32.993077    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-68b7fc9574-gmwds"]
Jul 27 21:06:33.147853 ip-10-0-131-133 kubenswrapper[2559]: W0727 21:06:33.147805    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3ba37884_dc16_419a_b9ca_ba49c6171c9c.slice/crio-fee8c8b3d743693fd858daca673689bbbe2831dbc67239136cdad89ce324aa90 WatchSource:0}: Error finding container fee8c8b3d743693fd858daca673689bbbe2831dbc67239136cdad89ce324aa90: Status 404 returned error can't find the container with id fee8c8b3d743693fd858daca673689bbbe2831dbc67239136cdad89ce324aa90
Jul 27 21:06:33.148609 ip-10-0-131-133 kubenswrapper[2559]: W0727 21:06:33.148500    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfbf8db18_7974_4032_b919_6be505ffe0e1.slice/crio-af7d63e38301426fc36c58d8b8eeea087668b9d6eeae4fc5c57e90f5e1fb9662 WatchSource:0}: Error finding container af7d63e38301426fc36c58d8b8eeea087668b9d6eeae4fc5c57e90f5e1fb9662: Status 404 returned error can't find the container with id af7d63e38301426fc36c58d8b8eeea087668b9d6eeae4fc5c57e90f5e1fb9662
Jul 27 21:06:33.225537 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:06:33.225509    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-686dbf5f69-zb2mj"]
Jul 27 21:06:33.226541 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:06:33.226518    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-56d9f944b4-dbv9r"]
Jul 27 21:06:33.227942 ip-10-0-131-133 kubenswrapper[2559]: W0727 21:06:33.227913    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7854d704_78c9_468b_9973_8799da0fe0eb.slice/crio-ed7d7b8363eb23b3f93f33ae348899f74b68f32e6759539db44a2921d247a5b0 WatchSource:0}: Error finding container ed7d7b8363eb23b3f93f33ae348899f74b68f32e6759539db44a2921d247a5b0: Status 404 returned error can't find the container with id ed7d7b8363eb23b3f93f33ae348899f74b68f32e6759539db44a2921d247a5b0
Jul 27 21:06:33.228700 ip-10-0-131-133 kubenswrapper[2559]: W0727 21:06:33.228633    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod45e10f06_0e63_4eac_b3c6_42287beb9a75.slice/crio-758cbcee30e3fa7c0d9d170f6b881c6cbb0c81f2c7fb4a67d1afdbfa6c640cf4 WatchSource:0}: Error finding container 758cbcee30e3fa7c0d9d170f6b881c6cbb0c81f2c7fb4a67d1afdbfa6c640cf4: Status 404 returned error can't find the container with id 758cbcee30e3fa7c0d9d170f6b881c6cbb0c81f2c7fb4a67d1afdbfa6c640cf4
Jul 27 21:06:33.582929 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:06:33.582835    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-7c97dc4555-7x89h" event={"ID":"2ded1152-f764-48fc-b45f-744519e49e62","Type":"ContainerStarted","Data":"b6e4950f5f642eaba8359d0ac364a5fc4c9a2b3fb18be85ff7e67da71b92d283"}
Jul 27 21:06:33.583725 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:06:33.583656    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-7c97dc4555-7x89h"
Jul 27 21:06:33.585517 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:06:33.585194    2559 patch_prober.go:28] interesting pod/downloads-7c97dc4555-7x89h container/download-server namespace/openshift-console: Readiness probe status=failure output="Get \"http://10.132.0.15:8080/\": dial tcp 10.132.0.15:8080: connect: connection refused" start-of-body=
Jul 27 21:06:33.585517 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:06:33.585262    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-console/downloads-7c97dc4555-7x89h" podUID="2ded1152-f764-48fc-b45f-744519e49e62" containerName="download-server" probeResult="failure" output="Get \"http://10.132.0.15:8080/\": dial tcp 10.132.0.15:8080: connect: connection refused"
Jul 27 21:06:33.585517 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:06:33.585486    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-68b7fc9574-gmwds" event={"ID":"fbf8db18-7974-4032-b919-6be505ffe0e1","Type":"ContainerStarted","Data":"af7d63e38301426fc36c58d8b8eeea087668b9d6eeae4fc5c57e90f5e1fb9662"}
Jul 27 21:06:33.587094 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:06:33.586998    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-56d9f944b4-dbv9r" event={"ID":"7854d704-78c9-468b-9973-8799da0fe0eb","Type":"ContainerStarted","Data":"ed7d7b8363eb23b3f93f33ae348899f74b68f32e6759539db44a2921d247a5b0"}
Jul 27 21:06:33.588554 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:06:33.588521    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-686dbf5f69-zb2mj" event={"ID":"45e10f06-0e63-4eac-b3c6-42287beb9a75","Type":"ContainerStarted","Data":"758cbcee30e3fa7c0d9d170f6b881c6cbb0c81f2c7fb4a67d1afdbfa6c640cf4"}
Jul 27 21:06:33.590276 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:06:33.590250    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-64dfb7c55b-blsbx" event={"ID":"3ba37884-dc16-419a-b9ca-ba49c6171c9c","Type":"ContainerStarted","Data":"fee8c8b3d743693fd858daca673689bbbe2831dbc67239136cdad89ce324aa90"}
Jul 27 21:06:33.591907 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:06:33.591800    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-6ngt5" event={"ID":"1b8f587c-fd7c-44dd-903f-0eced954b117","Type":"ContainerStarted","Data":"c8976e7749ac1273c379023e4ad2f4790e900ce0162abb6c339726477674bfe0"}
Jul 27 21:06:33.629765 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:06:33.629467    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/downloads-7c97dc4555-7x89h" podStartSLOduration=2.0928622 podStartE2EDuration="20.629442109s" totalImagesPullingTime="18.536579909s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 21:06:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 21:06:33.627556387 +0000 UTC m=+122.206740394" watchObservedRunningTime="2026-07-27 21:06:33.629442109 +0000 UTC m=+122.208626117"
Jul 27 21:06:34.605883 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:06:34.605182    2559 generic.go:356] "Generic (PLEG): container finished" podID="1b8f587c-fd7c-44dd-903f-0eced954b117" containerID="6c1d87d22f34bb8ce7cd6bc9f23a5e7576d7d77df28dd7ad0c8b35b9f50d32ae" exitCode=0
Jul 27 21:06:34.605883 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:06:34.605328    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-6ngt5" event={"ID":"1b8f587c-fd7c-44dd-903f-0eced954b117","Type":"ContainerDied","Data":"6c1d87d22f34bb8ce7cd6bc9f23a5e7576d7d77df28dd7ad0c8b35b9f50d32ae"}
Jul 27 21:06:34.621372 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:06:34.621342    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-7c97dc4555-7x89h"
Jul 27 21:06:35.108895 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:06:35.108633    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-669984f9db-74b9w"]
Jul 27 21:06:35.141889 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:06:35.141833    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-5748766fc5-bbnsx"]
Jul 27 21:06:35.184558 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:06:35.184521    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5748766fc5-bbnsx"]
Jul 27 21:06:35.185468 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:06:35.184660    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5748766fc5-bbnsx"
Jul 27 21:06:35.203734 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:06:35.203158    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\""
Jul 27 21:06:35.258395 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:06:35.258346    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/f242f265-2dc0-467c-a8cd-f413c1d2f254-console-serving-cert\") pod \"console-5748766fc5-bbnsx\" (UID: \"f242f265-2dc0-467c-a8cd-f413c1d2f254\") " pod="openshift-console/console-5748766fc5-bbnsx"
Jul 27 21:06:35.258886 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:06:35.258817    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/f242f265-2dc0-467c-a8cd-f413c1d2f254-service-ca\") pod \"console-5748766fc5-bbnsx\" (UID: \"f242f265-2dc0-467c-a8cd-f413c1d2f254\") " pod="openshift-console/console-5748766fc5-bbnsx"
Jul 27 21:06:35.259267 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:06:35.259197    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/f242f265-2dc0-467c-a8cd-f413c1d2f254-oauth-serving-cert\") pod \"console-5748766fc5-bbnsx\" (UID: \"f242f265-2dc0-467c-a8cd-f413c1d2f254\") " pod="openshift-console/console-5748766fc5-bbnsx"
Jul 27 21:06:35.259791 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:06:35.259498    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/f242f265-2dc0-467c-a8cd-f413c1d2f254-console-config\") pod \"console-5748766fc5-bbnsx\" (UID: \"f242f265-2dc0-467c-a8cd-f413c1d2f254\") " pod="openshift-console/console-5748766fc5-bbnsx"
Jul 27 21:06:35.260112 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:06:35.260000    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wz9gz\" (UniqueName: \"kubernetes.io/projected/f242f265-2dc0-467c-a8cd-f413c1d2f254-kube-api-access-wz9gz\") pod \"console-5748766fc5-bbnsx\" (UID: \"f242f265-2dc0-467c-a8cd-f413c1d2f254\") " pod="openshift-console/console-5748766fc5-bbnsx"
Jul 27 21:06:35.260112 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:06:35.260073    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/f242f265-2dc0-467c-a8cd-f413c1d2f254-console-oauth-config\") pod \"console-5748766fc5-bbnsx\" (UID: \"f242f265-2dc0-467c-a8cd-f413c1d2f254\") " pod="openshift-console/console-5748766fc5-bbnsx"
Jul 27 21:06:35.260529 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:06:35.260460    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f242f265-2dc0-467c-a8cd-f413c1d2f254-trusted-ca-bundle\") pod \"console-5748766fc5-bbnsx\" (UID: \"f242f265-2dc0-467c-a8cd-f413c1d2f254\") " pod="openshift-console/console-5748766fc5-bbnsx"
Jul 27 21:06:35.363289 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:06:35.362022    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/f242f265-2dc0-467c-a8cd-f413c1d2f254-console-config\") pod \"console-5748766fc5-bbnsx\" (UID: \"f242f265-2dc0-467c-a8cd-f413c1d2f254\") " pod="openshift-console/console-5748766fc5-bbnsx"
Jul 27 21:06:35.363289 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:06:35.362077    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wz9gz\" (UniqueName: \"kubernetes.io/projected/f242f265-2dc0-467c-a8cd-f413c1d2f254-kube-api-access-wz9gz\") pod \"console-5748766fc5-bbnsx\" (UID: \"f242f265-2dc0-467c-a8cd-f413c1d2f254\") " pod="openshift-console/console-5748766fc5-bbnsx"
Jul 27 21:06:35.363289 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:06:35.362106    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/f242f265-2dc0-467c-a8cd-f413c1d2f254-console-oauth-config\") pod \"console-5748766fc5-bbnsx\" (UID: \"f242f265-2dc0-467c-a8cd-f413c1d2f254\") " pod="openshift-console/console-5748766fc5-bbnsx"
Jul 27 21:06:35.363289 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:06:35.362138    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f242f265-2dc0-467c-a8cd-f413c1d2f254-trusted-ca-bundle\") pod \"console-5748766fc5-bbnsx\" (UID: \"f242f265-2dc0-467c-a8cd-f413c1d2f254\") " pod="openshift-console/console-5748766fc5-bbnsx"
Jul 27 21:06:35.363289 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:06:35.362235    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/f242f265-2dc0-467c-a8cd-f413c1d2f254-console-serving-cert\") pod \"console-5748766fc5-bbnsx\" (UID: \"f242f265-2dc0-467c-a8cd-f413c1d2f254\") " pod="openshift-console/console-5748766fc5-bbnsx"
Jul 27 21:06:35.363289 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:06:35.362270    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/f242f265-2dc0-467c-a8cd-f413c1d2f254-service-ca\") pod \"console-5748766fc5-bbnsx\" (UID: \"f242f265-2dc0-467c-a8cd-f413c1d2f254\") " pod="openshift-console/console-5748766fc5-bbnsx"
Jul 27 21:06:35.363289 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:06:35.362334    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/f242f265-2dc0-467c-a8cd-f413c1d2f254-oauth-serving-cert\") pod \"console-5748766fc5-bbnsx\" (UID: \"f242f265-2dc0-467c-a8cd-f413c1d2f254\") " pod="openshift-console/console-5748766fc5-bbnsx"
Jul 27 21:06:35.363289 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:06:35.363192    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/f242f265-2dc0-467c-a8cd-f413c1d2f254-oauth-serving-cert\") pod \"console-5748766fc5-bbnsx\" (UID: \"f242f265-2dc0-467c-a8cd-f413c1d2f254\") " pod="openshift-console/console-5748766fc5-bbnsx"
Jul 27 21:06:35.365567 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:06:35.365474    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/f242f265-2dc0-467c-a8cd-f413c1d2f254-console-config\") pod \"console-5748766fc5-bbnsx\" (UID: \"f242f265-2dc0-467c-a8cd-f413c1d2f254\") " pod="openshift-console/console-5748766fc5-bbnsx"
Jul 27 21:06:35.365713 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:06:35.365578    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/f242f265-2dc0-467c-a8cd-f413c1d2f254-service-ca\") pod \"console-5748766fc5-bbnsx\" (UID: \"f242f265-2dc0-467c-a8cd-f413c1d2f254\") " pod="openshift-console/console-5748766fc5-bbnsx"
Jul 27 21:06:35.366528 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:06:35.366499    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f242f265-2dc0-467c-a8cd-f413c1d2f254-trusted-ca-bundle\") pod \"console-5748766fc5-bbnsx\" (UID: \"f242f265-2dc0-467c-a8cd-f413c1d2f254\") " pod="openshift-console/console-5748766fc5-bbnsx"
Jul 27 21:06:35.367852 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:06:35.367803    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/f242f265-2dc0-467c-a8cd-f413c1d2f254-console-serving-cert\") pod \"console-5748766fc5-bbnsx\" (UID: \"f242f265-2dc0-467c-a8cd-f413c1d2f254\") " pod="openshift-console/console-5748766fc5-bbnsx"
Jul 27 21:06:35.378421 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:06:35.377753    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/f242f265-2dc0-467c-a8cd-f413c1d2f254-console-oauth-config\") pod \"console-5748766fc5-bbnsx\" (UID: \"f242f265-2dc0-467c-a8cd-f413c1d2f254\") " pod="openshift-console/console-5748766fc5-bbnsx"
Jul 27 21:06:35.387595 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:06:35.387511    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wz9gz\" (UniqueName: \"kubernetes.io/projected/f242f265-2dc0-467c-a8cd-f413c1d2f254-kube-api-access-wz9gz\") pod \"console-5748766fc5-bbnsx\" (UID: \"f242f265-2dc0-467c-a8cd-f413c1d2f254\") " pod="openshift-console/console-5748766fc5-bbnsx"
Jul 27 21:06:35.494436 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:06:35.493730    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-5b565b454b-2kr9d"
Jul 27 21:06:35.503775 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:06:35.503743    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5748766fc5-bbnsx"
Jul 27 21:06:35.622000 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:06:35.621554    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-6ngt5" event={"ID":"1b8f587c-fd7c-44dd-903f-0eced954b117","Type":"ContainerStarted","Data":"52e68f1bef8e0d634b9e0cb67d0fd45631d909e14559be88ae322c117eb9e4e5"}
Jul 27 21:06:35.622000 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:06:35.621656    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-6ngt5" event={"ID":"1b8f587c-fd7c-44dd-903f-0eced954b117","Type":"ContainerStarted","Data":"c35d672fe58b64158a79b23e76c33e29ec72ba1ea8a699c60875193b74759af6"}
Jul 27 21:06:35.978595 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:06:35.977561    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-6ngt5" podStartSLOduration=7.492901704 podStartE2EDuration="8.977535713s" totalImagesPullingTime="1.351435589s" totalInitContainerRuntime="133.19842ms" isStatefulPod=true podCreationTimestamp="2026-07-27 21:06:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 21:06:35.647603914 +0000 UTC m=+124.226787922" watchObservedRunningTime="2026-07-27 21:06:35.977535713 +0000 UTC m=+124.556719722"
Jul 27 21:06:35.982828 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:06:35.982792    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-5b565b454b-2kr9d"]
Jul 27 21:06:38.574790 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:06:38.574755    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5748766fc5-bbnsx"]
Jul 27 21:06:38.639965 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:06:38.639390    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-686dbf5f69-zb2mj" event={"ID":"45e10f06-0e63-4eac-b3c6-42287beb9a75","Type":"ContainerStarted","Data":"54b9243226f22a949a877f4ff48f1639d3ab4d7c7185ac2fe596acfad4743eaf"}
Jul 27 21:06:38.642563 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:06:38.642488    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5748766fc5-bbnsx" event={"ID":"f242f265-2dc0-467c-a8cd-f413c1d2f254","Type":"ContainerStarted","Data":"2331eb37e15ae8fce246ac296c8eac2ceed5e477f3f4413f4b19b399d79e6ef2"}
Jul 27 21:06:38.648208 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:06:38.648096    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-64dfb7c55b-blsbx" event={"ID":"3ba37884-dc16-419a-b9ca-ba49c6171c9c","Type":"ContainerStarted","Data":"8cc1fcda15fbc9b9fe28ac84d34dafda6604642872974a27143acf8b31dd0353"}
Jul 27 21:06:38.654781 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:06:38.654472    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-68b7fc9574-gmwds" event={"ID":"fbf8db18-7974-4032-b919-6be505ffe0e1","Type":"ContainerStarted","Data":"134320f9203c9a058e4df9fd1d840eaf864b35fa1ca444f09889dc39db7c0122"}
Jul 27 21:06:38.654781 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:06:38.654548    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-68b7fc9574-gmwds"
Jul 27 21:06:38.658724 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:06:38.658695    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-56d9f944b4-dbv9r" event={"ID":"7854d704-78c9-468b-9973-8799da0fe0eb","Type":"ContainerStarted","Data":"db9191391def9585e641672a6e80f62dff0274df2f5caf8739158e5420798a30"}
Jul 27 21:06:38.660128 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:06:38.660074    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-686dbf5f69-zb2mj" podStartSLOduration=2.490590802 podStartE2EDuration="7.660056472s" totalImagesPullingTime="5.16946567s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 21:06:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 21:06:38.65972619 +0000 UTC m=+127.238910197" watchObservedRunningTime="2026-07-27 21:06:38.660056472 +0000 UTC m=+127.239240480"
Jul 27 21:06:38.663661 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:06:38.663582    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-68b7fc9574-gmwds"
Jul 27 21:06:38.678387 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:06:38.677984    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-68b7fc9574-gmwds" podStartSLOduration=2.4539920410000002 podStartE2EDuration="7.677958875s" totalImagesPullingTime="5.223966834s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 21:06:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 21:06:38.675818871 +0000 UTC m=+127.255002881" watchObservedRunningTime="2026-07-27 21:06:38.677958875 +0000 UTC m=+127.257142881"
Jul 27 21:06:39.670143 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:06:39.670104    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-64dfb7c55b-blsbx" event={"ID":"3ba37884-dc16-419a-b9ca-ba49c6171c9c","Type":"ContainerStarted","Data":"17099a16430d5b35e37b6a4035e88a000d316d5080a63b7e194a81e492510023"}
Jul 27 21:06:39.670762 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:06:39.670240    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-64dfb7c55b-blsbx" event={"ID":"3ba37884-dc16-419a-b9ca-ba49c6171c9c","Type":"ContainerStarted","Data":"659ee8670bc989fad5b6a4bbe74ab2022db5d8925462dad38f0b399860280ea0"}
Jul 27 21:06:39.673769 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:06:39.673670    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-56d9f944b4-dbv9r" event={"ID":"7854d704-78c9-468b-9973-8799da0fe0eb","Type":"ContainerStarted","Data":"c1426c7b3a916a7650b532663adeba58e85c45a8c9b9f2fbfb0770e1c538b528"}
Jul 27 21:06:39.673769 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:06:39.673734    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-56d9f944b4-dbv9r" event={"ID":"7854d704-78c9-468b-9973-8799da0fe0eb","Type":"ContainerStarted","Data":"6add945f404d080bf270342d49318a2ae480a657f89b44740dba586586c2d9ea"}
Jul 27 21:06:39.676090 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:06:39.675995    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5748766fc5-bbnsx" event={"ID":"f242f265-2dc0-467c-a8cd-f413c1d2f254","Type":"ContainerStarted","Data":"5004543d95b694e2413d55c313acd1c8e27ed613b607650bb65bd34a44c1e821"}
Jul 27 21:06:39.697392 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:06:39.697181    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-56d9f944b4-dbv9r" podStartSLOduration=7.527968896 podStartE2EDuration="12.697160903s" totalImagesPullingTime="5.169192007s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 21:06:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 21:06:39.69481218 +0000 UTC m=+128.273996189" watchObservedRunningTime="2026-07-27 21:06:39.697160903 +0000 UTC m=+128.276344914"
Jul 27 21:06:40.684073 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:06:40.683906    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-64dfb7c55b-blsbx" event={"ID":"3ba37884-dc16-419a-b9ca-ba49c6171c9c","Type":"ContainerStarted","Data":"13c3402db9cca31594d3979e6d618be1beae7bc68bcf869f34f43f0878169a0e"}
Jul 27 21:06:40.684073 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:06:40.683996    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-64dfb7c55b-blsbx" event={"ID":"3ba37884-dc16-419a-b9ca-ba49c6171c9c","Type":"ContainerStarted","Data":"e37fbd3e29f278806212f1c0caedbca73a880d23a2b8418387b9b41e8d2b41d4"}
Jul 27 21:06:40.684073 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:06:40.684041    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-64dfb7c55b-blsbx" event={"ID":"3ba37884-dc16-419a-b9ca-ba49c6171c9c","Type":"ContainerStarted","Data":"2c4e5b5c8bd19112ae60eddb771e7281937dfab82830bf0fd0750d453b60c59d"}
Jul 27 21:06:40.684708 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:06:40.684395    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-64dfb7c55b-blsbx"
Jul 27 21:06:40.707815 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:06:40.707757    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-64dfb7c55b-blsbx" podStartSLOduration=4.17350253 podStartE2EDuration="10.707736708s" totalImagesPullingTime="6.534234178s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 21:06:30 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 21:06:40.706672109 +0000 UTC m=+129.285856141" watchObservedRunningTime="2026-07-27 21:06:40.707736708 +0000 UTC m=+129.286920718"
Jul 27 21:06:40.708884 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:06:40.708838    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-5748766fc5-bbnsx" podStartSLOduration=5.708827094 podStartE2EDuration="5.708827094s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 21:06:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 21:06:39.718208709 +0000 UTC m=+128.297392718" watchObservedRunningTime="2026-07-27 21:06:40.708827094 +0000 UTC m=+129.288011102"
Jul 27 21:06:41.833720 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:06:41.833675    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5851de39-848d-4a52-bc17-fefe1bf5407a-metrics-certs\") pod \"network-metrics-daemon-pr4c2\" (UID: \"5851de39-848d-4a52-bc17-fefe1bf5407a\") " pod="openshift-multus/network-metrics-daemon-pr4c2"
Jul 27 21:06:41.836606 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:06:41.836574    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5851de39-848d-4a52-bc17-fefe1bf5407a-metrics-certs\") pod \"network-metrics-daemon-pr4c2\" (UID: \"5851de39-848d-4a52-bc17-fefe1bf5407a\") " pod="openshift-multus/network-metrics-daemon-pr4c2"
Jul 27 21:06:41.991517 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:06:41.991488    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-vs64j\""
Jul 27 21:06:41.998517 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:06:41.998486    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pr4c2"
Jul 27 21:06:42.189422 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:06:42.189399    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-pr4c2"]
Jul 27 21:06:42.191258 ip-10-0-131-133 kubenswrapper[2559]: W0727 21:06:42.191229    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5851de39_848d_4a52_bc17_fefe1bf5407a.slice/crio-f884d873ba0ae3e7af286758ef67ab4edfa7f0ccb9e2e6a1e2f0e25f24079cb1 WatchSource:0}: Error finding container f884d873ba0ae3e7af286758ef67ab4edfa7f0ccb9e2e6a1e2f0e25f24079cb1: Status 404 returned error can't find the container with id f884d873ba0ae3e7af286758ef67ab4edfa7f0ccb9e2e6a1e2f0e25f24079cb1
Jul 27 21:06:42.710543 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:06:42.710508    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-pr4c2" event={"ID":"5851de39-848d-4a52-bc17-fefe1bf5407a","Type":"ContainerStarted","Data":"f884d873ba0ae3e7af286758ef67ab4edfa7f0ccb9e2e6a1e2f0e25f24079cb1"}
Jul 27 21:06:43.716384 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:06:43.716356    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-pr4c2" event={"ID":"5851de39-848d-4a52-bc17-fefe1bf5407a","Type":"ContainerStarted","Data":"e1297cfd517493e7645648a95c6e1cac4de477c30187bb1698fd6c525b6f2acc"}
Jul 27 21:06:44.722062 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:06:44.722027    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-pr4c2" event={"ID":"5851de39-848d-4a52-bc17-fefe1bf5407a","Type":"ContainerStarted","Data":"e06b87847ec523d2399413bf09c5d297ffed8fa55ebee9d086bb8cb45991f5e8"}
Jul 27 21:06:44.739226 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:06:44.739166    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-pr4c2" podStartSLOduration=131.460611489 podStartE2EDuration="2m12.73914814s" totalImagesPullingTime="1.278536651s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 21:04:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 21:06:44.738518728 +0000 UTC m=+133.317702738" watchObservedRunningTime="2026-07-27 21:06:44.73914814 +0000 UTC m=+133.318332180"
Jul 27 21:06:45.504882 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:06:45.504842    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-5748766fc5-bbnsx"
Jul 27 21:06:45.505099 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:06:45.504910    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-5748766fc5-bbnsx"
Jul 27 21:06:45.510083 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:06:45.510057    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-5748766fc5-bbnsx"
Jul 27 21:06:45.729373 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:06:45.729343    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-5748766fc5-bbnsx"
Jul 27 21:06:46.709132 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:06:46.709106    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-64dfb7c55b-blsbx"
Jul 27 21:06:50.745295 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:06:50.745261    2559 generic.go:356] "Generic (PLEG): container finished" podID="9e560b08-69f7-4ee3-8236-22a5c398a476" containerID="497e36afe3a388209e358dbc69ab906fffe45bb0712b69ebf4dba141ec699ca9" exitCode=0
Jul 27 21:06:50.745758 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:06:50.745317    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-9d4zq" event={"ID":"9e560b08-69f7-4ee3-8236-22a5c398a476","Type":"ContainerDied","Data":"497e36afe3a388209e358dbc69ab906fffe45bb0712b69ebf4dba141ec699ca9"}
Jul 27 21:06:50.745758 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:06:50.745645    2559 scope.go:117] "RemoveContainer" containerID="497e36afe3a388209e358dbc69ab906fffe45bb0712b69ebf4dba141ec699ca9"
Jul 27 21:06:51.751490 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:06:51.751455    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-9d4zq" event={"ID":"9e560b08-69f7-4ee3-8236-22a5c398a476","Type":"ContainerStarted","Data":"c3c50cbdcaee4dfccbc62c734f6b3c5f04b3bc5af3248a710f3839d69815026b"}
Jul 27 21:06:52.023439 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:06:52.023345    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/metrics-server-686dbf5f69-zb2mj"
Jul 27 21:06:52.023439 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:06:52.023415    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/metrics-server-686dbf5f69-zb2mj"
Jul 27 21:07:00.140642 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:07:00.140601    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-669984f9db-74b9w" podUID="f93aded2-05e8-4d10-8485-630a97a87fcf" containerName="console" containerID="cri-o://321705e386693335094cbd1c936a6103cd8fa771f828c51d967d4a0672ed76b5" gracePeriod=15
Jul 27 21:07:00.424475 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:07:00.424446    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-669984f9db-74b9w"
Jul 27 21:07:00.494197 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:07:00.494165    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/f93aded2-05e8-4d10-8485-630a97a87fcf-service-ca\") pod \"f93aded2-05e8-4d10-8485-630a97a87fcf\" (UID: \"f93aded2-05e8-4d10-8485-630a97a87fcf\") "
Jul 27 21:07:00.494197 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:07:00.494204    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/f93aded2-05e8-4d10-8485-630a97a87fcf-console-config\") pod \"f93aded2-05e8-4d10-8485-630a97a87fcf\" (UID: \"f93aded2-05e8-4d10-8485-630a97a87fcf\") "
Jul 27 21:07:00.494473 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:07:00.494276    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/f93aded2-05e8-4d10-8485-630a97a87fcf-console-serving-cert\") pod \"f93aded2-05e8-4d10-8485-630a97a87fcf\" (UID: \"f93aded2-05e8-4d10-8485-630a97a87fcf\") "
Jul 27 21:07:00.494473 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:07:00.494293    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/f93aded2-05e8-4d10-8485-630a97a87fcf-oauth-serving-cert\") pod \"f93aded2-05e8-4d10-8485-630a97a87fcf\" (UID: \"f93aded2-05e8-4d10-8485-630a97a87fcf\") "
Jul 27 21:07:00.494473 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:07:00.494345    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/f93aded2-05e8-4d10-8485-630a97a87fcf-console-oauth-config\") pod \"f93aded2-05e8-4d10-8485-630a97a87fcf\" (UID: \"f93aded2-05e8-4d10-8485-630a97a87fcf\") "
Jul 27 21:07:00.494473 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:07:00.494461    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4lh67\" (UniqueName: \"kubernetes.io/projected/f93aded2-05e8-4d10-8485-630a97a87fcf-kube-api-access-4lh67\") pod \"f93aded2-05e8-4d10-8485-630a97a87fcf\" (UID: \"f93aded2-05e8-4d10-8485-630a97a87fcf\") "
Jul 27 21:07:00.494670 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:07:00.494623    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f93aded2-05e8-4d10-8485-630a97a87fcf-service-ca" (OuterVolumeSpecName: "service-ca") pod "f93aded2-05e8-4d10-8485-630a97a87fcf" (UID: "f93aded2-05e8-4d10-8485-630a97a87fcf"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 21:07:00.494728 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:07:00.494666    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f93aded2-05e8-4d10-8485-630a97a87fcf-console-config" (OuterVolumeSpecName: "console-config") pod "f93aded2-05e8-4d10-8485-630a97a87fcf" (UID: "f93aded2-05e8-4d10-8485-630a97a87fcf"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 21:07:00.494728 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:07:00.494706    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f93aded2-05e8-4d10-8485-630a97a87fcf-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "f93aded2-05e8-4d10-8485-630a97a87fcf" (UID: "f93aded2-05e8-4d10-8485-630a97a87fcf"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 21:07:00.494833 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:07:00.494787    2559 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/f93aded2-05e8-4d10-8485-630a97a87fcf-service-ca\") on node \"ip-10-0-131-133.ec2.internal\" DevicePath \"\""
Jul 27 21:07:00.494833 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:07:00.494803    2559 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/f93aded2-05e8-4d10-8485-630a97a87fcf-console-config\") on node \"ip-10-0-131-133.ec2.internal\" DevicePath \"\""
Jul 27 21:07:00.494833 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:07:00.494812    2559 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/f93aded2-05e8-4d10-8485-630a97a87fcf-oauth-serving-cert\") on node \"ip-10-0-131-133.ec2.internal\" DevicePath \"\""
Jul 27 21:07:00.496694 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:07:00.496664    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f93aded2-05e8-4d10-8485-630a97a87fcf-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "f93aded2-05e8-4d10-8485-630a97a87fcf" (UID: "f93aded2-05e8-4d10-8485-630a97a87fcf"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 21:07:00.496962 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:07:00.496938    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f93aded2-05e8-4d10-8485-630a97a87fcf-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "f93aded2-05e8-4d10-8485-630a97a87fcf" (UID: "f93aded2-05e8-4d10-8485-630a97a87fcf"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 21:07:00.497034 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:07:00.496937    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f93aded2-05e8-4d10-8485-630a97a87fcf-kube-api-access-4lh67" (OuterVolumeSpecName: "kube-api-access-4lh67") pod "f93aded2-05e8-4d10-8485-630a97a87fcf" (UID: "f93aded2-05e8-4d10-8485-630a97a87fcf"). InnerVolumeSpecName "kube-api-access-4lh67". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 21:07:00.595243 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:07:00.595194    2559 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/f93aded2-05e8-4d10-8485-630a97a87fcf-console-serving-cert\") on node \"ip-10-0-131-133.ec2.internal\" DevicePath \"\""
Jul 27 21:07:00.595243 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:07:00.595240    2559 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/f93aded2-05e8-4d10-8485-630a97a87fcf-console-oauth-config\") on node \"ip-10-0-131-133.ec2.internal\" DevicePath \"\""
Jul 27 21:07:00.595243 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:07:00.595251    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4lh67\" (UniqueName: \"kubernetes.io/projected/f93aded2-05e8-4d10-8485-630a97a87fcf-kube-api-access-4lh67\") on node \"ip-10-0-131-133.ec2.internal\" DevicePath \"\""
Jul 27 21:07:00.789322 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:07:00.789205    2559 generic.go:356] "Generic (PLEG): container finished" podID="f93aded2-05e8-4d10-8485-630a97a87fcf" containerID="321705e386693335094cbd1c936a6103cd8fa771f828c51d967d4a0672ed76b5" exitCode=0
Jul 27 21:07:00.789322 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:07:00.789260    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-669984f9db-74b9w" event={"ID":"f93aded2-05e8-4d10-8485-630a97a87fcf","Type":"ContainerDied","Data":"321705e386693335094cbd1c936a6103cd8fa771f828c51d967d4a0672ed76b5"}
Jul 27 21:07:00.789322 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:07:00.789307    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-669984f9db-74b9w"
Jul 27 21:07:00.789590 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:07:00.789346    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-669984f9db-74b9w" event={"ID":"f93aded2-05e8-4d10-8485-630a97a87fcf","Type":"ContainerDied","Data":"fbc6de46ab01d17d480bbfe1babd1ff6ad4d7b924a5ac3ea722709a189b04a0a"}
Jul 27 21:07:00.789590 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:07:00.789367    2559 scope.go:117] "RemoveContainer" containerID="321705e386693335094cbd1c936a6103cd8fa771f828c51d967d4a0672ed76b5"
Jul 27 21:07:00.799294 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:07:00.799269    2559 scope.go:117] "RemoveContainer" containerID="321705e386693335094cbd1c936a6103cd8fa771f828c51d967d4a0672ed76b5"
Jul 27 21:07:00.799607 ip-10-0-131-133 kubenswrapper[2559]: E0727 21:07:00.799583    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"321705e386693335094cbd1c936a6103cd8fa771f828c51d967d4a0672ed76b5\": container with ID starting with 321705e386693335094cbd1c936a6103cd8fa771f828c51d967d4a0672ed76b5 not found: ID does not exist" containerID="321705e386693335094cbd1c936a6103cd8fa771f828c51d967d4a0672ed76b5"
Jul 27 21:07:00.799697 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:07:00.799613    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"321705e386693335094cbd1c936a6103cd8fa771f828c51d967d4a0672ed76b5"} err="failed to get container status \"321705e386693335094cbd1c936a6103cd8fa771f828c51d967d4a0672ed76b5\": rpc error: code = NotFound desc = could not find container \"321705e386693335094cbd1c936a6103cd8fa771f828c51d967d4a0672ed76b5\": container with ID starting with 321705e386693335094cbd1c936a6103cd8fa771f828c51d967d4a0672ed76b5 not found: ID does not exist"
Jul 27 21:07:00.820895 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:07:00.820867    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-669984f9db-74b9w"]
Jul 27 21:07:00.824196 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:07:00.824169    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-669984f9db-74b9w"]
Jul 27 21:07:01.013960 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:07:01.013918    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-image-registry/image-registry-5b565b454b-2kr9d" podUID="097262d9-cf78-4d89-98cc-cf92d23a4f04" containerName="registry" containerID="cri-o://8b1d854e5710d1523faad73121e9b3b88250fc3dabcfc04c09cb2d56430fd0ed" gracePeriod=30
Jul 27 21:07:01.295955 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:07:01.295922    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5b565b454b-2kr9d"
Jul 27 21:07:01.298058 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:07:01.298032    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/097262d9-cf78-4d89-98cc-cf92d23a4f04-ca-trust-extracted\") pod \"097262d9-cf78-4d89-98cc-cf92d23a4f04\" (UID: \"097262d9-cf78-4d89-98cc-cf92d23a4f04\") "
Jul 27 21:07:01.298192 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:07:01.298072    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/097262d9-cf78-4d89-98cc-cf92d23a4f04-image-registry-private-configuration\") pod \"097262d9-cf78-4d89-98cc-cf92d23a4f04\" (UID: \"097262d9-cf78-4d89-98cc-cf92d23a4f04\") "
Jul 27 21:07:01.298192 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:07:01.298103    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/097262d9-cf78-4d89-98cc-cf92d23a4f04-installation-pull-secrets\") pod \"097262d9-cf78-4d89-98cc-cf92d23a4f04\" (UID: \"097262d9-cf78-4d89-98cc-cf92d23a4f04\") "
Jul 27 21:07:01.298192 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:07:01.298136    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/097262d9-cf78-4d89-98cc-cf92d23a4f04-trusted-ca\") pod \"097262d9-cf78-4d89-98cc-cf92d23a4f04\" (UID: \"097262d9-cf78-4d89-98cc-cf92d23a4f04\") "
Jul 27 21:07:01.298192 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:07:01.298159    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/097262d9-cf78-4d89-98cc-cf92d23a4f04-registry-tls\") pod \"097262d9-cf78-4d89-98cc-cf92d23a4f04\" (UID: \"097262d9-cf78-4d89-98cc-cf92d23a4f04\") "
Jul 27 21:07:01.298192 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:07:01.298192    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cjjp7\" (UniqueName: \"kubernetes.io/projected/097262d9-cf78-4d89-98cc-cf92d23a4f04-kube-api-access-cjjp7\") pod \"097262d9-cf78-4d89-98cc-cf92d23a4f04\" (UID: \"097262d9-cf78-4d89-98cc-cf92d23a4f04\") "
Jul 27 21:07:01.298481 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:07:01.298263    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/097262d9-cf78-4d89-98cc-cf92d23a4f04-registry-certificates\") pod \"097262d9-cf78-4d89-98cc-cf92d23a4f04\" (UID: \"097262d9-cf78-4d89-98cc-cf92d23a4f04\") "
Jul 27 21:07:01.298481 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:07:01.298354    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/097262d9-cf78-4d89-98cc-cf92d23a4f04-bound-sa-token\") pod \"097262d9-cf78-4d89-98cc-cf92d23a4f04\" (UID: \"097262d9-cf78-4d89-98cc-cf92d23a4f04\") "
Jul 27 21:07:01.298620 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:07:01.298592    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/097262d9-cf78-4d89-98cc-cf92d23a4f04-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "097262d9-cf78-4d89-98cc-cf92d23a4f04" (UID: "097262d9-cf78-4d89-98cc-cf92d23a4f04"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 21:07:01.299068 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:07:01.298981    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/097262d9-cf78-4d89-98cc-cf92d23a4f04-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "097262d9-cf78-4d89-98cc-cf92d23a4f04" (UID: "097262d9-cf78-4d89-98cc-cf92d23a4f04"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 21:07:01.301226 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:07:01.301176    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/097262d9-cf78-4d89-98cc-cf92d23a4f04-kube-api-access-cjjp7" (OuterVolumeSpecName: "kube-api-access-cjjp7") pod "097262d9-cf78-4d89-98cc-cf92d23a4f04" (UID: "097262d9-cf78-4d89-98cc-cf92d23a4f04"). InnerVolumeSpecName "kube-api-access-cjjp7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 21:07:01.301226 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:07:01.301193    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/097262d9-cf78-4d89-98cc-cf92d23a4f04-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "097262d9-cf78-4d89-98cc-cf92d23a4f04" (UID: "097262d9-cf78-4d89-98cc-cf92d23a4f04"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 21:07:01.301359 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:07:01.301246    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/097262d9-cf78-4d89-98cc-cf92d23a4f04-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "097262d9-cf78-4d89-98cc-cf92d23a4f04" (UID: "097262d9-cf78-4d89-98cc-cf92d23a4f04"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 21:07:01.301359 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:07:01.301279    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/097262d9-cf78-4d89-98cc-cf92d23a4f04-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "097262d9-cf78-4d89-98cc-cf92d23a4f04" (UID: "097262d9-cf78-4d89-98cc-cf92d23a4f04"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 21:07:01.301359 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:07:01.301331    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/097262d9-cf78-4d89-98cc-cf92d23a4f04-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "097262d9-cf78-4d89-98cc-cf92d23a4f04" (UID: "097262d9-cf78-4d89-98cc-cf92d23a4f04"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 21:07:01.307911 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:07:01.307863    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/097262d9-cf78-4d89-98cc-cf92d23a4f04-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "097262d9-cf78-4d89-98cc-cf92d23a4f04" (UID: "097262d9-cf78-4d89-98cc-cf92d23a4f04"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 21:07:01.399615 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:07:01.399579    2559 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/097262d9-cf78-4d89-98cc-cf92d23a4f04-bound-sa-token\") on node \"ip-10-0-131-133.ec2.internal\" DevicePath \"\""
Jul 27 21:07:01.399615 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:07:01.399608    2559 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/097262d9-cf78-4d89-98cc-cf92d23a4f04-ca-trust-extracted\") on node \"ip-10-0-131-133.ec2.internal\" DevicePath \"\""
Jul 27 21:07:01.399615 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:07:01.399620    2559 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/097262d9-cf78-4d89-98cc-cf92d23a4f04-image-registry-private-configuration\") on node \"ip-10-0-131-133.ec2.internal\" DevicePath \"\""
Jul 27 21:07:01.399916 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:07:01.399633    2559 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/097262d9-cf78-4d89-98cc-cf92d23a4f04-installation-pull-secrets\") on node \"ip-10-0-131-133.ec2.internal\" DevicePath \"\""
Jul 27 21:07:01.399916 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:07:01.399645    2559 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/097262d9-cf78-4d89-98cc-cf92d23a4f04-trusted-ca\") on node \"ip-10-0-131-133.ec2.internal\" DevicePath \"\""
Jul 27 21:07:01.399916 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:07:01.399657    2559 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/097262d9-cf78-4d89-98cc-cf92d23a4f04-registry-tls\") on node \"ip-10-0-131-133.ec2.internal\" DevicePath \"\""
Jul 27 21:07:01.399916 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:07:01.399666    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cjjp7\" (UniqueName: \"kubernetes.io/projected/097262d9-cf78-4d89-98cc-cf92d23a4f04-kube-api-access-cjjp7\") on node \"ip-10-0-131-133.ec2.internal\" DevicePath \"\""
Jul 27 21:07:01.399916 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:07:01.399674    2559 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/097262d9-cf78-4d89-98cc-cf92d23a4f04-registry-certificates\") on node \"ip-10-0-131-133.ec2.internal\" DevicePath \"\""
Jul 27 21:07:01.795841 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:07:01.795808    2559 generic.go:356] "Generic (PLEG): container finished" podID="097262d9-cf78-4d89-98cc-cf92d23a4f04" containerID="8b1d854e5710d1523faad73121e9b3b88250fc3dabcfc04c09cb2d56430fd0ed" exitCode=0
Jul 27 21:07:01.796033 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:07:01.795860    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5b565b454b-2kr9d"
Jul 27 21:07:01.796033 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:07:01.795882    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5b565b454b-2kr9d" event={"ID":"097262d9-cf78-4d89-98cc-cf92d23a4f04","Type":"ContainerDied","Data":"8b1d854e5710d1523faad73121e9b3b88250fc3dabcfc04c09cb2d56430fd0ed"}
Jul 27 21:07:01.796033 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:07:01.795941    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5b565b454b-2kr9d" event={"ID":"097262d9-cf78-4d89-98cc-cf92d23a4f04","Type":"ContainerDied","Data":"b00495faa9d4af48eddb7c7598d8d1c16d98b4d05ebdef3c65b70e113cf88bc7"}
Jul 27 21:07:01.796033 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:07:01.795969    2559 scope.go:117] "RemoveContainer" containerID="8b1d854e5710d1523faad73121e9b3b88250fc3dabcfc04c09cb2d56430fd0ed"
Jul 27 21:07:01.804864 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:07:01.804843    2559 scope.go:117] "RemoveContainer" containerID="8b1d854e5710d1523faad73121e9b3b88250fc3dabcfc04c09cb2d56430fd0ed"
Jul 27 21:07:01.805125 ip-10-0-131-133 kubenswrapper[2559]: E0727 21:07:01.805108    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8b1d854e5710d1523faad73121e9b3b88250fc3dabcfc04c09cb2d56430fd0ed\": container with ID starting with 8b1d854e5710d1523faad73121e9b3b88250fc3dabcfc04c09cb2d56430fd0ed not found: ID does not exist" containerID="8b1d854e5710d1523faad73121e9b3b88250fc3dabcfc04c09cb2d56430fd0ed"
Jul 27 21:07:01.805163 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:07:01.805134    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8b1d854e5710d1523faad73121e9b3b88250fc3dabcfc04c09cb2d56430fd0ed"} err="failed to get container status \"8b1d854e5710d1523faad73121e9b3b88250fc3dabcfc04c09cb2d56430fd0ed\": rpc error: code = NotFound desc = could not find container \"8b1d854e5710d1523faad73121e9b3b88250fc3dabcfc04c09cb2d56430fd0ed\": container with ID starting with 8b1d854e5710d1523faad73121e9b3b88250fc3dabcfc04c09cb2d56430fd0ed not found: ID does not exist"
Jul 27 21:07:01.817208 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:07:01.817175    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-5b565b454b-2kr9d"]
Jul 27 21:07:01.820991 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:07:01.820965    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-5b565b454b-2kr9d"]
Jul 27 21:07:01.980962 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:07:01.980927    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="097262d9-cf78-4d89-98cc-cf92d23a4f04" path="/var/lib/kubelet/pods/097262d9-cf78-4d89-98cc-cf92d23a4f04/volumes"
Jul 27 21:07:01.981329 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:07:01.981314    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f93aded2-05e8-4d10-8485-630a97a87fcf" path="/var/lib/kubelet/pods/f93aded2-05e8-4d10-8485-630a97a87fcf/volumes"
Jul 27 21:07:12.028915 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:07:12.028885    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/metrics-server-686dbf5f69-zb2mj"
Jul 27 21:07:12.032822 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:07:12.032793    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-686dbf5f69-zb2mj"
Jul 27 21:07:49.030352 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:07:49.030314    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 27 21:07:49.030801 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:07:49.030621    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f93aded2-05e8-4d10-8485-630a97a87fcf" containerName="console"
Jul 27 21:07:49.030801 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:07:49.030632    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="f93aded2-05e8-4d10-8485-630a97a87fcf" containerName="console"
Jul 27 21:07:49.030801 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:07:49.030641    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="097262d9-cf78-4d89-98cc-cf92d23a4f04" containerName="registry"
Jul 27 21:07:49.030801 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:07:49.030646    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="097262d9-cf78-4d89-98cc-cf92d23a4f04" containerName="registry"
Jul 27 21:07:49.030801 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:07:49.030704    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="097262d9-cf78-4d89-98cc-cf92d23a4f04" containerName="registry"
Jul 27 21:07:49.030801 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:07:49.030713    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="f93aded2-05e8-4d10-8485-630a97a87fcf" containerName="console"
Jul 27 21:07:49.034268 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:07:49.034251    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Jul 27 21:07:49.037372 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:07:49.037348    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
Jul 27 21:07:49.038233 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:07:49.038190    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
Jul 27 21:07:49.038584 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:07:49.038195    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
Jul 27 21:07:49.038584 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:07:49.038195    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
Jul 27 21:07:49.038584 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:07:49.038314    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
Jul 27 21:07:49.038584 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:07:49.038358    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-qbqch\""
Jul 27 21:07:49.038584 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:07:49.038356    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
Jul 27 21:07:49.038839 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:07:49.038647    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
Jul 27 21:07:49.038839 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:07:49.038675    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
Jul 27 21:07:49.044814 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:07:49.044797    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
Jul 27 21:07:49.054005 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:07:49.053983    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 27 21:07:49.107559 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:07:49.107523    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/364d8616-829c-4e2c-8531-cf977166dbf3-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"364d8616-829c-4e2c-8531-cf977166dbf3\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 21:07:49.107559 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:07:49.107558    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/364d8616-829c-4e2c-8531-cf977166dbf3-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"364d8616-829c-4e2c-8531-cf977166dbf3\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 21:07:49.107765 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:07:49.107575    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nx94d\" (UniqueName: \"kubernetes.io/projected/364d8616-829c-4e2c-8531-cf977166dbf3-kube-api-access-nx94d\") pod \"alertmanager-main-0\" (UID: \"364d8616-829c-4e2c-8531-cf977166dbf3\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 21:07:49.107765 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:07:49.107659    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/364d8616-829c-4e2c-8531-cf977166dbf3-tls-assets\") pod \"alertmanager-main-0\" (UID: \"364d8616-829c-4e2c-8531-cf977166dbf3\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 21:07:49.107765 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:07:49.107685    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/364d8616-829c-4e2c-8531-cf977166dbf3-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"364d8616-829c-4e2c-8531-cf977166dbf3\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 21:07:49.107765 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:07:49.107716    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/364d8616-829c-4e2c-8531-cf977166dbf3-web-config\") pod \"alertmanager-main-0\" (UID: \"364d8616-829c-4e2c-8531-cf977166dbf3\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 21:07:49.107765 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:07:49.107756    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/364d8616-829c-4e2c-8531-cf977166dbf3-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"364d8616-829c-4e2c-8531-cf977166dbf3\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 21:07:49.107920 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:07:49.107773    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/364d8616-829c-4e2c-8531-cf977166dbf3-config-volume\") pod \"alertmanager-main-0\" (UID: \"364d8616-829c-4e2c-8531-cf977166dbf3\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 21:07:49.107920 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:07:49.107790    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/364d8616-829c-4e2c-8531-cf977166dbf3-config-out\") pod \"alertmanager-main-0\" (UID: \"364d8616-829c-4e2c-8531-cf977166dbf3\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 21:07:49.107920 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:07:49.107817    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/364d8616-829c-4e2c-8531-cf977166dbf3-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"364d8616-829c-4e2c-8531-cf977166dbf3\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 21:07:49.107920 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:07:49.107849    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/364d8616-829c-4e2c-8531-cf977166dbf3-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"364d8616-829c-4e2c-8531-cf977166dbf3\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 21:07:49.107920 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:07:49.107868    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/364d8616-829c-4e2c-8531-cf977166dbf3-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"364d8616-829c-4e2c-8531-cf977166dbf3\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 21:07:49.107920 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:07:49.107890    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/364d8616-829c-4e2c-8531-cf977166dbf3-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"364d8616-829c-4e2c-8531-cf977166dbf3\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 21:07:49.208414 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:07:49.208378    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/364d8616-829c-4e2c-8531-cf977166dbf3-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"364d8616-829c-4e2c-8531-cf977166dbf3\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 21:07:49.208414 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:07:49.208413    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/364d8616-829c-4e2c-8531-cf977166dbf3-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"364d8616-829c-4e2c-8531-cf977166dbf3\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 21:07:49.208660 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:07:49.208435    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nx94d\" (UniqueName: \"kubernetes.io/projected/364d8616-829c-4e2c-8531-cf977166dbf3-kube-api-access-nx94d\") pod \"alertmanager-main-0\" (UID: \"364d8616-829c-4e2c-8531-cf977166dbf3\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 21:07:49.208660 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:07:49.208465    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/364d8616-829c-4e2c-8531-cf977166dbf3-tls-assets\") pod \"alertmanager-main-0\" (UID: \"364d8616-829c-4e2c-8531-cf977166dbf3\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 21:07:49.208660 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:07:49.208482    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/364d8616-829c-4e2c-8531-cf977166dbf3-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"364d8616-829c-4e2c-8531-cf977166dbf3\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 21:07:49.208660 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:07:49.208507    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/364d8616-829c-4e2c-8531-cf977166dbf3-web-config\") pod \"alertmanager-main-0\" (UID: \"364d8616-829c-4e2c-8531-cf977166dbf3\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 21:07:49.208660 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:07:49.208616    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/364d8616-829c-4e2c-8531-cf977166dbf3-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"364d8616-829c-4e2c-8531-cf977166dbf3\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 21:07:49.208660 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:07:49.208653    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/364d8616-829c-4e2c-8531-cf977166dbf3-config-volume\") pod \"alertmanager-main-0\" (UID: \"364d8616-829c-4e2c-8531-cf977166dbf3\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 21:07:49.208979 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:07:49.208686    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/364d8616-829c-4e2c-8531-cf977166dbf3-config-out\") pod \"alertmanager-main-0\" (UID: \"364d8616-829c-4e2c-8531-cf977166dbf3\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 21:07:49.208979 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:07:49.208716    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/364d8616-829c-4e2c-8531-cf977166dbf3-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"364d8616-829c-4e2c-8531-cf977166dbf3\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 21:07:49.208979 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:07:49.208753    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/364d8616-829c-4e2c-8531-cf977166dbf3-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"364d8616-829c-4e2c-8531-cf977166dbf3\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 21:07:49.208979 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:07:49.208783    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/364d8616-829c-4e2c-8531-cf977166dbf3-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"364d8616-829c-4e2c-8531-cf977166dbf3\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 21:07:49.208979 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:07:49.208820    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/364d8616-829c-4e2c-8531-cf977166dbf3-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"364d8616-829c-4e2c-8531-cf977166dbf3\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 21:07:49.209261 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:07:49.209047    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/364d8616-829c-4e2c-8531-cf977166dbf3-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"364d8616-829c-4e2c-8531-cf977166dbf3\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 21:07:49.209261 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:07:49.209207    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/364d8616-829c-4e2c-8531-cf977166dbf3-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"364d8616-829c-4e2c-8531-cf977166dbf3\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 21:07:49.211797 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:07:49.211632    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/364d8616-829c-4e2c-8531-cf977166dbf3-config-out\") pod \"alertmanager-main-0\" (UID: \"364d8616-829c-4e2c-8531-cf977166dbf3\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 21:07:49.211797 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:07:49.211641    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/364d8616-829c-4e2c-8531-cf977166dbf3-tls-assets\") pod \"alertmanager-main-0\" (UID: \"364d8616-829c-4e2c-8531-cf977166dbf3\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 21:07:49.211797 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:07:49.211754    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/364d8616-829c-4e2c-8531-cf977166dbf3-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"364d8616-829c-4e2c-8531-cf977166dbf3\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 21:07:49.211797 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:07:49.211780    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/364d8616-829c-4e2c-8531-cf977166dbf3-web-config\") pod \"alertmanager-main-0\" (UID: \"364d8616-829c-4e2c-8531-cf977166dbf3\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 21:07:49.212092 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:07:49.211997    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/364d8616-829c-4e2c-8531-cf977166dbf3-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"364d8616-829c-4e2c-8531-cf977166dbf3\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 21:07:49.212092 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:07:49.212001    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/364d8616-829c-4e2c-8531-cf977166dbf3-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"364d8616-829c-4e2c-8531-cf977166dbf3\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 21:07:49.212092 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:07:49.212054    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/364d8616-829c-4e2c-8531-cf977166dbf3-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"364d8616-829c-4e2c-8531-cf977166dbf3\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 21:07:49.212574 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:07:49.212550    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/364d8616-829c-4e2c-8531-cf977166dbf3-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"364d8616-829c-4e2c-8531-cf977166dbf3\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 21:07:49.212664 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:07:49.212647    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/364d8616-829c-4e2c-8531-cf977166dbf3-config-volume\") pod \"alertmanager-main-0\" (UID: \"364d8616-829c-4e2c-8531-cf977166dbf3\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 21:07:49.213566 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:07:49.213549    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/364d8616-829c-4e2c-8531-cf977166dbf3-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"364d8616-829c-4e2c-8531-cf977166dbf3\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 21:07:49.217738 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:07:49.217717    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nx94d\" (UniqueName: \"kubernetes.io/projected/364d8616-829c-4e2c-8531-cf977166dbf3-kube-api-access-nx94d\") pod \"alertmanager-main-0\" (UID: \"364d8616-829c-4e2c-8531-cf977166dbf3\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 21:07:49.344569 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:07:49.344471    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Jul 27 21:07:49.488957 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:07:49.488926    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 27 21:07:49.490862 ip-10-0-131-133 kubenswrapper[2559]: W0727 21:07:49.490835    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod364d8616_829c_4e2c_8531_cf977166dbf3.slice/crio-fb0d52c0759dc99e54ad0994803d7e010de26064abffb5348e8bd9229ca5898b WatchSource:0}: Error finding container fb0d52c0759dc99e54ad0994803d7e010de26064abffb5348e8bd9229ca5898b: Status 404 returned error can't find the container with id fb0d52c0759dc99e54ad0994803d7e010de26064abffb5348e8bd9229ca5898b
Jul 27 21:07:49.992166 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:07:49.992128    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"364d8616-829c-4e2c-8531-cf977166dbf3","Type":"ContainerStarted","Data":"fb0d52c0759dc99e54ad0994803d7e010de26064abffb5348e8bd9229ca5898b"}
Jul 27 21:07:50.999738 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:07:50.999706    2559 generic.go:356] "Generic (PLEG): container finished" podID="364d8616-829c-4e2c-8531-cf977166dbf3" containerID="c1c41fd34cd7f2ca5108ecd328480bc984248bc0d27c1cb963eb55998fb40eb5" exitCode=0
Jul 27 21:07:51.000174 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:07:50.999769    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"364d8616-829c-4e2c-8531-cf977166dbf3","Type":"ContainerDied","Data":"c1c41fd34cd7f2ca5108ecd328480bc984248bc0d27c1cb963eb55998fb40eb5"}
Jul 27 21:07:51.509580 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:07:51.509548    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-5d57f6db47-l8m59"]
Jul 27 21:07:51.512258 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:07:51.512237    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-5d57f6db47-l8m59"
Jul 27 21:07:51.515557 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:07:51.515530    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\""
Jul 27 21:07:51.515678 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:07:51.515530    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\""
Jul 27 21:07:51.515678 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:07:51.515618    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\""
Jul 27 21:07:51.515827 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:07:51.515810    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-4qrsw\""
Jul 27 21:07:51.515993 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:07:51.515973    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\""
Jul 27 21:07:51.516095 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:07:51.516079    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\""
Jul 27 21:07:51.530120 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:07:51.530096    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/c5a90613-ef7f-4695-a77e-ad78f0327434-federate-client-tls\") pod \"telemeter-client-5d57f6db47-l8m59\" (UID: \"c5a90613-ef7f-4695-a77e-ad78f0327434\") " pod="openshift-monitoring/telemeter-client-5d57f6db47-l8m59"
Jul 27 21:07:51.530250 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:07:51.530123    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ft4jz\" (UniqueName: \"kubernetes.io/projected/c5a90613-ef7f-4695-a77e-ad78f0327434-kube-api-access-ft4jz\") pod \"telemeter-client-5d57f6db47-l8m59\" (UID: \"c5a90613-ef7f-4695-a77e-ad78f0327434\") " pod="openshift-monitoring/telemeter-client-5d57f6db47-l8m59"
Jul 27 21:07:51.530250 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:07:51.530154    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c5a90613-ef7f-4695-a77e-ad78f0327434-telemeter-trusted-ca-bundle\") pod \"telemeter-client-5d57f6db47-l8m59\" (UID: \"c5a90613-ef7f-4695-a77e-ad78f0327434\") " pod="openshift-monitoring/telemeter-client-5d57f6db47-l8m59"
Jul 27 21:07:51.530250 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:07:51.530197    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/c5a90613-ef7f-4695-a77e-ad78f0327434-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-5d57f6db47-l8m59\" (UID: \"c5a90613-ef7f-4695-a77e-ad78f0327434\") " pod="openshift-monitoring/telemeter-client-5d57f6db47-l8m59"
Jul 27 21:07:51.530389 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:07:51.530267    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c5a90613-ef7f-4695-a77e-ad78f0327434-metrics-client-ca\") pod \"telemeter-client-5d57f6db47-l8m59\" (UID: \"c5a90613-ef7f-4695-a77e-ad78f0327434\") " pod="openshift-monitoring/telemeter-client-5d57f6db47-l8m59"
Jul 27 21:07:51.530389 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:07:51.530286    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/c5a90613-ef7f-4695-a77e-ad78f0327434-telemeter-client-tls\") pod \"telemeter-client-5d57f6db47-l8m59\" (UID: \"c5a90613-ef7f-4695-a77e-ad78f0327434\") " pod="openshift-monitoring/telemeter-client-5d57f6db47-l8m59"
Jul 27 21:07:51.530389 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:07:51.530310    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/c5a90613-ef7f-4695-a77e-ad78f0327434-secret-telemeter-client\") pod \"telemeter-client-5d57f6db47-l8m59\" (UID: \"c5a90613-ef7f-4695-a77e-ad78f0327434\") " pod="openshift-monitoring/telemeter-client-5d57f6db47-l8m59"
Jul 27 21:07:51.530389 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:07:51.530333    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c5a90613-ef7f-4695-a77e-ad78f0327434-serving-certs-ca-bundle\") pod \"telemeter-client-5d57f6db47-l8m59\" (UID: \"c5a90613-ef7f-4695-a77e-ad78f0327434\") " pod="openshift-monitoring/telemeter-client-5d57f6db47-l8m59"
Jul 27 21:07:51.543397 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:07:51.543370    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-8i12ta5c71j38\""
Jul 27 21:07:51.550097 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:07:51.550074    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-5d57f6db47-l8m59"]
Jul 27 21:07:51.631605 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:07:51.631557    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/c5a90613-ef7f-4695-a77e-ad78f0327434-telemeter-client-tls\") pod \"telemeter-client-5d57f6db47-l8m59\" (UID: \"c5a90613-ef7f-4695-a77e-ad78f0327434\") " pod="openshift-monitoring/telemeter-client-5d57f6db47-l8m59"
Jul 27 21:07:51.631800 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:07:51.631613    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/c5a90613-ef7f-4695-a77e-ad78f0327434-secret-telemeter-client\") pod \"telemeter-client-5d57f6db47-l8m59\" (UID: \"c5a90613-ef7f-4695-a77e-ad78f0327434\") " pod="openshift-monitoring/telemeter-client-5d57f6db47-l8m59"
Jul 27 21:07:51.631800 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:07:51.631646    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c5a90613-ef7f-4695-a77e-ad78f0327434-serving-certs-ca-bundle\") pod \"telemeter-client-5d57f6db47-l8m59\" (UID: \"c5a90613-ef7f-4695-a77e-ad78f0327434\") " pod="openshift-monitoring/telemeter-client-5d57f6db47-l8m59"
Jul 27 21:07:51.631800 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:07:51.631693    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/c5a90613-ef7f-4695-a77e-ad78f0327434-federate-client-tls\") pod \"telemeter-client-5d57f6db47-l8m59\" (UID: \"c5a90613-ef7f-4695-a77e-ad78f0327434\") " pod="openshift-monitoring/telemeter-client-5d57f6db47-l8m59"
Jul 27 21:07:51.631800 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:07:51.631715    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ft4jz\" (UniqueName: \"kubernetes.io/projected/c5a90613-ef7f-4695-a77e-ad78f0327434-kube-api-access-ft4jz\") pod \"telemeter-client-5d57f6db47-l8m59\" (UID: \"c5a90613-ef7f-4695-a77e-ad78f0327434\") " pod="openshift-monitoring/telemeter-client-5d57f6db47-l8m59"
Jul 27 21:07:51.631800 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:07:51.631778    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c5a90613-ef7f-4695-a77e-ad78f0327434-telemeter-trusted-ca-bundle\") pod \"telemeter-client-5d57f6db47-l8m59\" (UID: \"c5a90613-ef7f-4695-a77e-ad78f0327434\") " pod="openshift-monitoring/telemeter-client-5d57f6db47-l8m59"
Jul 27 21:07:51.632034 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:07:51.631804    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/c5a90613-ef7f-4695-a77e-ad78f0327434-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-5d57f6db47-l8m59\" (UID: \"c5a90613-ef7f-4695-a77e-ad78f0327434\") " pod="openshift-monitoring/telemeter-client-5d57f6db47-l8m59"
Jul 27 21:07:51.632034 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:07:51.631856    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c5a90613-ef7f-4695-a77e-ad78f0327434-metrics-client-ca\") pod \"telemeter-client-5d57f6db47-l8m59\" (UID: \"c5a90613-ef7f-4695-a77e-ad78f0327434\") " pod="openshift-monitoring/telemeter-client-5d57f6db47-l8m59"
Jul 27 21:07:51.632672 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:07:51.632644    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c5a90613-ef7f-4695-a77e-ad78f0327434-metrics-client-ca\") pod \"telemeter-client-5d57f6db47-l8m59\" (UID: \"c5a90613-ef7f-4695-a77e-ad78f0327434\") " pod="openshift-monitoring/telemeter-client-5d57f6db47-l8m59"
Jul 27 21:07:51.632804 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:07:51.632751    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c5a90613-ef7f-4695-a77e-ad78f0327434-serving-certs-ca-bundle\") pod \"telemeter-client-5d57f6db47-l8m59\" (UID: \"c5a90613-ef7f-4695-a77e-ad78f0327434\") " pod="openshift-monitoring/telemeter-client-5d57f6db47-l8m59"
Jul 27 21:07:51.632873 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:07:51.632802    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c5a90613-ef7f-4695-a77e-ad78f0327434-telemeter-trusted-ca-bundle\") pod \"telemeter-client-5d57f6db47-l8m59\" (UID: \"c5a90613-ef7f-4695-a77e-ad78f0327434\") " pod="openshift-monitoring/telemeter-client-5d57f6db47-l8m59"
Jul 27 21:07:51.634462 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:07:51.634433    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/c5a90613-ef7f-4695-a77e-ad78f0327434-secret-telemeter-client\") pod \"telemeter-client-5d57f6db47-l8m59\" (UID: \"c5a90613-ef7f-4695-a77e-ad78f0327434\") " pod="openshift-monitoring/telemeter-client-5d57f6db47-l8m59"
Jul 27 21:07:51.634556 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:07:51.634476    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/c5a90613-ef7f-4695-a77e-ad78f0327434-telemeter-client-tls\") pod \"telemeter-client-5d57f6db47-l8m59\" (UID: \"c5a90613-ef7f-4695-a77e-ad78f0327434\") " pod="openshift-monitoring/telemeter-client-5d57f6db47-l8m59"
Jul 27 21:07:51.634722 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:07:51.634701    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/c5a90613-ef7f-4695-a77e-ad78f0327434-federate-client-tls\") pod \"telemeter-client-5d57f6db47-l8m59\" (UID: \"c5a90613-ef7f-4695-a77e-ad78f0327434\") " pod="openshift-monitoring/telemeter-client-5d57f6db47-l8m59"
Jul 27 21:07:51.634763 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:07:51.634737    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/c5a90613-ef7f-4695-a77e-ad78f0327434-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-5d57f6db47-l8m59\" (UID: \"c5a90613-ef7f-4695-a77e-ad78f0327434\") " pod="openshift-monitoring/telemeter-client-5d57f6db47-l8m59"
Jul 27 21:07:51.640704 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:07:51.640681    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ft4jz\" (UniqueName: \"kubernetes.io/projected/c5a90613-ef7f-4695-a77e-ad78f0327434-kube-api-access-ft4jz\") pod \"telemeter-client-5d57f6db47-l8m59\" (UID: \"c5a90613-ef7f-4695-a77e-ad78f0327434\") " pod="openshift-monitoring/telemeter-client-5d57f6db47-l8m59"
Jul 27 21:07:51.821926 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:07:51.821844    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-5d57f6db47-l8m59"
Jul 27 21:07:51.983646 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:07:51.983615    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-5d57f6db47-l8m59"]
Jul 27 21:07:51.984494 ip-10-0-131-133 kubenswrapper[2559]: W0727 21:07:51.984462    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc5a90613_ef7f_4695_a77e_ad78f0327434.slice/crio-5128548d207ec99c6c12e9a59c1d925afbdcdedbf784cc5266e3aec29f6da8a7 WatchSource:0}: Error finding container 5128548d207ec99c6c12e9a59c1d925afbdcdedbf784cc5266e3aec29f6da8a7: Status 404 returned error can't find the container with id 5128548d207ec99c6c12e9a59c1d925afbdcdedbf784cc5266e3aec29f6da8a7
Jul 27 21:07:52.005337 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:07:52.005293    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-5d57f6db47-l8m59" event={"ID":"c5a90613-ef7f-4695-a77e-ad78f0327434","Type":"ContainerStarted","Data":"5128548d207ec99c6c12e9a59c1d925afbdcdedbf784cc5266e3aec29f6da8a7"}
Jul 27 21:07:53.012506 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:07:53.012425    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"364d8616-829c-4e2c-8531-cf977166dbf3","Type":"ContainerStarted","Data":"eb551f03007d498ea045344cf475ad59b81c5903d6469a75789359804fffa1be"}
Jul 27 21:07:53.012868 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:07:53.012551    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"364d8616-829c-4e2c-8531-cf977166dbf3","Type":"ContainerStarted","Data":"947bc3459bbbf711deb306e1aee31b7f49fc11799fe79deb9f0aac464644e4e6"}
Jul 27 21:07:53.012868 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:07:53.012593    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"364d8616-829c-4e2c-8531-cf977166dbf3","Type":"ContainerStarted","Data":"2130a0490f537cb537feff7ad8022ff69e6513a9be5659ac5b0b2361067da915"}
Jul 27 21:07:53.012868 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:07:53.012653    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"364d8616-829c-4e2c-8531-cf977166dbf3","Type":"ContainerStarted","Data":"92cd46bd77fdf87004b9c230b2cc0fbe0ff5e08440ae66365bdf2342b8aeba33"}
Jul 27 21:07:53.012868 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:07:53.012691    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"364d8616-829c-4e2c-8531-cf977166dbf3","Type":"ContainerStarted","Data":"6350842bb6cf62802900fe2899925847a96d79be06bcec6fdcc76c6e0fb744c3"}
Jul 27 21:07:53.012868 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:07:53.012726    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"364d8616-829c-4e2c-8531-cf977166dbf3","Type":"ContainerStarted","Data":"34acf98f2f0d6540c6422fe7b431cec25bcb91fbf4d12fbf90d9de4b72ef0b00"}
Jul 27 21:07:53.042584 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:07:53.042529    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=1.4539044620000001 podStartE2EDuration="4.042510969s" totalImagesPullingTime="2.572129549s" totalInitContainerRuntime="16.476958ms" isStatefulPod=false podCreationTimestamp="2026-07-27 21:07:49 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 21:07:53.039778937 +0000 UTC m=+201.618962944" watchObservedRunningTime="2026-07-27 21:07:53.042510969 +0000 UTC m=+201.621695025"
Jul 27 21:07:54.019519 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:07:54.019490    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-5d57f6db47-l8m59" event={"ID":"c5a90613-ef7f-4695-a77e-ad78f0327434","Type":"ContainerStarted","Data":"4e68771438874d8cbf01c60acf1ab7aef93c0d18522c758196450bf70118fee2"}
Jul 27 21:07:54.019798 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:07:54.019556    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-5d57f6db47-l8m59" event={"ID":"c5a90613-ef7f-4695-a77e-ad78f0327434","Type":"ContainerStarted","Data":"8947591ae7b83d8758bae9b47d4ed26be77b64bfb07231d52a83f61d07dc75e3"}
Jul 27 21:07:55.025990 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:07:55.025951    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-5d57f6db47-l8m59" event={"ID":"c5a90613-ef7f-4695-a77e-ad78f0327434","Type":"ContainerStarted","Data":"683a9a132156f7cd47b9782d131ddf57e844babc7ebac4f3fbcbed022e8a3f98"}
Jul 27 21:07:55.050149 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:07:55.050080    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-5d57f6db47-l8m59" podStartSLOduration=2.155180833 podStartE2EDuration="4.05006145s" totalImagesPullingTime="1.894880617s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 21:07:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 21:07:55.048309952 +0000 UTC m=+203.627493958" watchObservedRunningTime="2026-07-27 21:07:55.05006145 +0000 UTC m=+203.629245456"
Jul 27 21:07:55.901232 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:07:55.901188    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-58b6b694f4-gqmmt"]
Jul 27 21:07:55.903663 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:07:55.903642    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-58b6b694f4-gqmmt"
Jul 27 21:07:55.920563 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:07:55.920527    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-58b6b694f4-gqmmt"]
Jul 27 21:07:55.971844 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:07:55.971806    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/81b570c4-4195-43da-8b76-4d77909a5a43-trusted-ca-bundle\") pod \"console-58b6b694f4-gqmmt\" (UID: \"81b570c4-4195-43da-8b76-4d77909a5a43\") " pod="openshift-console/console-58b6b694f4-gqmmt"
Jul 27 21:07:55.971991 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:07:55.971883    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/81b570c4-4195-43da-8b76-4d77909a5a43-console-serving-cert\") pod \"console-58b6b694f4-gqmmt\" (UID: \"81b570c4-4195-43da-8b76-4d77909a5a43\") " pod="openshift-console/console-58b6b694f4-gqmmt"
Jul 27 21:07:55.971991 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:07:55.971910    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/81b570c4-4195-43da-8b76-4d77909a5a43-service-ca\") pod \"console-58b6b694f4-gqmmt\" (UID: \"81b570c4-4195-43da-8b76-4d77909a5a43\") " pod="openshift-console/console-58b6b694f4-gqmmt"
Jul 27 21:07:55.971991 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:07:55.971939    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g2bwz\" (UniqueName: \"kubernetes.io/projected/81b570c4-4195-43da-8b76-4d77909a5a43-kube-api-access-g2bwz\") pod \"console-58b6b694f4-gqmmt\" (UID: \"81b570c4-4195-43da-8b76-4d77909a5a43\") " pod="openshift-console/console-58b6b694f4-gqmmt"
Jul 27 21:07:55.971991 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:07:55.971957    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/81b570c4-4195-43da-8b76-4d77909a5a43-console-oauth-config\") pod \"console-58b6b694f4-gqmmt\" (UID: \"81b570c4-4195-43da-8b76-4d77909a5a43\") " pod="openshift-console/console-58b6b694f4-gqmmt"
Jul 27 21:07:55.972140 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:07:55.972036    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/81b570c4-4195-43da-8b76-4d77909a5a43-console-config\") pod \"console-58b6b694f4-gqmmt\" (UID: \"81b570c4-4195-43da-8b76-4d77909a5a43\") " pod="openshift-console/console-58b6b694f4-gqmmt"
Jul 27 21:07:55.972140 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:07:55.972059    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/81b570c4-4195-43da-8b76-4d77909a5a43-oauth-serving-cert\") pod \"console-58b6b694f4-gqmmt\" (UID: \"81b570c4-4195-43da-8b76-4d77909a5a43\") " pod="openshift-console/console-58b6b694f4-gqmmt"
Jul 27 21:07:56.073020 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:07:56.072981    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/81b570c4-4195-43da-8b76-4d77909a5a43-console-config\") pod \"console-58b6b694f4-gqmmt\" (UID: \"81b570c4-4195-43da-8b76-4d77909a5a43\") " pod="openshift-console/console-58b6b694f4-gqmmt"
Jul 27 21:07:56.073533 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:07:56.073051    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/81b570c4-4195-43da-8b76-4d77909a5a43-oauth-serving-cert\") pod \"console-58b6b694f4-gqmmt\" (UID: \"81b570c4-4195-43da-8b76-4d77909a5a43\") " pod="openshift-console/console-58b6b694f4-gqmmt"
Jul 27 21:07:56.073533 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:07:56.073459    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/81b570c4-4195-43da-8b76-4d77909a5a43-trusted-ca-bundle\") pod \"console-58b6b694f4-gqmmt\" (UID: \"81b570c4-4195-43da-8b76-4d77909a5a43\") " pod="openshift-console/console-58b6b694f4-gqmmt"
Jul 27 21:07:56.073860 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:07:56.073835    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/81b570c4-4195-43da-8b76-4d77909a5a43-console-config\") pod \"console-58b6b694f4-gqmmt\" (UID: \"81b570c4-4195-43da-8b76-4d77909a5a43\") " pod="openshift-console/console-58b6b694f4-gqmmt"
Jul 27 21:07:56.073860 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:07:56.073835    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/81b570c4-4195-43da-8b76-4d77909a5a43-oauth-serving-cert\") pod \"console-58b6b694f4-gqmmt\" (UID: \"81b570c4-4195-43da-8b76-4d77909a5a43\") " pod="openshift-console/console-58b6b694f4-gqmmt"
Jul 27 21:07:56.074002 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:07:56.073843    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/81b570c4-4195-43da-8b76-4d77909a5a43-console-serving-cert\") pod \"console-58b6b694f4-gqmmt\" (UID: \"81b570c4-4195-43da-8b76-4d77909a5a43\") " pod="openshift-console/console-58b6b694f4-gqmmt"
Jul 27 21:07:56.074002 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:07:56.073923    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/81b570c4-4195-43da-8b76-4d77909a5a43-service-ca\") pod \"console-58b6b694f4-gqmmt\" (UID: \"81b570c4-4195-43da-8b76-4d77909a5a43\") " pod="openshift-console/console-58b6b694f4-gqmmt"
Jul 27 21:07:56.074107 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:07:56.074014    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g2bwz\" (UniqueName: \"kubernetes.io/projected/81b570c4-4195-43da-8b76-4d77909a5a43-kube-api-access-g2bwz\") pod \"console-58b6b694f4-gqmmt\" (UID: \"81b570c4-4195-43da-8b76-4d77909a5a43\") " pod="openshift-console/console-58b6b694f4-gqmmt"
Jul 27 21:07:56.074107 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:07:56.074071    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/81b570c4-4195-43da-8b76-4d77909a5a43-console-oauth-config\") pod \"console-58b6b694f4-gqmmt\" (UID: \"81b570c4-4195-43da-8b76-4d77909a5a43\") " pod="openshift-console/console-58b6b694f4-gqmmt"
Jul 27 21:07:56.074207 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:07:56.074188    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/81b570c4-4195-43da-8b76-4d77909a5a43-trusted-ca-bundle\") pod \"console-58b6b694f4-gqmmt\" (UID: \"81b570c4-4195-43da-8b76-4d77909a5a43\") " pod="openshift-console/console-58b6b694f4-gqmmt"
Jul 27 21:07:56.074599 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:07:56.074579    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/81b570c4-4195-43da-8b76-4d77909a5a43-service-ca\") pod \"console-58b6b694f4-gqmmt\" (UID: \"81b570c4-4195-43da-8b76-4d77909a5a43\") " pod="openshift-console/console-58b6b694f4-gqmmt"
Jul 27 21:07:56.076116 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:07:56.076100    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/81b570c4-4195-43da-8b76-4d77909a5a43-console-serving-cert\") pod \"console-58b6b694f4-gqmmt\" (UID: \"81b570c4-4195-43da-8b76-4d77909a5a43\") " pod="openshift-console/console-58b6b694f4-gqmmt"
Jul 27 21:07:56.076476 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:07:56.076456    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/81b570c4-4195-43da-8b76-4d77909a5a43-console-oauth-config\") pod \"console-58b6b694f4-gqmmt\" (UID: \"81b570c4-4195-43da-8b76-4d77909a5a43\") " pod="openshift-console/console-58b6b694f4-gqmmt"
Jul 27 21:07:56.082510 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:07:56.082493    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g2bwz\" (UniqueName: \"kubernetes.io/projected/81b570c4-4195-43da-8b76-4d77909a5a43-kube-api-access-g2bwz\") pod \"console-58b6b694f4-gqmmt\" (UID: \"81b570c4-4195-43da-8b76-4d77909a5a43\") " pod="openshift-console/console-58b6b694f4-gqmmt"
Jul 27 21:07:56.213273 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:07:56.213157    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-58b6b694f4-gqmmt"
Jul 27 21:07:56.345999 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:07:56.345966    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-58b6b694f4-gqmmt"]
Jul 27 21:07:56.350872 ip-10-0-131-133 kubenswrapper[2559]: W0727 21:07:56.350847    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod81b570c4_4195_43da_8b76_4d77909a5a43.slice/crio-7494d323356f77abd2e607ecdf38adab11eeb51c5419f8aed74dd105f6460eb5 WatchSource:0}: Error finding container 7494d323356f77abd2e607ecdf38adab11eeb51c5419f8aed74dd105f6460eb5: Status 404 returned error can't find the container with id 7494d323356f77abd2e607ecdf38adab11eeb51c5419f8aed74dd105f6460eb5
Jul 27 21:07:57.035029 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:07:57.034987    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-58b6b694f4-gqmmt" event={"ID":"81b570c4-4195-43da-8b76-4d77909a5a43","Type":"ContainerStarted","Data":"22a793bf236f3d2b023bd88dd21c07f2c60aff2203952504f3009831016ca93d"}
Jul 27 21:07:57.035185 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:07:57.035051    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-58b6b694f4-gqmmt" event={"ID":"81b570c4-4195-43da-8b76-4d77909a5a43","Type":"ContainerStarted","Data":"7494d323356f77abd2e607ecdf38adab11eeb51c5419f8aed74dd105f6460eb5"}
Jul 27 21:07:57.054781 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:07:57.054721    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-58b6b694f4-gqmmt" podStartSLOduration=2.054700747 podStartE2EDuration="2.054700747s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 21:07:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 21:07:57.054320613 +0000 UTC m=+205.633504620" watchObservedRunningTime="2026-07-27 21:07:57.054700747 +0000 UTC m=+205.633884757"
Jul 27 21:08:06.213847 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:08:06.213800    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-58b6b694f4-gqmmt"
Jul 27 21:08:06.214330 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:08:06.213878    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-58b6b694f4-gqmmt"
Jul 27 21:08:06.218980 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:08:06.218955    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-58b6b694f4-gqmmt"
Jul 27 21:08:07.084859 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:08:07.084832    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-58b6b694f4-gqmmt"
Jul 27 21:08:07.132977 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:08:07.132909    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5748766fc5-bbnsx"]
Jul 27 21:08:32.154814 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:08:32.154771    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-5748766fc5-bbnsx" podUID="f242f265-2dc0-467c-a8cd-f413c1d2f254" containerName="console" containerID="cri-o://5004543d95b694e2413d55c313acd1c8e27ed613b607650bb65bd34a44c1e821" gracePeriod=15
Jul 27 21:08:32.412457 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:08:32.412398    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5748766fc5-bbnsx"
Jul 27 21:08:32.489191 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:08:32.489155    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/f242f265-2dc0-467c-a8cd-f413c1d2f254-console-config\") pod \"f242f265-2dc0-467c-a8cd-f413c1d2f254\" (UID: \"f242f265-2dc0-467c-a8cd-f413c1d2f254\") "
Jul 27 21:08:32.489394 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:08:32.489208    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/f242f265-2dc0-467c-a8cd-f413c1d2f254-console-oauth-config\") pod \"f242f265-2dc0-467c-a8cd-f413c1d2f254\" (UID: \"f242f265-2dc0-467c-a8cd-f413c1d2f254\") "
Jul 27 21:08:32.489394 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:08:32.489258    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/f242f265-2dc0-467c-a8cd-f413c1d2f254-console-serving-cert\") pod \"f242f265-2dc0-467c-a8cd-f413c1d2f254\" (UID: \"f242f265-2dc0-467c-a8cd-f413c1d2f254\") "
Jul 27 21:08:32.489394 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:08:32.489288    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/f242f265-2dc0-467c-a8cd-f413c1d2f254-oauth-serving-cert\") pod \"f242f265-2dc0-467c-a8cd-f413c1d2f254\" (UID: \"f242f265-2dc0-467c-a8cd-f413c1d2f254\") "
Jul 27 21:08:32.489394 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:08:32.489316    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f242f265-2dc0-467c-a8cd-f413c1d2f254-trusted-ca-bundle\") pod \"f242f265-2dc0-467c-a8cd-f413c1d2f254\" (UID: \"f242f265-2dc0-467c-a8cd-f413c1d2f254\") "
Jul 27 21:08:32.489394 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:08:32.489376    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wz9gz\" (UniqueName: \"kubernetes.io/projected/f242f265-2dc0-467c-a8cd-f413c1d2f254-kube-api-access-wz9gz\") pod \"f242f265-2dc0-467c-a8cd-f413c1d2f254\" (UID: \"f242f265-2dc0-467c-a8cd-f413c1d2f254\") "
Jul 27 21:08:32.489663 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:08:32.489441    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/f242f265-2dc0-467c-a8cd-f413c1d2f254-service-ca\") pod \"f242f265-2dc0-467c-a8cd-f413c1d2f254\" (UID: \"f242f265-2dc0-467c-a8cd-f413c1d2f254\") "
Jul 27 21:08:32.489733 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:08:32.489573    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f242f265-2dc0-467c-a8cd-f413c1d2f254-console-config" (OuterVolumeSpecName: "console-config") pod "f242f265-2dc0-467c-a8cd-f413c1d2f254" (UID: "f242f265-2dc0-467c-a8cd-f413c1d2f254"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 21:08:32.489794 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:08:32.489724    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f242f265-2dc0-467c-a8cd-f413c1d2f254-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "f242f265-2dc0-467c-a8cd-f413c1d2f254" (UID: "f242f265-2dc0-467c-a8cd-f413c1d2f254"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 21:08:32.489794 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:08:32.489749    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f242f265-2dc0-467c-a8cd-f413c1d2f254-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "f242f265-2dc0-467c-a8cd-f413c1d2f254" (UID: "f242f265-2dc0-467c-a8cd-f413c1d2f254"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 21:08:32.490006 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:08:32.489982    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f242f265-2dc0-467c-a8cd-f413c1d2f254-service-ca" (OuterVolumeSpecName: "service-ca") pod "f242f265-2dc0-467c-a8cd-f413c1d2f254" (UID: "f242f265-2dc0-467c-a8cd-f413c1d2f254"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 21:08:32.491522 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:08:32.491497    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f242f265-2dc0-467c-a8cd-f413c1d2f254-kube-api-access-wz9gz" (OuterVolumeSpecName: "kube-api-access-wz9gz") pod "f242f265-2dc0-467c-a8cd-f413c1d2f254" (UID: "f242f265-2dc0-467c-a8cd-f413c1d2f254"). InnerVolumeSpecName "kube-api-access-wz9gz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 21:08:32.491623 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:08:32.491605    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f242f265-2dc0-467c-a8cd-f413c1d2f254-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "f242f265-2dc0-467c-a8cd-f413c1d2f254" (UID: "f242f265-2dc0-467c-a8cd-f413c1d2f254"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 21:08:32.491664 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:08:32.491631    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f242f265-2dc0-467c-a8cd-f413c1d2f254-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "f242f265-2dc0-467c-a8cd-f413c1d2f254" (UID: "f242f265-2dc0-467c-a8cd-f413c1d2f254"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 21:08:32.590588 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:08:32.590537    2559 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/f242f265-2dc0-467c-a8cd-f413c1d2f254-service-ca\") on node \"ip-10-0-131-133.ec2.internal\" DevicePath \"\""
Jul 27 21:08:32.590588 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:08:32.590582    2559 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/f242f265-2dc0-467c-a8cd-f413c1d2f254-console-config\") on node \"ip-10-0-131-133.ec2.internal\" DevicePath \"\""
Jul 27 21:08:32.590588 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:08:32.590593    2559 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/f242f265-2dc0-467c-a8cd-f413c1d2f254-console-oauth-config\") on node \"ip-10-0-131-133.ec2.internal\" DevicePath \"\""
Jul 27 21:08:32.590588 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:08:32.590603    2559 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/f242f265-2dc0-467c-a8cd-f413c1d2f254-console-serving-cert\") on node \"ip-10-0-131-133.ec2.internal\" DevicePath \"\""
Jul 27 21:08:32.590872 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:08:32.590611    2559 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/f242f265-2dc0-467c-a8cd-f413c1d2f254-oauth-serving-cert\") on node \"ip-10-0-131-133.ec2.internal\" DevicePath \"\""
Jul 27 21:08:32.590872 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:08:32.590619    2559 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f242f265-2dc0-467c-a8cd-f413c1d2f254-trusted-ca-bundle\") on node \"ip-10-0-131-133.ec2.internal\" DevicePath \"\""
Jul 27 21:08:32.590872 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:08:32.590628    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wz9gz\" (UniqueName: \"kubernetes.io/projected/f242f265-2dc0-467c-a8cd-f413c1d2f254-kube-api-access-wz9gz\") on node \"ip-10-0-131-133.ec2.internal\" DevicePath \"\""
Jul 27 21:08:33.181310 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:08:33.181279    2559 generic.go:356] "Generic (PLEG): container finished" podID="f242f265-2dc0-467c-a8cd-f413c1d2f254" containerID="5004543d95b694e2413d55c313acd1c8e27ed613b607650bb65bd34a44c1e821" exitCode=0
Jul 27 21:08:33.181752 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:08:33.181326    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5748766fc5-bbnsx"
Jul 27 21:08:33.181752 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:08:33.181368    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5748766fc5-bbnsx" event={"ID":"f242f265-2dc0-467c-a8cd-f413c1d2f254","Type":"ContainerDied","Data":"5004543d95b694e2413d55c313acd1c8e27ed613b607650bb65bd34a44c1e821"}
Jul 27 21:08:33.181752 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:08:33.181421    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5748766fc5-bbnsx" event={"ID":"f242f265-2dc0-467c-a8cd-f413c1d2f254","Type":"ContainerDied","Data":"2331eb37e15ae8fce246ac296c8eac2ceed5e477f3f4413f4b19b399d79e6ef2"}
Jul 27 21:08:33.181752 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:08:33.181441    2559 scope.go:117] "RemoveContainer" containerID="5004543d95b694e2413d55c313acd1c8e27ed613b607650bb65bd34a44c1e821"
Jul 27 21:08:33.189983 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:08:33.189965    2559 scope.go:117] "RemoveContainer" containerID="5004543d95b694e2413d55c313acd1c8e27ed613b607650bb65bd34a44c1e821"
Jul 27 21:08:33.190281 ip-10-0-131-133 kubenswrapper[2559]: E0727 21:08:33.190261    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5004543d95b694e2413d55c313acd1c8e27ed613b607650bb65bd34a44c1e821\": container with ID starting with 5004543d95b694e2413d55c313acd1c8e27ed613b607650bb65bd34a44c1e821 not found: ID does not exist" containerID="5004543d95b694e2413d55c313acd1c8e27ed613b607650bb65bd34a44c1e821"
Jul 27 21:08:33.190355 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:08:33.190290    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5004543d95b694e2413d55c313acd1c8e27ed613b607650bb65bd34a44c1e821"} err="failed to get container status \"5004543d95b694e2413d55c313acd1c8e27ed613b607650bb65bd34a44c1e821\": rpc error: code = NotFound desc = could not find container \"5004543d95b694e2413d55c313acd1c8e27ed613b607650bb65bd34a44c1e821\": container with ID starting with 5004543d95b694e2413d55c313acd1c8e27ed613b607650bb65bd34a44c1e821 not found: ID does not exist"
Jul 27 21:08:33.203078 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:08:33.203046    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5748766fc5-bbnsx"]
Jul 27 21:08:33.206474 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:08:33.206449    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-5748766fc5-bbnsx"]
Jul 27 21:08:33.981017 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:08:33.980984    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f242f265-2dc0-467c-a8cd-f413c1d2f254" path="/var/lib/kubelet/pods/f242f265-2dc0-467c-a8cd-f413c1d2f254/volumes"
Jul 27 21:09:07.315761 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:09:07.315730    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-745654d88d-xcrr4"]
Jul 27 21:09:07.316392 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:09:07.316191    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f242f265-2dc0-467c-a8cd-f413c1d2f254" containerName="console"
Jul 27 21:09:07.316392 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:09:07.316208    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="f242f265-2dc0-467c-a8cd-f413c1d2f254" containerName="console"
Jul 27 21:09:07.316392 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:09:07.316343    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="f242f265-2dc0-467c-a8cd-f413c1d2f254" containerName="console"
Jul 27 21:09:07.319722 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:09:07.319687    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-745654d88d-xcrr4"
Jul 27 21:09:07.332949 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:09:07.332923    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-745654d88d-xcrr4"]
Jul 27 21:09:07.384822 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:09:07.384790    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/508ed916-ee7b-44c9-a831-5bddfafdc8b1-service-ca\") pod \"console-745654d88d-xcrr4\" (UID: \"508ed916-ee7b-44c9-a831-5bddfafdc8b1\") " pod="openshift-console/console-745654d88d-xcrr4"
Jul 27 21:09:07.385014 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:09:07.384857    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/508ed916-ee7b-44c9-a831-5bddfafdc8b1-console-serving-cert\") pod \"console-745654d88d-xcrr4\" (UID: \"508ed916-ee7b-44c9-a831-5bddfafdc8b1\") " pod="openshift-console/console-745654d88d-xcrr4"
Jul 27 21:09:07.385014 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:09:07.384902    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/508ed916-ee7b-44c9-a831-5bddfafdc8b1-console-oauth-config\") pod \"console-745654d88d-xcrr4\" (UID: \"508ed916-ee7b-44c9-a831-5bddfafdc8b1\") " pod="openshift-console/console-745654d88d-xcrr4"
Jul 27 21:09:07.385014 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:09:07.384931    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/508ed916-ee7b-44c9-a831-5bddfafdc8b1-console-config\") pod \"console-745654d88d-xcrr4\" (UID: \"508ed916-ee7b-44c9-a831-5bddfafdc8b1\") " pod="openshift-console/console-745654d88d-xcrr4"
Jul 27 21:09:07.385014 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:09:07.384957    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/508ed916-ee7b-44c9-a831-5bddfafdc8b1-oauth-serving-cert\") pod \"console-745654d88d-xcrr4\" (UID: \"508ed916-ee7b-44c9-a831-5bddfafdc8b1\") " pod="openshift-console/console-745654d88d-xcrr4"
Jul 27 21:09:07.385014 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:09:07.384983    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dfxp6\" (UniqueName: \"kubernetes.io/projected/508ed916-ee7b-44c9-a831-5bddfafdc8b1-kube-api-access-dfxp6\") pod \"console-745654d88d-xcrr4\" (UID: \"508ed916-ee7b-44c9-a831-5bddfafdc8b1\") " pod="openshift-console/console-745654d88d-xcrr4"
Jul 27 21:09:07.385245 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:09:07.385052    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/508ed916-ee7b-44c9-a831-5bddfafdc8b1-trusted-ca-bundle\") pod \"console-745654d88d-xcrr4\" (UID: \"508ed916-ee7b-44c9-a831-5bddfafdc8b1\") " pod="openshift-console/console-745654d88d-xcrr4"
Jul 27 21:09:07.486471 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:09:07.486437    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dfxp6\" (UniqueName: \"kubernetes.io/projected/508ed916-ee7b-44c9-a831-5bddfafdc8b1-kube-api-access-dfxp6\") pod \"console-745654d88d-xcrr4\" (UID: \"508ed916-ee7b-44c9-a831-5bddfafdc8b1\") " pod="openshift-console/console-745654d88d-xcrr4"
Jul 27 21:09:07.486471 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:09:07.486474    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/508ed916-ee7b-44c9-a831-5bddfafdc8b1-trusted-ca-bundle\") pod \"console-745654d88d-xcrr4\" (UID: \"508ed916-ee7b-44c9-a831-5bddfafdc8b1\") " pod="openshift-console/console-745654d88d-xcrr4"
Jul 27 21:09:07.486744 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:09:07.486498    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/508ed916-ee7b-44c9-a831-5bddfafdc8b1-service-ca\") pod \"console-745654d88d-xcrr4\" (UID: \"508ed916-ee7b-44c9-a831-5bddfafdc8b1\") " pod="openshift-console/console-745654d88d-xcrr4"
Jul 27 21:09:07.486744 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:09:07.486547    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/508ed916-ee7b-44c9-a831-5bddfafdc8b1-console-serving-cert\") pod \"console-745654d88d-xcrr4\" (UID: \"508ed916-ee7b-44c9-a831-5bddfafdc8b1\") " pod="openshift-console/console-745654d88d-xcrr4"
Jul 27 21:09:07.486744 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:09:07.486578    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/508ed916-ee7b-44c9-a831-5bddfafdc8b1-console-oauth-config\") pod \"console-745654d88d-xcrr4\" (UID: \"508ed916-ee7b-44c9-a831-5bddfafdc8b1\") " pod="openshift-console/console-745654d88d-xcrr4"
Jul 27 21:09:07.486744 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:09:07.486609    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/508ed916-ee7b-44c9-a831-5bddfafdc8b1-console-config\") pod \"console-745654d88d-xcrr4\" (UID: \"508ed916-ee7b-44c9-a831-5bddfafdc8b1\") " pod="openshift-console/console-745654d88d-xcrr4"
Jul 27 21:09:07.486744 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:09:07.486633    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/508ed916-ee7b-44c9-a831-5bddfafdc8b1-oauth-serving-cert\") pod \"console-745654d88d-xcrr4\" (UID: \"508ed916-ee7b-44c9-a831-5bddfafdc8b1\") " pod="openshift-console/console-745654d88d-xcrr4"
Jul 27 21:09:07.487414 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:09:07.487387    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/508ed916-ee7b-44c9-a831-5bddfafdc8b1-oauth-serving-cert\") pod \"console-745654d88d-xcrr4\" (UID: \"508ed916-ee7b-44c9-a831-5bddfafdc8b1\") " pod="openshift-console/console-745654d88d-xcrr4"
Jul 27 21:09:07.487414 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:09:07.487402    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/508ed916-ee7b-44c9-a831-5bddfafdc8b1-service-ca\") pod \"console-745654d88d-xcrr4\" (UID: \"508ed916-ee7b-44c9-a831-5bddfafdc8b1\") " pod="openshift-console/console-745654d88d-xcrr4"
Jul 27 21:09:07.487553 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:09:07.487465    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/508ed916-ee7b-44c9-a831-5bddfafdc8b1-trusted-ca-bundle\") pod \"console-745654d88d-xcrr4\" (UID: \"508ed916-ee7b-44c9-a831-5bddfafdc8b1\") " pod="openshift-console/console-745654d88d-xcrr4"
Jul 27 21:09:07.487553 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:09:07.487473    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/508ed916-ee7b-44c9-a831-5bddfafdc8b1-console-config\") pod \"console-745654d88d-xcrr4\" (UID: \"508ed916-ee7b-44c9-a831-5bddfafdc8b1\") " pod="openshift-console/console-745654d88d-xcrr4"
Jul 27 21:09:07.489082 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:09:07.489057    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/508ed916-ee7b-44c9-a831-5bddfafdc8b1-console-oauth-config\") pod \"console-745654d88d-xcrr4\" (UID: \"508ed916-ee7b-44c9-a831-5bddfafdc8b1\") " pod="openshift-console/console-745654d88d-xcrr4"
Jul 27 21:09:07.489171 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:09:07.489153    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/508ed916-ee7b-44c9-a831-5bddfafdc8b1-console-serving-cert\") pod \"console-745654d88d-xcrr4\" (UID: \"508ed916-ee7b-44c9-a831-5bddfafdc8b1\") " pod="openshift-console/console-745654d88d-xcrr4"
Jul 27 21:09:07.495698 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:09:07.495678    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dfxp6\" (UniqueName: \"kubernetes.io/projected/508ed916-ee7b-44c9-a831-5bddfafdc8b1-kube-api-access-dfxp6\") pod \"console-745654d88d-xcrr4\" (UID: \"508ed916-ee7b-44c9-a831-5bddfafdc8b1\") " pod="openshift-console/console-745654d88d-xcrr4"
Jul 27 21:09:07.629761 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:09:07.629731    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-745654d88d-xcrr4"
Jul 27 21:09:07.764245 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:09:07.764205    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-745654d88d-xcrr4"]
Jul 27 21:09:07.766532 ip-10-0-131-133 kubenswrapper[2559]: W0727 21:09:07.766501    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod508ed916_ee7b_44c9_a831_5bddfafdc8b1.slice/crio-d377f1f0ce47542d584f06ee7be7fba60c4d9910c38926456422a78a4692343f WatchSource:0}: Error finding container d377f1f0ce47542d584f06ee7be7fba60c4d9910c38926456422a78a4692343f: Status 404 returned error can't find the container with id d377f1f0ce47542d584f06ee7be7fba60c4d9910c38926456422a78a4692343f
Jul 27 21:09:08.322634 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:09:08.322602    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-745654d88d-xcrr4" event={"ID":"508ed916-ee7b-44c9-a831-5bddfafdc8b1","Type":"ContainerStarted","Data":"eba074652f86161a9183ba579dfba86e41adbff2acf8ecbf04082b7d94277ff7"}
Jul 27 21:09:08.323018 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:09:08.322647    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-745654d88d-xcrr4" event={"ID":"508ed916-ee7b-44c9-a831-5bddfafdc8b1","Type":"ContainerStarted","Data":"d377f1f0ce47542d584f06ee7be7fba60c4d9910c38926456422a78a4692343f"}
Jul 27 21:09:08.344155 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:09:08.344081    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-745654d88d-xcrr4" podStartSLOduration=1.344065337 podStartE2EDuration="1.344065337s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 21:09:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 21:09:08.343495424 +0000 UTC m=+276.922679432" watchObservedRunningTime="2026-07-27 21:09:08.344065337 +0000 UTC m=+276.923249343"
Jul 27 21:09:17.630128 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:09:17.630091    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-745654d88d-xcrr4"
Jul 27 21:09:17.630615 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:09:17.630193    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-745654d88d-xcrr4"
Jul 27 21:09:17.635292 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:09:17.635271    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-745654d88d-xcrr4"
Jul 27 21:09:18.366052 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:09:18.366024    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-745654d88d-xcrr4"
Jul 27 21:09:18.415642 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:09:18.415609    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-58b6b694f4-gqmmt"]
Jul 27 21:09:31.847186 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:09:31.847159    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-bjvbc_cba53c9a-64c0-4afa-8731-35521e196622/console-operator/1.log"
Jul 27 21:09:31.853722 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:09:31.853698    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-bjvbc_cba53c9a-64c0-4afa-8731-35521e196622/console-operator/1.log"
Jul 27 21:09:31.857451 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:09:31.857415    2559 kubelet.go:1648] "Image garbage collection succeeded"
Jul 27 21:09:43.435714 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:09:43.435651    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-58b6b694f4-gqmmt" podUID="81b570c4-4195-43da-8b76-4d77909a5a43" containerName="console" containerID="cri-o://22a793bf236f3d2b023bd88dd21c07f2c60aff2203952504f3009831016ca93d" gracePeriod=15
Jul 27 21:09:43.675986 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:09:43.675956    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-58b6b694f4-gqmmt"
Jul 27 21:09:43.783982 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:09:43.783893    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/81b570c4-4195-43da-8b76-4d77909a5a43-console-oauth-config\") pod \"81b570c4-4195-43da-8b76-4d77909a5a43\" (UID: \"81b570c4-4195-43da-8b76-4d77909a5a43\") "
Jul 27 21:09:43.783982 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:09:43.783935    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/81b570c4-4195-43da-8b76-4d77909a5a43-console-serving-cert\") pod \"81b570c4-4195-43da-8b76-4d77909a5a43\" (UID: \"81b570c4-4195-43da-8b76-4d77909a5a43\") "
Jul 27 21:09:43.784210 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:09:43.784002    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/81b570c4-4195-43da-8b76-4d77909a5a43-console-config\") pod \"81b570c4-4195-43da-8b76-4d77909a5a43\" (UID: \"81b570c4-4195-43da-8b76-4d77909a5a43\") "
Jul 27 21:09:43.784210 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:09:43.784032    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/81b570c4-4195-43da-8b76-4d77909a5a43-service-ca\") pod \"81b570c4-4195-43da-8b76-4d77909a5a43\" (UID: \"81b570c4-4195-43da-8b76-4d77909a5a43\") "
Jul 27 21:09:43.784210 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:09:43.784048    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-g2bwz\" (UniqueName: \"kubernetes.io/projected/81b570c4-4195-43da-8b76-4d77909a5a43-kube-api-access-g2bwz\") pod \"81b570c4-4195-43da-8b76-4d77909a5a43\" (UID: \"81b570c4-4195-43da-8b76-4d77909a5a43\") "
Jul 27 21:09:43.784210 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:09:43.784075    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/81b570c4-4195-43da-8b76-4d77909a5a43-oauth-serving-cert\") pod \"81b570c4-4195-43da-8b76-4d77909a5a43\" (UID: \"81b570c4-4195-43da-8b76-4d77909a5a43\") "
Jul 27 21:09:43.784210 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:09:43.784110    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/81b570c4-4195-43da-8b76-4d77909a5a43-trusted-ca-bundle\") pod \"81b570c4-4195-43da-8b76-4d77909a5a43\" (UID: \"81b570c4-4195-43da-8b76-4d77909a5a43\") "
Jul 27 21:09:43.784516 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:09:43.784489    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/81b570c4-4195-43da-8b76-4d77909a5a43-service-ca" (OuterVolumeSpecName: "service-ca") pod "81b570c4-4195-43da-8b76-4d77909a5a43" (UID: "81b570c4-4195-43da-8b76-4d77909a5a43"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 21:09:43.784567 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:09:43.784540    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/81b570c4-4195-43da-8b76-4d77909a5a43-console-config" (OuterVolumeSpecName: "console-config") pod "81b570c4-4195-43da-8b76-4d77909a5a43" (UID: "81b570c4-4195-43da-8b76-4d77909a5a43"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 21:09:43.784618 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:09:43.784561    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/81b570c4-4195-43da-8b76-4d77909a5a43-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "81b570c4-4195-43da-8b76-4d77909a5a43" (UID: "81b570c4-4195-43da-8b76-4d77909a5a43"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 21:09:43.784618 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:09:43.784603    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/81b570c4-4195-43da-8b76-4d77909a5a43-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "81b570c4-4195-43da-8b76-4d77909a5a43" (UID: "81b570c4-4195-43da-8b76-4d77909a5a43"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 21:09:43.786180 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:09:43.786154    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/81b570c4-4195-43da-8b76-4d77909a5a43-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "81b570c4-4195-43da-8b76-4d77909a5a43" (UID: "81b570c4-4195-43da-8b76-4d77909a5a43"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 21:09:43.786321 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:09:43.786180    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/81b570c4-4195-43da-8b76-4d77909a5a43-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "81b570c4-4195-43da-8b76-4d77909a5a43" (UID: "81b570c4-4195-43da-8b76-4d77909a5a43"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 21:09:43.786321 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:09:43.786294    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/81b570c4-4195-43da-8b76-4d77909a5a43-kube-api-access-g2bwz" (OuterVolumeSpecName: "kube-api-access-g2bwz") pod "81b570c4-4195-43da-8b76-4d77909a5a43" (UID: "81b570c4-4195-43da-8b76-4d77909a5a43"). InnerVolumeSpecName "kube-api-access-g2bwz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 21:09:43.885156 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:09:43.885124    2559 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/81b570c4-4195-43da-8b76-4d77909a5a43-console-config\") on node \"ip-10-0-131-133.ec2.internal\" DevicePath \"\""
Jul 27 21:09:43.885156 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:09:43.885150    2559 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/81b570c4-4195-43da-8b76-4d77909a5a43-service-ca\") on node \"ip-10-0-131-133.ec2.internal\" DevicePath \"\""
Jul 27 21:09:43.885156 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:09:43.885160    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g2bwz\" (UniqueName: \"kubernetes.io/projected/81b570c4-4195-43da-8b76-4d77909a5a43-kube-api-access-g2bwz\") on node \"ip-10-0-131-133.ec2.internal\" DevicePath \"\""
Jul 27 21:09:43.885156 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:09:43.885168    2559 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/81b570c4-4195-43da-8b76-4d77909a5a43-oauth-serving-cert\") on node \"ip-10-0-131-133.ec2.internal\" DevicePath \"\""
Jul 27 21:09:43.885448 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:09:43.885177    2559 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/81b570c4-4195-43da-8b76-4d77909a5a43-trusted-ca-bundle\") on node \"ip-10-0-131-133.ec2.internal\" DevicePath \"\""
Jul 27 21:09:43.885448 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:09:43.885185    2559 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/81b570c4-4195-43da-8b76-4d77909a5a43-console-oauth-config\") on node \"ip-10-0-131-133.ec2.internal\" DevicePath \"\""
Jul 27 21:09:43.885448 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:09:43.885193    2559 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/81b570c4-4195-43da-8b76-4d77909a5a43-console-serving-cert\") on node \"ip-10-0-131-133.ec2.internal\" DevicePath \"\""
Jul 27 21:09:44.475555 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:09:44.475517    2559 generic.go:356] "Generic (PLEG): container finished" podID="81b570c4-4195-43da-8b76-4d77909a5a43" containerID="22a793bf236f3d2b023bd88dd21c07f2c60aff2203952504f3009831016ca93d" exitCode=0
Jul 27 21:09:44.476005 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:09:44.475574    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-58b6b694f4-gqmmt"
Jul 27 21:09:44.476005 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:09:44.475606    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-58b6b694f4-gqmmt" event={"ID":"81b570c4-4195-43da-8b76-4d77909a5a43","Type":"ContainerDied","Data":"22a793bf236f3d2b023bd88dd21c07f2c60aff2203952504f3009831016ca93d"}
Jul 27 21:09:44.476005 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:09:44.475677    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-58b6b694f4-gqmmt" event={"ID":"81b570c4-4195-43da-8b76-4d77909a5a43","Type":"ContainerDied","Data":"7494d323356f77abd2e607ecdf38adab11eeb51c5419f8aed74dd105f6460eb5"}
Jul 27 21:09:44.476005 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:09:44.475698    2559 scope.go:117] "RemoveContainer" containerID="22a793bf236f3d2b023bd88dd21c07f2c60aff2203952504f3009831016ca93d"
Jul 27 21:09:44.483900 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:09:44.483884    2559 scope.go:117] "RemoveContainer" containerID="22a793bf236f3d2b023bd88dd21c07f2c60aff2203952504f3009831016ca93d"
Jul 27 21:09:44.484163 ip-10-0-131-133 kubenswrapper[2559]: E0727 21:09:44.484145    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"22a793bf236f3d2b023bd88dd21c07f2c60aff2203952504f3009831016ca93d\": container with ID starting with 22a793bf236f3d2b023bd88dd21c07f2c60aff2203952504f3009831016ca93d not found: ID does not exist" containerID="22a793bf236f3d2b023bd88dd21c07f2c60aff2203952504f3009831016ca93d"
Jul 27 21:09:44.484228 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:09:44.484171    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"22a793bf236f3d2b023bd88dd21c07f2c60aff2203952504f3009831016ca93d"} err="failed to get container status \"22a793bf236f3d2b023bd88dd21c07f2c60aff2203952504f3009831016ca93d\": rpc error: code = NotFound desc = could not find container \"22a793bf236f3d2b023bd88dd21c07f2c60aff2203952504f3009831016ca93d\": container with ID starting with 22a793bf236f3d2b023bd88dd21c07f2c60aff2203952504f3009831016ca93d not found: ID does not exist"
Jul 27 21:09:44.494833 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:09:44.494802    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-58b6b694f4-gqmmt"]
Jul 27 21:09:44.498432 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:09:44.498407    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-58b6b694f4-gqmmt"]
Jul 27 21:09:45.981108 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:09:45.981077    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="81b570c4-4195-43da-8b76-4d77909a5a43" path="/var/lib/kubelet/pods/81b570c4-4195-43da-8b76-4d77909a5a43/volumes"
Jul 27 21:11:01.927932 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:11:01.927897    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5mz7tw"]
Jul 27 21:11:01.928394 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:11:01.928233    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="81b570c4-4195-43da-8b76-4d77909a5a43" containerName="console"
Jul 27 21:11:01.928394 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:11:01.928244    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="81b570c4-4195-43da-8b76-4d77909a5a43" containerName="console"
Jul 27 21:11:01.928394 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:11:01.928316    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="81b570c4-4195-43da-8b76-4d77909a5a43" containerName="console"
Jul 27 21:11:01.931401 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:11:01.931383    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5mz7tw"
Jul 27 21:11:01.934050 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:11:01.934028    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-gznkt\""
Jul 27 21:11:01.934360 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:11:01.934343    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Jul 27 21:11:01.934777 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:11:01.934761    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Jul 27 21:11:01.941037 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:11:01.941011    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5mz7tw"]
Jul 27 21:11:02.031032 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:11:02.030992    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/02047607-a034-4744-9766-6b44f684d426-bundle\") pod \"320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5mz7tw\" (UID: \"02047607-a034-4744-9766-6b44f684d426\") " pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5mz7tw"
Jul 27 21:11:02.031192 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:11:02.031057    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/02047607-a034-4744-9766-6b44f684d426-util\") pod \"320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5mz7tw\" (UID: \"02047607-a034-4744-9766-6b44f684d426\") " pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5mz7tw"
Jul 27 21:11:02.031192 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:11:02.031087    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-srqn9\" (UniqueName: \"kubernetes.io/projected/02047607-a034-4744-9766-6b44f684d426-kube-api-access-srqn9\") pod \"320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5mz7tw\" (UID: \"02047607-a034-4744-9766-6b44f684d426\") " pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5mz7tw"
Jul 27 21:11:02.131866 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:11:02.131831    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/02047607-a034-4744-9766-6b44f684d426-util\") pod \"320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5mz7tw\" (UID: \"02047607-a034-4744-9766-6b44f684d426\") " pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5mz7tw"
Jul 27 21:11:02.132046 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:11:02.131882    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-srqn9\" (UniqueName: \"kubernetes.io/projected/02047607-a034-4744-9766-6b44f684d426-kube-api-access-srqn9\") pod \"320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5mz7tw\" (UID: \"02047607-a034-4744-9766-6b44f684d426\") " pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5mz7tw"
Jul 27 21:11:02.132046 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:11:02.131940    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/02047607-a034-4744-9766-6b44f684d426-bundle\") pod \"320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5mz7tw\" (UID: \"02047607-a034-4744-9766-6b44f684d426\") " pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5mz7tw"
Jul 27 21:11:02.132344 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:11:02.132321    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/02047607-a034-4744-9766-6b44f684d426-util\") pod \"320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5mz7tw\" (UID: \"02047607-a034-4744-9766-6b44f684d426\") " pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5mz7tw"
Jul 27 21:11:02.132385 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:11:02.132331    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/02047607-a034-4744-9766-6b44f684d426-bundle\") pod \"320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5mz7tw\" (UID: \"02047607-a034-4744-9766-6b44f684d426\") " pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5mz7tw"
Jul 27 21:11:02.140563 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:11:02.140531    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-srqn9\" (UniqueName: \"kubernetes.io/projected/02047607-a034-4744-9766-6b44f684d426-kube-api-access-srqn9\") pod \"320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5mz7tw\" (UID: \"02047607-a034-4744-9766-6b44f684d426\") " pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5mz7tw"
Jul 27 21:11:02.240525 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:11:02.240444    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5mz7tw"
Jul 27 21:11:02.385428 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:11:02.385389    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5mz7tw"]
Jul 27 21:11:02.386649 ip-10-0-131-133 kubenswrapper[2559]: W0727 21:11:02.386616    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod02047607_a034_4744_9766_6b44f684d426.slice/crio-9b1a8fe9fc69caacfd4842b6d19b2a8ffddd0d47237f4fc567044a53c4e6916e WatchSource:0}: Error finding container 9b1a8fe9fc69caacfd4842b6d19b2a8ffddd0d47237f4fc567044a53c4e6916e: Status 404 returned error can't find the container with id 9b1a8fe9fc69caacfd4842b6d19b2a8ffddd0d47237f4fc567044a53c4e6916e
Jul 27 21:11:02.388695 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:11:02.388672    2559 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 27 21:11:02.789379 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:11:02.789346    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5mz7tw" event={"ID":"02047607-a034-4744-9766-6b44f684d426","Type":"ContainerStarted","Data":"9b1a8fe9fc69caacfd4842b6d19b2a8ffddd0d47237f4fc567044a53c4e6916e"}
Jul 27 21:11:08.817087 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:11:08.817058    2559 generic.go:356] "Generic (PLEG): container finished" podID="02047607-a034-4744-9766-6b44f684d426" containerID="6077ab77a9b0871038225697327b4ee0221a06e6cb67e1e05e93cd6c0ba53df1" exitCode=0
Jul 27 21:11:08.817541 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:11:08.817102    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5mz7tw" event={"ID":"02047607-a034-4744-9766-6b44f684d426","Type":"ContainerDied","Data":"6077ab77a9b0871038225697327b4ee0221a06e6cb67e1e05e93cd6c0ba53df1"}
Jul 27 21:11:11.835274 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:11:11.835232    2559 generic.go:356] "Generic (PLEG): container finished" podID="02047607-a034-4744-9766-6b44f684d426" containerID="ffb6e24d96267d4ff3fff8200d5305c7d460e94399e906287e1b4158e921091a" exitCode=0
Jul 27 21:11:11.835731 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:11:11.835311    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5mz7tw" event={"ID":"02047607-a034-4744-9766-6b44f684d426","Type":"ContainerDied","Data":"ffb6e24d96267d4ff3fff8200d5305c7d460e94399e906287e1b4158e921091a"}
Jul 27 21:11:19.872433 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:11:19.872389    2559 generic.go:356] "Generic (PLEG): container finished" podID="02047607-a034-4744-9766-6b44f684d426" containerID="822bfd2042cfd113a9c9b5a81301e8c634d9c24894e971a8970c942094fafa15" exitCode=0
Jul 27 21:11:19.872823 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:11:19.872476    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5mz7tw" event={"ID":"02047607-a034-4744-9766-6b44f684d426","Type":"ContainerDied","Data":"822bfd2042cfd113a9c9b5a81301e8c634d9c24894e971a8970c942094fafa15"}
Jul 27 21:11:21.006107 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:11:21.006077    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5mz7tw"
Jul 27 21:11:21.098931 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:11:21.098899    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/02047607-a034-4744-9766-6b44f684d426-bundle\") pod \"02047607-a034-4744-9766-6b44f684d426\" (UID: \"02047607-a034-4744-9766-6b44f684d426\") "
Jul 27 21:11:21.099130 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:11:21.098975    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/02047607-a034-4744-9766-6b44f684d426-util\") pod \"02047607-a034-4744-9766-6b44f684d426\" (UID: \"02047607-a034-4744-9766-6b44f684d426\") "
Jul 27 21:11:21.099130 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:11:21.099008    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-srqn9\" (UniqueName: \"kubernetes.io/projected/02047607-a034-4744-9766-6b44f684d426-kube-api-access-srqn9\") pod \"02047607-a034-4744-9766-6b44f684d426\" (UID: \"02047607-a034-4744-9766-6b44f684d426\") "
Jul 27 21:11:21.099796 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:11:21.099757    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/02047607-a034-4744-9766-6b44f684d426-bundle" (OuterVolumeSpecName: "bundle") pod "02047607-a034-4744-9766-6b44f684d426" (UID: "02047607-a034-4744-9766-6b44f684d426"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 21:11:21.101360 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:11:21.101329    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/02047607-a034-4744-9766-6b44f684d426-kube-api-access-srqn9" (OuterVolumeSpecName: "kube-api-access-srqn9") pod "02047607-a034-4744-9766-6b44f684d426" (UID: "02047607-a034-4744-9766-6b44f684d426"). InnerVolumeSpecName "kube-api-access-srqn9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 21:11:21.103490 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:11:21.103461    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/02047607-a034-4744-9766-6b44f684d426-util" (OuterVolumeSpecName: "util") pod "02047607-a034-4744-9766-6b44f684d426" (UID: "02047607-a034-4744-9766-6b44f684d426"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 21:11:21.199713 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:11:21.199624    2559 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/02047607-a034-4744-9766-6b44f684d426-util\") on node \"ip-10-0-131-133.ec2.internal\" DevicePath \"\""
Jul 27 21:11:21.199713 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:11:21.199654    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-srqn9\" (UniqueName: \"kubernetes.io/projected/02047607-a034-4744-9766-6b44f684d426-kube-api-access-srqn9\") on node \"ip-10-0-131-133.ec2.internal\" DevicePath \"\""
Jul 27 21:11:21.199713 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:11:21.199664    2559 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/02047607-a034-4744-9766-6b44f684d426-bundle\") on node \"ip-10-0-131-133.ec2.internal\" DevicePath \"\""
Jul 27 21:11:21.882307 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:11:21.882270    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5mz7tw" event={"ID":"02047607-a034-4744-9766-6b44f684d426","Type":"ContainerDied","Data":"9b1a8fe9fc69caacfd4842b6d19b2a8ffddd0d47237f4fc567044a53c4e6916e"}
Jul 27 21:11:21.882484 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:11:21.882324    2559 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9b1a8fe9fc69caacfd4842b6d19b2a8ffddd0d47237f4fc567044a53c4e6916e"
Jul 27 21:11:21.882484 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:11:21.882283    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5mz7tw"
Jul 27 21:11:25.507598 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:11:25.507562    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-keda/custom-metrics-autoscaler-operator-6969cb7dfb-44flg"]
Jul 27 21:11:25.507996 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:11:25.507896    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="02047607-a034-4744-9766-6b44f684d426" containerName="extract"
Jul 27 21:11:25.507996 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:11:25.507909    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="02047607-a034-4744-9766-6b44f684d426" containerName="extract"
Jul 27 21:11:25.507996 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:11:25.507921    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="02047607-a034-4744-9766-6b44f684d426" containerName="util"
Jul 27 21:11:25.507996 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:11:25.507926    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="02047607-a034-4744-9766-6b44f684d426" containerName="util"
Jul 27 21:11:25.507996 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:11:25.507935    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="02047607-a034-4744-9766-6b44f684d426" containerName="pull"
Jul 27 21:11:25.507996 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:11:25.507941    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="02047607-a034-4744-9766-6b44f684d426" containerName="pull"
Jul 27 21:11:25.507996 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:11:25.507993    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="02047607-a034-4744-9766-6b44f684d426" containerName="extract"
Jul 27 21:11:25.523877 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:11:25.523844    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/custom-metrics-autoscaler-operator-6969cb7dfb-44flg"
Jul 27 21:11:25.525571 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:11:25.525546    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/custom-metrics-autoscaler-operator-6969cb7dfb-44flg"]
Jul 27 21:11:25.526294 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:11:25.526269    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"openshift-service-ca.crt\""
Jul 27 21:11:25.526401 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:11:25.526351    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"kedaorg-certs\""
Jul 27 21:11:25.526458 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:11:25.526402    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"custom-metrics-autoscaler-operator-dockercfg-hk44z\""
Jul 27 21:11:25.526458 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:11:25.526356    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"kube-root-ca.crt\""
Jul 27 21:11:25.536629 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:11:25.536606    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bs98q\" (UniqueName: \"kubernetes.io/projected/30139d97-cbed-4e2c-b2a5-dcf539ae2e65-kube-api-access-bs98q\") pod \"custom-metrics-autoscaler-operator-6969cb7dfb-44flg\" (UID: \"30139d97-cbed-4e2c-b2a5-dcf539ae2e65\") " pod="openshift-keda/custom-metrics-autoscaler-operator-6969cb7dfb-44flg"
Jul 27 21:11:25.536727 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:11:25.536638    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/secret/30139d97-cbed-4e2c-b2a5-dcf539ae2e65-certificates\") pod \"custom-metrics-autoscaler-operator-6969cb7dfb-44flg\" (UID: \"30139d97-cbed-4e2c-b2a5-dcf539ae2e65\") " pod="openshift-keda/custom-metrics-autoscaler-operator-6969cb7dfb-44flg"
Jul 27 21:11:25.637479 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:11:25.637440    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bs98q\" (UniqueName: \"kubernetes.io/projected/30139d97-cbed-4e2c-b2a5-dcf539ae2e65-kube-api-access-bs98q\") pod \"custom-metrics-autoscaler-operator-6969cb7dfb-44flg\" (UID: \"30139d97-cbed-4e2c-b2a5-dcf539ae2e65\") " pod="openshift-keda/custom-metrics-autoscaler-operator-6969cb7dfb-44flg"
Jul 27 21:11:25.637682 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:11:25.637483    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/secret/30139d97-cbed-4e2c-b2a5-dcf539ae2e65-certificates\") pod \"custom-metrics-autoscaler-operator-6969cb7dfb-44flg\" (UID: \"30139d97-cbed-4e2c-b2a5-dcf539ae2e65\") " pod="openshift-keda/custom-metrics-autoscaler-operator-6969cb7dfb-44flg"
Jul 27 21:11:25.639844 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:11:25.639817    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"certificates\" (UniqueName: \"kubernetes.io/secret/30139d97-cbed-4e2c-b2a5-dcf539ae2e65-certificates\") pod \"custom-metrics-autoscaler-operator-6969cb7dfb-44flg\" (UID: \"30139d97-cbed-4e2c-b2a5-dcf539ae2e65\") " pod="openshift-keda/custom-metrics-autoscaler-operator-6969cb7dfb-44flg"
Jul 27 21:11:25.645157 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:11:25.645133    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bs98q\" (UniqueName: \"kubernetes.io/projected/30139d97-cbed-4e2c-b2a5-dcf539ae2e65-kube-api-access-bs98q\") pod \"custom-metrics-autoscaler-operator-6969cb7dfb-44flg\" (UID: \"30139d97-cbed-4e2c-b2a5-dcf539ae2e65\") " pod="openshift-keda/custom-metrics-autoscaler-operator-6969cb7dfb-44flg"
Jul 27 21:11:25.835410 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:11:25.835322    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/custom-metrics-autoscaler-operator-6969cb7dfb-44flg"
Jul 27 21:11:25.989228 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:11:25.989175    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/custom-metrics-autoscaler-operator-6969cb7dfb-44flg"]
Jul 27 21:11:25.992190 ip-10-0-131-133 kubenswrapper[2559]: W0727 21:11:25.992165    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod30139d97_cbed_4e2c_b2a5_dcf539ae2e65.slice/crio-4c6375d34fbccb4925f17dd2de77f063936cdaf1d7570c2e5a3d4a53e43fe188 WatchSource:0}: Error finding container 4c6375d34fbccb4925f17dd2de77f063936cdaf1d7570c2e5a3d4a53e43fe188: Status 404 returned error can't find the container with id 4c6375d34fbccb4925f17dd2de77f063936cdaf1d7570c2e5a3d4a53e43fe188
Jul 27 21:11:26.904515 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:11:26.904478    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/custom-metrics-autoscaler-operator-6969cb7dfb-44flg" event={"ID":"30139d97-cbed-4e2c-b2a5-dcf539ae2e65","Type":"ContainerStarted","Data":"4c6375d34fbccb4925f17dd2de77f063936cdaf1d7570c2e5a3d4a53e43fe188"}
Jul 27 21:11:29.674763 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:11:29.674724    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-keda/keda-operator-77f648d694-glhkm"]
Jul 27 21:11:29.677127 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:11:29.677108    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-operator-77f648d694-glhkm"
Jul 27 21:11:29.679413 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:11:29.679393    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-operator-certs\""
Jul 27 21:11:29.679413 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:11:29.679407    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-operator-dockercfg-62lhs\""
Jul 27 21:11:29.679567 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:11:29.679396    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"keda-ocp-cabundle\""
Jul 27 21:11:29.689248 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:11:29.689198    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-operator-77f648d694-glhkm"]
Jul 27 21:11:29.775748 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:11:29.775716    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lp4m2\" (UniqueName: \"kubernetes.io/projected/2d06d4e0-e27c-485c-a0d5-c0adf2552fda-kube-api-access-lp4m2\") pod \"keda-operator-77f648d694-glhkm\" (UID: \"2d06d4e0-e27c-485c-a0d5-c0adf2552fda\") " pod="openshift-keda/keda-operator-77f648d694-glhkm"
Jul 27 21:11:29.775941 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:11:29.775760    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/2d06d4e0-e27c-485c-a0d5-c0adf2552fda-certificates\") pod \"keda-operator-77f648d694-glhkm\" (UID: \"2d06d4e0-e27c-485c-a0d5-c0adf2552fda\") " pod="openshift-keda/keda-operator-77f648d694-glhkm"
Jul 27 21:11:29.775941 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:11:29.775885    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle0\" (UniqueName: \"kubernetes.io/configmap/2d06d4e0-e27c-485c-a0d5-c0adf2552fda-cabundle0\") pod \"keda-operator-77f648d694-glhkm\" (UID: \"2d06d4e0-e27c-485c-a0d5-c0adf2552fda\") " pod="openshift-keda/keda-operator-77f648d694-glhkm"
Jul 27 21:11:29.876748 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:11:29.876715    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lp4m2\" (UniqueName: \"kubernetes.io/projected/2d06d4e0-e27c-485c-a0d5-c0adf2552fda-kube-api-access-lp4m2\") pod \"keda-operator-77f648d694-glhkm\" (UID: \"2d06d4e0-e27c-485c-a0d5-c0adf2552fda\") " pod="openshift-keda/keda-operator-77f648d694-glhkm"
Jul 27 21:11:29.876748 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:11:29.876760    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/2d06d4e0-e27c-485c-a0d5-c0adf2552fda-certificates\") pod \"keda-operator-77f648d694-glhkm\" (UID: \"2d06d4e0-e27c-485c-a0d5-c0adf2552fda\") " pod="openshift-keda/keda-operator-77f648d694-glhkm"
Jul 27 21:11:29.876980 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:11:29.876815    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle0\" (UniqueName: \"kubernetes.io/configmap/2d06d4e0-e27c-485c-a0d5-c0adf2552fda-cabundle0\") pod \"keda-operator-77f648d694-glhkm\" (UID: \"2d06d4e0-e27c-485c-a0d5-c0adf2552fda\") " pod="openshift-keda/keda-operator-77f648d694-glhkm"
Jul 27 21:11:29.876980 ip-10-0-131-133 kubenswrapper[2559]: E0727 21:11:29.876944    2559 projected.go:266] Couldn't get secret openshift-keda/keda-operator-certs: secret "keda-operator-certs" not found
Jul 27 21:11:29.876980 ip-10-0-131-133 kubenswrapper[2559]: E0727 21:11:29.876967    2559 secret.go:281] references non-existent secret key: ca.crt
Jul 27 21:11:29.876980 ip-10-0-131-133 kubenswrapper[2559]: E0727 21:11:29.876976    2559 projected.go:279] Couldn't get secret payload openshift-keda/kedaorg-certs: references non-existent secret key: ca.crt
Jul 27 21:11:29.877124 ip-10-0-131-133 kubenswrapper[2559]: E0727 21:11:29.876991    2559 projected.go:196] Error preparing data for projected volume certificates for pod openshift-keda/keda-operator-77f648d694-glhkm: [secret "keda-operator-certs" not found, references non-existent secret key: ca.crt]
Jul 27 21:11:29.877124 ip-10-0-131-133 kubenswrapper[2559]: E0727 21:11:29.877070    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2d06d4e0-e27c-485c-a0d5-c0adf2552fda-certificates podName:2d06d4e0-e27c-485c-a0d5-c0adf2552fda nodeName:}" failed. No retries permitted until 2026-07-27 21:11:30.377047531 +0000 UTC m=+418.956231536 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "certificates" (UniqueName: "kubernetes.io/projected/2d06d4e0-e27c-485c-a0d5-c0adf2552fda-certificates") pod "keda-operator-77f648d694-glhkm" (UID: "2d06d4e0-e27c-485c-a0d5-c0adf2552fda") : [secret "keda-operator-certs" not found, references non-existent secret key: ca.crt]
Jul 27 21:11:29.877724 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:11:29.877705    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle0\" (UniqueName: \"kubernetes.io/configmap/2d06d4e0-e27c-485c-a0d5-c0adf2552fda-cabundle0\") pod \"keda-operator-77f648d694-glhkm\" (UID: \"2d06d4e0-e27c-485c-a0d5-c0adf2552fda\") " pod="openshift-keda/keda-operator-77f648d694-glhkm"
Jul 27 21:11:29.886717 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:11:29.886682    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lp4m2\" (UniqueName: \"kubernetes.io/projected/2d06d4e0-e27c-485c-a0d5-c0adf2552fda-kube-api-access-lp4m2\") pod \"keda-operator-77f648d694-glhkm\" (UID: \"2d06d4e0-e27c-485c-a0d5-c0adf2552fda\") " pod="openshift-keda/keda-operator-77f648d694-glhkm"
Jul 27 21:11:29.922506 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:11:29.922465    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/custom-metrics-autoscaler-operator-6969cb7dfb-44flg" event={"ID":"30139d97-cbed-4e2c-b2a5-dcf539ae2e65","Type":"ContainerStarted","Data":"48620a9af35dfa2e06eff0cd38bf1a2b4ff9efcc03b6cef7b6351b415cb9c9dd"}
Jul 27 21:11:29.922676 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:11:29.922538    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-keda/custom-metrics-autoscaler-operator-6969cb7dfb-44flg"
Jul 27 21:11:29.942639 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:11:29.942518    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-keda/custom-metrics-autoscaler-operator-6969cb7dfb-44flg" podStartSLOduration=1.9778206759999999 podStartE2EDuration="4.94249841s" totalImagesPullingTime="2.964677734s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 21:11:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 21:11:29.941715272 +0000 UTC m=+418.520899277" watchObservedRunningTime="2026-07-27 21:11:29.94249841 +0000 UTC m=+418.521682418"
Jul 27 21:11:30.359579 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:11:30.359509    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-keda/keda-admission-56f9798bb4-9wmwk"]
Jul 27 21:11:30.361721 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:11:30.361705    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-admission-56f9798bb4-9wmwk"
Jul 27 21:11:30.363751 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:11:30.363731    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-admission-webhooks-certs\""
Jul 27 21:11:30.379596 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:11:30.379574    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-admission-56f9798bb4-9wmwk"]
Jul 27 21:11:30.381122 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:11:30.381097    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/2d06d4e0-e27c-485c-a0d5-c0adf2552fda-certificates\") pod \"keda-operator-77f648d694-glhkm\" (UID: \"2d06d4e0-e27c-485c-a0d5-c0adf2552fda\") " pod="openshift-keda/keda-operator-77f648d694-glhkm"
Jul 27 21:11:30.383876 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:11:30.383857    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/2d06d4e0-e27c-485c-a0d5-c0adf2552fda-certificates\") pod \"keda-operator-77f648d694-glhkm\" (UID: \"2d06d4e0-e27c-485c-a0d5-c0adf2552fda\") " pod="openshift-keda/keda-operator-77f648d694-glhkm"
Jul 27 21:11:30.482396 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:11:30.482361    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9vvfq\" (UniqueName: \"kubernetes.io/projected/3175fa00-6664-486e-95c7-4d04ac1c4a8a-kube-api-access-9vvfq\") pod \"keda-admission-56f9798bb4-9wmwk\" (UID: \"3175fa00-6664-486e-95c7-4d04ac1c4a8a\") " pod="openshift-keda/keda-admission-56f9798bb4-9wmwk"
Jul 27 21:11:30.482577 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:11:30.482416    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/3175fa00-6664-486e-95c7-4d04ac1c4a8a-certificates\") pod \"keda-admission-56f9798bb4-9wmwk\" (UID: \"3175fa00-6664-486e-95c7-4d04ac1c4a8a\") " pod="openshift-keda/keda-admission-56f9798bb4-9wmwk"
Jul 27 21:11:30.583274 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:11:30.583242    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9vvfq\" (UniqueName: \"kubernetes.io/projected/3175fa00-6664-486e-95c7-4d04ac1c4a8a-kube-api-access-9vvfq\") pod \"keda-admission-56f9798bb4-9wmwk\" (UID: \"3175fa00-6664-486e-95c7-4d04ac1c4a8a\") " pod="openshift-keda/keda-admission-56f9798bb4-9wmwk"
Jul 27 21:11:30.583463 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:11:30.583309    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/3175fa00-6664-486e-95c7-4d04ac1c4a8a-certificates\") pod \"keda-admission-56f9798bb4-9wmwk\" (UID: \"3175fa00-6664-486e-95c7-4d04ac1c4a8a\") " pod="openshift-keda/keda-admission-56f9798bb4-9wmwk"
Jul 27 21:11:30.585828 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:11:30.585801    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/3175fa00-6664-486e-95c7-4d04ac1c4a8a-certificates\") pod \"keda-admission-56f9798bb4-9wmwk\" (UID: \"3175fa00-6664-486e-95c7-4d04ac1c4a8a\") " pod="openshift-keda/keda-admission-56f9798bb4-9wmwk"
Jul 27 21:11:30.586746 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:11:30.586717    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-operator-77f648d694-glhkm"
Jul 27 21:11:30.594527 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:11:30.594501    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9vvfq\" (UniqueName: \"kubernetes.io/projected/3175fa00-6664-486e-95c7-4d04ac1c4a8a-kube-api-access-9vvfq\") pod \"keda-admission-56f9798bb4-9wmwk\" (UID: \"3175fa00-6664-486e-95c7-4d04ac1c4a8a\") " pod="openshift-keda/keda-admission-56f9798bb4-9wmwk"
Jul 27 21:11:30.672618 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:11:30.672587    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-admission-56f9798bb4-9wmwk"
Jul 27 21:11:30.731776 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:11:30.731640    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-operator-77f648d694-glhkm"]
Jul 27 21:11:30.736319 ip-10-0-131-133 kubenswrapper[2559]: W0727 21:11:30.736274    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2d06d4e0_e27c_485c_a0d5_c0adf2552fda.slice/crio-82b7c6a3f05fccfeb8ce86c34c4697abec8bfa70e07b4ecedc9c89b45d893559 WatchSource:0}: Error finding container 82b7c6a3f05fccfeb8ce86c34c4697abec8bfa70e07b4ecedc9c89b45d893559: Status 404 returned error can't find the container with id 82b7c6a3f05fccfeb8ce86c34c4697abec8bfa70e07b4ecedc9c89b45d893559
Jul 27 21:11:30.813571 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:11:30.813545    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-admission-56f9798bb4-9wmwk"]
Jul 27 21:11:30.815912 ip-10-0-131-133 kubenswrapper[2559]: W0727 21:11:30.815884    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3175fa00_6664_486e_95c7_4d04ac1c4a8a.slice/crio-6718dafefe526ba380de210feec3d50c8477907f1a9909837b7d6cc3346a52ae WatchSource:0}: Error finding container 6718dafefe526ba380de210feec3d50c8477907f1a9909837b7d6cc3346a52ae: Status 404 returned error can't find the container with id 6718dafefe526ba380de210feec3d50c8477907f1a9909837b7d6cc3346a52ae
Jul 27 21:11:30.928028 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:11:30.927927    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-admission-56f9798bb4-9wmwk" event={"ID":"3175fa00-6664-486e-95c7-4d04ac1c4a8a","Type":"ContainerStarted","Data":"6718dafefe526ba380de210feec3d50c8477907f1a9909837b7d6cc3346a52ae"}
Jul 27 21:11:30.928983 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:11:30.928952    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-operator-77f648d694-glhkm" event={"ID":"2d06d4e0-e27c-485c-a0d5-c0adf2552fda","Type":"ContainerStarted","Data":"82b7c6a3f05fccfeb8ce86c34c4697abec8bfa70e07b4ecedc9c89b45d893559"}
Jul 27 21:11:32.942163 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:11:32.942124    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-admission-56f9798bb4-9wmwk" event={"ID":"3175fa00-6664-486e-95c7-4d04ac1c4a8a","Type":"ContainerStarted","Data":"eaf4fcaf30ee53dece078f90278334108a9c09292e53d1481491ba75d9da8d04"}
Jul 27 21:11:32.942632 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:11:32.942262    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-keda/keda-admission-56f9798bb4-9wmwk"
Jul 27 21:11:32.963855 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:11:32.963793    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-keda/keda-admission-56f9798bb4-9wmwk" podStartSLOduration=1.5504496570000001 podStartE2EDuration="2.963771654s" totalImagesPullingTime="1.413321997s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 21:11:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 21:11:32.961206302 +0000 UTC m=+421.540390313" watchObservedRunningTime="2026-07-27 21:11:32.963771654 +0000 UTC m=+421.542955662"
Jul 27 21:11:34.952513 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:11:34.952478    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-operator-77f648d694-glhkm" event={"ID":"2d06d4e0-e27c-485c-a0d5-c0adf2552fda","Type":"ContainerStarted","Data":"615e9da3d3adb6da83cb3264306fcd8e4895e6c1e3d7f315e0622576552551e8"}
Jul 27 21:11:34.952912 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:11:34.952650    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-keda/keda-operator-77f648d694-glhkm"
Jul 27 21:11:34.970039 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:11:34.969990    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-keda/keda-operator-77f648d694-glhkm" podStartSLOduration=2.7550597039999998 podStartE2EDuration="5.969973595s" totalImagesPullingTime="3.214913891s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 21:11:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 21:11:34.96851727 +0000 UTC m=+423.547701279" watchObservedRunningTime="2026-07-27 21:11:34.969973595 +0000 UTC m=+423.549157602"
Jul 27 21:11:50.931756 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:11:50.931723    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-keda/custom-metrics-autoscaler-operator-6969cb7dfb-44flg"
Jul 27 21:11:53.949921 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:11:53.949892    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-keda/keda-admission-56f9798bb4-9wmwk"
Jul 27 21:11:55.959979 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:11:55.959949    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-keda/keda-operator-77f648d694-glhkm"
Jul 27 21:12:48.064250 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:12:48.064203    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/llmisvc-controller-manager-84fc959947-t8pnm"]
Jul 27 21:12:48.066599 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:12:48.066584    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/llmisvc-controller-manager-84fc959947-t8pnm"
Jul 27 21:12:48.069601 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:12:48.069579    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"openshift-service-ca.crt\""
Jul 27 21:12:48.070323 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:12:48.070301    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"llmisvc-controller-manager-dockercfg-ztrkl\""
Jul 27 21:12:48.070452 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:12:48.070345    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"llmisvc-webhook-server-cert\""
Jul 27 21:12:48.070452 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:12:48.070371    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"kube-root-ca.crt\""
Jul 27 21:12:48.079188 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:12:48.079166    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/llmisvc-controller-manager-84fc959947-t8pnm"]
Jul 27 21:12:48.091386 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:12:48.091364    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/seaweedfs-64568bcd49-p22l9"]
Jul 27 21:12:48.093911 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:12:48.093890    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-64568bcd49-p22l9"
Jul 27 21:12:48.096585 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:12:48.096567    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"default-dockercfg-d7slw\""
Jul 27 21:12:48.096696 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:12:48.096656    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"mlpipeline-s3-artifact\""
Jul 27 21:12:48.103068 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:12:48.103048    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-64568bcd49-p22l9"]
Jul 27 21:12:48.146568 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:12:48.146535    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fg7hp\" (UniqueName: \"kubernetes.io/projected/bb51a4e3-357d-4e40-9001-e5b6586860fd-kube-api-access-fg7hp\") pod \"llmisvc-controller-manager-84fc959947-t8pnm\" (UID: \"bb51a4e3-357d-4e40-9001-e5b6586860fd\") " pod="kserve/llmisvc-controller-manager-84fc959947-t8pnm"
Jul 27 21:12:48.146757 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:12:48.146583    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/bb51a4e3-357d-4e40-9001-e5b6586860fd-cert\") pod \"llmisvc-controller-manager-84fc959947-t8pnm\" (UID: \"bb51a4e3-357d-4e40-9001-e5b6586860fd\") " pod="kserve/llmisvc-controller-manager-84fc959947-t8pnm"
Jul 27 21:12:48.247886 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:12:48.247843    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fg7hp\" (UniqueName: \"kubernetes.io/projected/bb51a4e3-357d-4e40-9001-e5b6586860fd-kube-api-access-fg7hp\") pod \"llmisvc-controller-manager-84fc959947-t8pnm\" (UID: \"bb51a4e3-357d-4e40-9001-e5b6586860fd\") " pod="kserve/llmisvc-controller-manager-84fc959947-t8pnm"
Jul 27 21:12:48.247886 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:12:48.247888    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/bb51a4e3-357d-4e40-9001-e5b6586860fd-cert\") pod \"llmisvc-controller-manager-84fc959947-t8pnm\" (UID: \"bb51a4e3-357d-4e40-9001-e5b6586860fd\") " pod="kserve/llmisvc-controller-manager-84fc959947-t8pnm"
Jul 27 21:12:48.248138 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:12:48.247930    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9b86c\" (UniqueName: \"kubernetes.io/projected/349b045e-11fe-4879-ba47-a0f576ba6e25-kube-api-access-9b86c\") pod \"seaweedfs-64568bcd49-p22l9\" (UID: \"349b045e-11fe-4879-ba47-a0f576ba6e25\") " pod="kserve/seaweedfs-64568bcd49-p22l9"
Jul 27 21:12:48.248138 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:12:48.247962    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/349b045e-11fe-4879-ba47-a0f576ba6e25-data\") pod \"seaweedfs-64568bcd49-p22l9\" (UID: \"349b045e-11fe-4879-ba47-a0f576ba6e25\") " pod="kserve/seaweedfs-64568bcd49-p22l9"
Jul 27 21:12:48.248138 ip-10-0-131-133 kubenswrapper[2559]: E0727 21:12:48.248064    2559 secret.go:189] Couldn't get secret kserve/llmisvc-webhook-server-cert: secret "llmisvc-webhook-server-cert" not found
Jul 27 21:12:48.248138 ip-10-0-131-133 kubenswrapper[2559]: E0727 21:12:48.248120    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bb51a4e3-357d-4e40-9001-e5b6586860fd-cert podName:bb51a4e3-357d-4e40-9001-e5b6586860fd nodeName:}" failed. No retries permitted until 2026-07-27 21:12:48.748103478 +0000 UTC m=+497.327287462 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/bb51a4e3-357d-4e40-9001-e5b6586860fd-cert") pod "llmisvc-controller-manager-84fc959947-t8pnm" (UID: "bb51a4e3-357d-4e40-9001-e5b6586860fd") : secret "llmisvc-webhook-server-cert" not found
Jul 27 21:12:48.258576 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:12:48.258545    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fg7hp\" (UniqueName: \"kubernetes.io/projected/bb51a4e3-357d-4e40-9001-e5b6586860fd-kube-api-access-fg7hp\") pod \"llmisvc-controller-manager-84fc959947-t8pnm\" (UID: \"bb51a4e3-357d-4e40-9001-e5b6586860fd\") " pod="kserve/llmisvc-controller-manager-84fc959947-t8pnm"
Jul 27 21:12:48.349122 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:12:48.349033    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9b86c\" (UniqueName: \"kubernetes.io/projected/349b045e-11fe-4879-ba47-a0f576ba6e25-kube-api-access-9b86c\") pod \"seaweedfs-64568bcd49-p22l9\" (UID: \"349b045e-11fe-4879-ba47-a0f576ba6e25\") " pod="kserve/seaweedfs-64568bcd49-p22l9"
Jul 27 21:12:48.349122 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:12:48.349080    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/349b045e-11fe-4879-ba47-a0f576ba6e25-data\") pod \"seaweedfs-64568bcd49-p22l9\" (UID: \"349b045e-11fe-4879-ba47-a0f576ba6e25\") " pod="kserve/seaweedfs-64568bcd49-p22l9"
Jul 27 21:12:48.349492 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:12:48.349477    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/349b045e-11fe-4879-ba47-a0f576ba6e25-data\") pod \"seaweedfs-64568bcd49-p22l9\" (UID: \"349b045e-11fe-4879-ba47-a0f576ba6e25\") " pod="kserve/seaweedfs-64568bcd49-p22l9"
Jul 27 21:12:48.360335 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:12:48.360310    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9b86c\" (UniqueName: \"kubernetes.io/projected/349b045e-11fe-4879-ba47-a0f576ba6e25-kube-api-access-9b86c\") pod \"seaweedfs-64568bcd49-p22l9\" (UID: \"349b045e-11fe-4879-ba47-a0f576ba6e25\") " pod="kserve/seaweedfs-64568bcd49-p22l9"
Jul 27 21:12:48.403538 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:12:48.403499    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-64568bcd49-p22l9"
Jul 27 21:12:48.535073 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:12:48.535051    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-64568bcd49-p22l9"]
Jul 27 21:12:48.537836 ip-10-0-131-133 kubenswrapper[2559]: W0727 21:12:48.537811    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod349b045e_11fe_4879_ba47_a0f576ba6e25.slice/crio-786c8524100c2a29d910451de94c07df9fe0ef4d865f0cb700b6fd0a63bc53f6 WatchSource:0}: Error finding container 786c8524100c2a29d910451de94c07df9fe0ef4d865f0cb700b6fd0a63bc53f6: Status 404 returned error can't find the container with id 786c8524100c2a29d910451de94c07df9fe0ef4d865f0cb700b6fd0a63bc53f6
Jul 27 21:12:48.753507 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:12:48.753470    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/bb51a4e3-357d-4e40-9001-e5b6586860fd-cert\") pod \"llmisvc-controller-manager-84fc959947-t8pnm\" (UID: \"bb51a4e3-357d-4e40-9001-e5b6586860fd\") " pod="kserve/llmisvc-controller-manager-84fc959947-t8pnm"
Jul 27 21:12:48.755888 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:12:48.755864    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/bb51a4e3-357d-4e40-9001-e5b6586860fd-cert\") pod \"llmisvc-controller-manager-84fc959947-t8pnm\" (UID: \"bb51a4e3-357d-4e40-9001-e5b6586860fd\") " pod="kserve/llmisvc-controller-manager-84fc959947-t8pnm"
Jul 27 21:12:48.977160 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:12:48.977124    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/llmisvc-controller-manager-84fc959947-t8pnm"
Jul 27 21:12:49.139489 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:12:49.139453    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/llmisvc-controller-manager-84fc959947-t8pnm"]
Jul 27 21:12:49.162646 ip-10-0-131-133 kubenswrapper[2559]: W0727 21:12:49.162612    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podbb51a4e3_357d_4e40_9001_e5b6586860fd.slice/crio-5628b08346debbe914f4434bbba46201ec714b1cf736a32e5e9db6c728fa2810 WatchSource:0}: Error finding container 5628b08346debbe914f4434bbba46201ec714b1cf736a32e5e9db6c728fa2810: Status 404 returned error can't find the container with id 5628b08346debbe914f4434bbba46201ec714b1cf736a32e5e9db6c728fa2810
Jul 27 21:12:49.295398 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:12:49.295306    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/llmisvc-controller-manager-84fc959947-t8pnm" event={"ID":"bb51a4e3-357d-4e40-9001-e5b6586860fd","Type":"ContainerStarted","Data":"5628b08346debbe914f4434bbba46201ec714b1cf736a32e5e9db6c728fa2810"}
Jul 27 21:12:49.296872 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:12:49.296842    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-64568bcd49-p22l9" event={"ID":"349b045e-11fe-4879-ba47-a0f576ba6e25","Type":"ContainerStarted","Data":"786c8524100c2a29d910451de94c07df9fe0ef4d865f0cb700b6fd0a63bc53f6"}
Jul 27 21:12:52.316675 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:12:52.316630    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-64568bcd49-p22l9" event={"ID":"349b045e-11fe-4879-ba47-a0f576ba6e25","Type":"ContainerStarted","Data":"aa97dff46af8a351860a0fc431f75e9f69ce4864f08c918013457bb9868077a8"}
Jul 27 21:12:52.317066 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:12:52.316783    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/seaweedfs-64568bcd49-p22l9"
Jul 27 21:12:52.332848 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:12:52.332800    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/seaweedfs-64568bcd49-p22l9" podStartSLOduration=1.307781321 podStartE2EDuration="4.332785706s" totalImagesPullingTime="3.025004385s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 21:12:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 21:12:52.331814633 +0000 UTC m=+500.910998645" watchObservedRunningTime="2026-07-27 21:12:52.332785706 +0000 UTC m=+500.911969764"
Jul 27 21:12:53.321941 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:12:53.321905    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/llmisvc-controller-manager-84fc959947-t8pnm" event={"ID":"bb51a4e3-357d-4e40-9001-e5b6586860fd","Type":"ContainerStarted","Data":"4991c92d21f52369e0a5d83f86c63642a25954d31608b1825c59e2a9635d1a30"}
Jul 27 21:12:53.338886 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:12:53.338835    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/llmisvc-controller-manager-84fc959947-t8pnm" podStartSLOduration=2.108273504 podStartE2EDuration="5.338819097s" totalImagesPullingTime="3.230545593s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 21:12:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 21:12:53.337691616 +0000 UTC m=+501.916875624" watchObservedRunningTime="2026-07-27 21:12:53.338819097 +0000 UTC m=+501.918003102"
Jul 27 21:12:54.326116 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:12:54.326088    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/llmisvc-controller-manager-84fc959947-t8pnm"
Jul 27 21:12:58.324624 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:12:58.324593    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/seaweedfs-64568bcd49-p22l9"
Jul 27 21:13:02.708115 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:13:02.708079    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-9c975f545-d5zl9"]
Jul 27 21:13:02.711438 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:13:02.711411    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-9c975f545-d5zl9"
Jul 27 21:13:02.726036 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:13:02.726004    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-9c975f545-d5zl9"]
Jul 27 21:13:02.885811 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:13:02.885774    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/7fa641ac-3826-4461-8804-b57acc5863d9-console-config\") pod \"console-9c975f545-d5zl9\" (UID: \"7fa641ac-3826-4461-8804-b57acc5863d9\") " pod="openshift-console/console-9c975f545-d5zl9"
Jul 27 21:13:02.885811 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:13:02.885812    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/7fa641ac-3826-4461-8804-b57acc5863d9-console-serving-cert\") pod \"console-9c975f545-d5zl9\" (UID: \"7fa641ac-3826-4461-8804-b57acc5863d9\") " pod="openshift-console/console-9c975f545-d5zl9"
Jul 27 21:13:02.885994 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:13:02.885844    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/7fa641ac-3826-4461-8804-b57acc5863d9-oauth-serving-cert\") pod \"console-9c975f545-d5zl9\" (UID: \"7fa641ac-3826-4461-8804-b57acc5863d9\") " pod="openshift-console/console-9c975f545-d5zl9"
Jul 27 21:13:02.885994 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:13:02.885893    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q88l4\" (UniqueName: \"kubernetes.io/projected/7fa641ac-3826-4461-8804-b57acc5863d9-kube-api-access-q88l4\") pod \"console-9c975f545-d5zl9\" (UID: \"7fa641ac-3826-4461-8804-b57acc5863d9\") " pod="openshift-console/console-9c975f545-d5zl9"
Jul 27 21:13:02.885994 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:13:02.885962    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/7fa641ac-3826-4461-8804-b57acc5863d9-service-ca\") pod \"console-9c975f545-d5zl9\" (UID: \"7fa641ac-3826-4461-8804-b57acc5863d9\") " pod="openshift-console/console-9c975f545-d5zl9"
Jul 27 21:13:02.886115 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:13:02.886004    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/7fa641ac-3826-4461-8804-b57acc5863d9-console-oauth-config\") pod \"console-9c975f545-d5zl9\" (UID: \"7fa641ac-3826-4461-8804-b57acc5863d9\") " pod="openshift-console/console-9c975f545-d5zl9"
Jul 27 21:13:02.886115 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:13:02.886021    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7fa641ac-3826-4461-8804-b57acc5863d9-trusted-ca-bundle\") pod \"console-9c975f545-d5zl9\" (UID: \"7fa641ac-3826-4461-8804-b57acc5863d9\") " pod="openshift-console/console-9c975f545-d5zl9"
Jul 27 21:13:02.987376 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:13:02.987296    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/7fa641ac-3826-4461-8804-b57acc5863d9-console-config\") pod \"console-9c975f545-d5zl9\" (UID: \"7fa641ac-3826-4461-8804-b57acc5863d9\") " pod="openshift-console/console-9c975f545-d5zl9"
Jul 27 21:13:02.987376 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:13:02.987333    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/7fa641ac-3826-4461-8804-b57acc5863d9-console-serving-cert\") pod \"console-9c975f545-d5zl9\" (UID: \"7fa641ac-3826-4461-8804-b57acc5863d9\") " pod="openshift-console/console-9c975f545-d5zl9"
Jul 27 21:13:02.987376 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:13:02.987363    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/7fa641ac-3826-4461-8804-b57acc5863d9-oauth-serving-cert\") pod \"console-9c975f545-d5zl9\" (UID: \"7fa641ac-3826-4461-8804-b57acc5863d9\") " pod="openshift-console/console-9c975f545-d5zl9"
Jul 27 21:13:02.987646 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:13:02.987395    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q88l4\" (UniqueName: \"kubernetes.io/projected/7fa641ac-3826-4461-8804-b57acc5863d9-kube-api-access-q88l4\") pod \"console-9c975f545-d5zl9\" (UID: \"7fa641ac-3826-4461-8804-b57acc5863d9\") " pod="openshift-console/console-9c975f545-d5zl9"
Jul 27 21:13:02.987646 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:13:02.987438    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/7fa641ac-3826-4461-8804-b57acc5863d9-service-ca\") pod \"console-9c975f545-d5zl9\" (UID: \"7fa641ac-3826-4461-8804-b57acc5863d9\") " pod="openshift-console/console-9c975f545-d5zl9"
Jul 27 21:13:02.987646 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:13:02.987467    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/7fa641ac-3826-4461-8804-b57acc5863d9-console-oauth-config\") pod \"console-9c975f545-d5zl9\" (UID: \"7fa641ac-3826-4461-8804-b57acc5863d9\") " pod="openshift-console/console-9c975f545-d5zl9"
Jul 27 21:13:02.987646 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:13:02.987491    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7fa641ac-3826-4461-8804-b57acc5863d9-trusted-ca-bundle\") pod \"console-9c975f545-d5zl9\" (UID: \"7fa641ac-3826-4461-8804-b57acc5863d9\") " pod="openshift-console/console-9c975f545-d5zl9"
Jul 27 21:13:02.988134 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:13:02.988105    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/7fa641ac-3826-4461-8804-b57acc5863d9-oauth-serving-cert\") pod \"console-9c975f545-d5zl9\" (UID: \"7fa641ac-3826-4461-8804-b57acc5863d9\") " pod="openshift-console/console-9c975f545-d5zl9"
Jul 27 21:13:02.988267 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:13:02.988130    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/7fa641ac-3826-4461-8804-b57acc5863d9-console-config\") pod \"console-9c975f545-d5zl9\" (UID: \"7fa641ac-3826-4461-8804-b57acc5863d9\") " pod="openshift-console/console-9c975f545-d5zl9"
Jul 27 21:13:02.988267 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:13:02.988136    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/7fa641ac-3826-4461-8804-b57acc5863d9-service-ca\") pod \"console-9c975f545-d5zl9\" (UID: \"7fa641ac-3826-4461-8804-b57acc5863d9\") " pod="openshift-console/console-9c975f545-d5zl9"
Jul 27 21:13:02.988424 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:13:02.988402    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7fa641ac-3826-4461-8804-b57acc5863d9-trusted-ca-bundle\") pod \"console-9c975f545-d5zl9\" (UID: \"7fa641ac-3826-4461-8804-b57acc5863d9\") " pod="openshift-console/console-9c975f545-d5zl9"
Jul 27 21:13:02.989961 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:13:02.989935    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/7fa641ac-3826-4461-8804-b57acc5863d9-console-oauth-config\") pod \"console-9c975f545-d5zl9\" (UID: \"7fa641ac-3826-4461-8804-b57acc5863d9\") " pod="openshift-console/console-9c975f545-d5zl9"
Jul 27 21:13:02.990018 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:13:02.989957    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/7fa641ac-3826-4461-8804-b57acc5863d9-console-serving-cert\") pod \"console-9c975f545-d5zl9\" (UID: \"7fa641ac-3826-4461-8804-b57acc5863d9\") " pod="openshift-console/console-9c975f545-d5zl9"
Jul 27 21:13:02.995809 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:13:02.995785    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q88l4\" (UniqueName: \"kubernetes.io/projected/7fa641ac-3826-4461-8804-b57acc5863d9-kube-api-access-q88l4\") pod \"console-9c975f545-d5zl9\" (UID: \"7fa641ac-3826-4461-8804-b57acc5863d9\") " pod="openshift-console/console-9c975f545-d5zl9"
Jul 27 21:13:03.021909 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:13:03.021866    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-9c975f545-d5zl9"
Jul 27 21:13:03.164380 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:13:03.164352    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-9c975f545-d5zl9"]
Jul 27 21:13:03.165836 ip-10-0-131-133 kubenswrapper[2559]: W0727 21:13:03.165811    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7fa641ac_3826_4461_8804_b57acc5863d9.slice/crio-aa22080eadf43b19afa6e7b01e41157d69c5d25f64e9976dfcb329d8d67d2d5f WatchSource:0}: Error finding container aa22080eadf43b19afa6e7b01e41157d69c5d25f64e9976dfcb329d8d67d2d5f: Status 404 returned error can't find the container with id aa22080eadf43b19afa6e7b01e41157d69c5d25f64e9976dfcb329d8d67d2d5f
Jul 27 21:13:03.377389 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:13:03.377355    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-9c975f545-d5zl9" event={"ID":"7fa641ac-3826-4461-8804-b57acc5863d9","Type":"ContainerStarted","Data":"8719de7f7164646901d2804fcabbf4b19ce943b24b3614b1a99bcdbf650f6fb5"}
Jul 27 21:13:03.377591 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:13:03.377403    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-9c975f545-d5zl9" event={"ID":"7fa641ac-3826-4461-8804-b57acc5863d9","Type":"ContainerStarted","Data":"aa22080eadf43b19afa6e7b01e41157d69c5d25f64e9976dfcb329d8d67d2d5f"}
Jul 27 21:13:03.394800 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:13:03.394744    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-9c975f545-d5zl9" podStartSLOduration=1.394727986 podStartE2EDuration="1.394727986s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 21:13:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 21:13:03.393548822 +0000 UTC m=+511.972732830" watchObservedRunningTime="2026-07-27 21:13:03.394727986 +0000 UTC m=+511.973911991"
Jul 27 21:13:13.022299 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:13:13.022259    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-9c975f545-d5zl9"
Jul 27 21:13:13.022692 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:13:13.022320    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-9c975f545-d5zl9"
Jul 27 21:13:13.027315 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:13:13.027290    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-9c975f545-d5zl9"
Jul 27 21:13:13.428253 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:13:13.427528    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-9c975f545-d5zl9"
Jul 27 21:13:13.473063 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:13:13.473028    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-745654d88d-xcrr4"]
Jul 27 21:13:25.338966 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:13:25.338883    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/llmisvc-controller-manager-84fc959947-t8pnm"
Jul 27 21:13:38.493793 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:13:38.493749    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-745654d88d-xcrr4" podUID="508ed916-ee7b-44c9-a831-5bddfafdc8b1" containerName="console" containerID="cri-o://eba074652f86161a9183ba579dfba86e41adbff2acf8ecbf04082b7d94277ff7" gracePeriod=15
Jul 27 21:13:38.745495 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:13:38.745429    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-745654d88d-xcrr4"
Jul 27 21:13:38.912957 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:13:38.912924    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/508ed916-ee7b-44c9-a831-5bddfafdc8b1-console-serving-cert\") pod \"508ed916-ee7b-44c9-a831-5bddfafdc8b1\" (UID: \"508ed916-ee7b-44c9-a831-5bddfafdc8b1\") "
Jul 27 21:13:38.912957 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:13:38.912965    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/508ed916-ee7b-44c9-a831-5bddfafdc8b1-console-oauth-config\") pod \"508ed916-ee7b-44c9-a831-5bddfafdc8b1\" (UID: \"508ed916-ee7b-44c9-a831-5bddfafdc8b1\") "
Jul 27 21:13:38.913201 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:13:38.912990    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/508ed916-ee7b-44c9-a831-5bddfafdc8b1-oauth-serving-cert\") pod \"508ed916-ee7b-44c9-a831-5bddfafdc8b1\" (UID: \"508ed916-ee7b-44c9-a831-5bddfafdc8b1\") "
Jul 27 21:13:38.913201 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:13:38.913070    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/508ed916-ee7b-44c9-a831-5bddfafdc8b1-trusted-ca-bundle\") pod \"508ed916-ee7b-44c9-a831-5bddfafdc8b1\" (UID: \"508ed916-ee7b-44c9-a831-5bddfafdc8b1\") "
Jul 27 21:13:38.913201 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:13:38.913109    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/508ed916-ee7b-44c9-a831-5bddfafdc8b1-service-ca\") pod \"508ed916-ee7b-44c9-a831-5bddfafdc8b1\" (UID: \"508ed916-ee7b-44c9-a831-5bddfafdc8b1\") "
Jul 27 21:13:38.913201 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:13:38.913167    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/508ed916-ee7b-44c9-a831-5bddfafdc8b1-console-config\") pod \"508ed916-ee7b-44c9-a831-5bddfafdc8b1\" (UID: \"508ed916-ee7b-44c9-a831-5bddfafdc8b1\") "
Jul 27 21:13:38.913201 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:13:38.913192    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dfxp6\" (UniqueName: \"kubernetes.io/projected/508ed916-ee7b-44c9-a831-5bddfafdc8b1-kube-api-access-dfxp6\") pod \"508ed916-ee7b-44c9-a831-5bddfafdc8b1\" (UID: \"508ed916-ee7b-44c9-a831-5bddfafdc8b1\") "
Jul 27 21:13:38.913503 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:13:38.913471    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/508ed916-ee7b-44c9-a831-5bddfafdc8b1-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "508ed916-ee7b-44c9-a831-5bddfafdc8b1" (UID: "508ed916-ee7b-44c9-a831-5bddfafdc8b1"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 21:13:38.913562 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:13:38.913520    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/508ed916-ee7b-44c9-a831-5bddfafdc8b1-console-config" (OuterVolumeSpecName: "console-config") pod "508ed916-ee7b-44c9-a831-5bddfafdc8b1" (UID: "508ed916-ee7b-44c9-a831-5bddfafdc8b1"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 21:13:38.913562 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:13:38.913531    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/508ed916-ee7b-44c9-a831-5bddfafdc8b1-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "508ed916-ee7b-44c9-a831-5bddfafdc8b1" (UID: "508ed916-ee7b-44c9-a831-5bddfafdc8b1"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 21:13:38.913645 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:13:38.913560    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/508ed916-ee7b-44c9-a831-5bddfafdc8b1-service-ca" (OuterVolumeSpecName: "service-ca") pod "508ed916-ee7b-44c9-a831-5bddfafdc8b1" (UID: "508ed916-ee7b-44c9-a831-5bddfafdc8b1"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 21:13:38.915158 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:13:38.915132    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/508ed916-ee7b-44c9-a831-5bddfafdc8b1-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "508ed916-ee7b-44c9-a831-5bddfafdc8b1" (UID: "508ed916-ee7b-44c9-a831-5bddfafdc8b1"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 21:13:38.915158 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:13:38.915148    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/508ed916-ee7b-44c9-a831-5bddfafdc8b1-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "508ed916-ee7b-44c9-a831-5bddfafdc8b1" (UID: "508ed916-ee7b-44c9-a831-5bddfafdc8b1"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 21:13:38.915326 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:13:38.915259    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/508ed916-ee7b-44c9-a831-5bddfafdc8b1-kube-api-access-dfxp6" (OuterVolumeSpecName: "kube-api-access-dfxp6") pod "508ed916-ee7b-44c9-a831-5bddfafdc8b1" (UID: "508ed916-ee7b-44c9-a831-5bddfafdc8b1"). InnerVolumeSpecName "kube-api-access-dfxp6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 21:13:39.014760 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:13:39.014671    2559 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/508ed916-ee7b-44c9-a831-5bddfafdc8b1-service-ca\") on node \"ip-10-0-131-133.ec2.internal\" DevicePath \"\""
Jul 27 21:13:39.014760 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:13:39.014702    2559 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/508ed916-ee7b-44c9-a831-5bddfafdc8b1-console-config\") on node \"ip-10-0-131-133.ec2.internal\" DevicePath \"\""
Jul 27 21:13:39.014760 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:13:39.014712    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dfxp6\" (UniqueName: \"kubernetes.io/projected/508ed916-ee7b-44c9-a831-5bddfafdc8b1-kube-api-access-dfxp6\") on node \"ip-10-0-131-133.ec2.internal\" DevicePath \"\""
Jul 27 21:13:39.014760 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:13:39.014722    2559 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/508ed916-ee7b-44c9-a831-5bddfafdc8b1-console-serving-cert\") on node \"ip-10-0-131-133.ec2.internal\" DevicePath \"\""
Jul 27 21:13:39.014760 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:13:39.014730    2559 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/508ed916-ee7b-44c9-a831-5bddfafdc8b1-console-oauth-config\") on node \"ip-10-0-131-133.ec2.internal\" DevicePath \"\""
Jul 27 21:13:39.014760 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:13:39.014739    2559 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/508ed916-ee7b-44c9-a831-5bddfafdc8b1-oauth-serving-cert\") on node \"ip-10-0-131-133.ec2.internal\" DevicePath \"\""
Jul 27 21:13:39.014760 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:13:39.014747    2559 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/508ed916-ee7b-44c9-a831-5bddfafdc8b1-trusted-ca-bundle\") on node \"ip-10-0-131-133.ec2.internal\" DevicePath \"\""
Jul 27 21:13:39.541354 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:13:39.541317    2559 generic.go:356] "Generic (PLEG): container finished" podID="508ed916-ee7b-44c9-a831-5bddfafdc8b1" containerID="eba074652f86161a9183ba579dfba86e41adbff2acf8ecbf04082b7d94277ff7" exitCode=0
Jul 27 21:13:39.541815 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:13:39.541377    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-745654d88d-xcrr4"
Jul 27 21:13:39.541815 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:13:39.541385    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-745654d88d-xcrr4" event={"ID":"508ed916-ee7b-44c9-a831-5bddfafdc8b1","Type":"ContainerDied","Data":"eba074652f86161a9183ba579dfba86e41adbff2acf8ecbf04082b7d94277ff7"}
Jul 27 21:13:39.541815 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:13:39.541476    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-745654d88d-xcrr4" event={"ID":"508ed916-ee7b-44c9-a831-5bddfafdc8b1","Type":"ContainerDied","Data":"d377f1f0ce47542d584f06ee7be7fba60c4d9910c38926456422a78a4692343f"}
Jul 27 21:13:39.541815 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:13:39.541510    2559 scope.go:117] "RemoveContainer" containerID="eba074652f86161a9183ba579dfba86e41adbff2acf8ecbf04082b7d94277ff7"
Jul 27 21:13:39.550351 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:13:39.550327    2559 scope.go:117] "RemoveContainer" containerID="eba074652f86161a9183ba579dfba86e41adbff2acf8ecbf04082b7d94277ff7"
Jul 27 21:13:39.550635 ip-10-0-131-133 kubenswrapper[2559]: E0727 21:13:39.550615    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eba074652f86161a9183ba579dfba86e41adbff2acf8ecbf04082b7d94277ff7\": container with ID starting with eba074652f86161a9183ba579dfba86e41adbff2acf8ecbf04082b7d94277ff7 not found: ID does not exist" containerID="eba074652f86161a9183ba579dfba86e41adbff2acf8ecbf04082b7d94277ff7"
Jul 27 21:13:39.550705 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:13:39.550644    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eba074652f86161a9183ba579dfba86e41adbff2acf8ecbf04082b7d94277ff7"} err="failed to get container status \"eba074652f86161a9183ba579dfba86e41adbff2acf8ecbf04082b7d94277ff7\": rpc error: code = NotFound desc = could not find container \"eba074652f86161a9183ba579dfba86e41adbff2acf8ecbf04082b7d94277ff7\": container with ID starting with eba074652f86161a9183ba579dfba86e41adbff2acf8ecbf04082b7d94277ff7 not found: ID does not exist"
Jul 27 21:13:39.561476 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:13:39.561443    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-745654d88d-xcrr4"]
Jul 27 21:13:39.565529 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:13:39.565500    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-745654d88d-xcrr4"]
Jul 27 21:13:39.981960 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:13:39.981929    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="508ed916-ee7b-44c9-a831-5bddfafdc8b1" path="/var/lib/kubelet/pods/508ed916-ee7b-44c9-a831-5bddfafdc8b1/volumes"
Jul 27 21:14:31.883499 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:14:31.883469    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-bjvbc_cba53c9a-64c0-4afa-8731-35521e196622/console-operator/1.log"
Jul 27 21:14:31.887021 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:14:31.886997    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-bjvbc_cba53c9a-64c0-4afa-8731-35521e196622/console-operator/1.log"
Jul 27 21:15:40.387227 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:15:40.387176    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/seaweedfs-tls-custom-6644bf5d6c-th2hc"]
Jul 27 21:15:40.387794 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:15:40.387695    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="508ed916-ee7b-44c9-a831-5bddfafdc8b1" containerName="console"
Jul 27 21:15:40.387794 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:15:40.387713    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="508ed916-ee7b-44c9-a831-5bddfafdc8b1" containerName="console"
Jul 27 21:15:40.387907 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:15:40.387820    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="508ed916-ee7b-44c9-a831-5bddfafdc8b1" containerName="console"
Jul 27 21:15:40.389893 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:15:40.389870    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-tls-custom-6644bf5d6c-th2hc"
Jul 27 21:15:40.392274 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:15:40.392256    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"seaweedfs-tls-custom-artifact\""
Jul 27 21:15:40.397182 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:15:40.396831    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-tls-custom-6644bf5d6c-th2hc"]
Jul 27 21:15:40.551326 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:15:40.551286    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/2627e29d-c0db-4c2b-900c-944ad752eb21-data\") pod \"seaweedfs-tls-custom-6644bf5d6c-th2hc\" (UID: \"2627e29d-c0db-4c2b-900c-944ad752eb21\") " pod="kserve/seaweedfs-tls-custom-6644bf5d6c-th2hc"
Jul 27 21:15:40.551566 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:15:40.551354    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6m5n5\" (UniqueName: \"kubernetes.io/projected/2627e29d-c0db-4c2b-900c-944ad752eb21-kube-api-access-6m5n5\") pod \"seaweedfs-tls-custom-6644bf5d6c-th2hc\" (UID: \"2627e29d-c0db-4c2b-900c-944ad752eb21\") " pod="kserve/seaweedfs-tls-custom-6644bf5d6c-th2hc"
Jul 27 21:15:40.652556 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:15:40.652465    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/2627e29d-c0db-4c2b-900c-944ad752eb21-data\") pod \"seaweedfs-tls-custom-6644bf5d6c-th2hc\" (UID: \"2627e29d-c0db-4c2b-900c-944ad752eb21\") " pod="kserve/seaweedfs-tls-custom-6644bf5d6c-th2hc"
Jul 27 21:15:40.652556 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:15:40.652538    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6m5n5\" (UniqueName: \"kubernetes.io/projected/2627e29d-c0db-4c2b-900c-944ad752eb21-kube-api-access-6m5n5\") pod \"seaweedfs-tls-custom-6644bf5d6c-th2hc\" (UID: \"2627e29d-c0db-4c2b-900c-944ad752eb21\") " pod="kserve/seaweedfs-tls-custom-6644bf5d6c-th2hc"
Jul 27 21:15:40.652874 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:15:40.652853    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/2627e29d-c0db-4c2b-900c-944ad752eb21-data\") pod \"seaweedfs-tls-custom-6644bf5d6c-th2hc\" (UID: \"2627e29d-c0db-4c2b-900c-944ad752eb21\") " pod="kserve/seaweedfs-tls-custom-6644bf5d6c-th2hc"
Jul 27 21:15:40.660887 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:15:40.660852    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6m5n5\" (UniqueName: \"kubernetes.io/projected/2627e29d-c0db-4c2b-900c-944ad752eb21-kube-api-access-6m5n5\") pod \"seaweedfs-tls-custom-6644bf5d6c-th2hc\" (UID: \"2627e29d-c0db-4c2b-900c-944ad752eb21\") " pod="kserve/seaweedfs-tls-custom-6644bf5d6c-th2hc"
Jul 27 21:15:40.700490 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:15:40.700450    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-tls-custom-6644bf5d6c-th2hc"
Jul 27 21:15:40.833683 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:15:40.833656    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-tls-custom-6644bf5d6c-th2hc"]
Jul 27 21:15:40.835850 ip-10-0-131-133 kubenswrapper[2559]: W0727 21:15:40.835821    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2627e29d_c0db_4c2b_900c_944ad752eb21.slice/crio-0ab86bde56137409982ad851da75430477c70a237218de85ec4158b2fc51bdc1 WatchSource:0}: Error finding container 0ab86bde56137409982ad851da75430477c70a237218de85ec4158b2fc51bdc1: Status 404 returned error can't find the container with id 0ab86bde56137409982ad851da75430477c70a237218de85ec4158b2fc51bdc1
Jul 27 21:15:41.093274 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:15:41.093116    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-tls-custom-6644bf5d6c-th2hc" event={"ID":"2627e29d-c0db-4c2b-900c-944ad752eb21","Type":"ContainerStarted","Data":"0ab86bde56137409982ad851da75430477c70a237218de85ec4158b2fc51bdc1"}
Jul 27 21:15:42.098209 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:15:42.098168    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-tls-custom-6644bf5d6c-th2hc" event={"ID":"2627e29d-c0db-4c2b-900c-944ad752eb21","Type":"ContainerStarted","Data":"6d577f5518d8b78455dafe280f6c4b153c1740872eda15b45645ec10cc489198"}
Jul 27 21:15:42.113798 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:15:42.113748    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/seaweedfs-tls-custom-6644bf5d6c-th2hc" podStartSLOduration=1.777714383 podStartE2EDuration="2.113732292s" totalImagesPullingTime="336.017909ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 21:15:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 21:15:42.11242369 +0000 UTC m=+670.691607693" watchObservedRunningTime="2026-07-27 21:15:42.113732292 +0000 UTC m=+670.692916316"
Jul 27 21:15:42.864621 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:15:42.864583    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve/seaweedfs-tls-custom-6644bf5d6c-th2hc"]
Jul 27 21:15:44.106377 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:15:44.106340    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve/seaweedfs-tls-custom-6644bf5d6c-th2hc" podUID="2627e29d-c0db-4c2b-900c-944ad752eb21" containerName="seaweedfs-tls-custom" containerID="cri-o://6d577f5518d8b78455dafe280f6c4b153c1740872eda15b45645ec10cc489198" gracePeriod=30
Jul 27 21:15:45.352131 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:15:45.352102    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-tls-custom-6644bf5d6c-th2hc"
Jul 27 21:15:45.496046 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:15:45.496005    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/2627e29d-c0db-4c2b-900c-944ad752eb21-data\") pod \"2627e29d-c0db-4c2b-900c-944ad752eb21\" (UID: \"2627e29d-c0db-4c2b-900c-944ad752eb21\") "
Jul 27 21:15:45.496292 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:15:45.496081    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6m5n5\" (UniqueName: \"kubernetes.io/projected/2627e29d-c0db-4c2b-900c-944ad752eb21-kube-api-access-6m5n5\") pod \"2627e29d-c0db-4c2b-900c-944ad752eb21\" (UID: \"2627e29d-c0db-4c2b-900c-944ad752eb21\") "
Jul 27 21:15:45.496979 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:15:45.496951    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2627e29d-c0db-4c2b-900c-944ad752eb21-data" (OuterVolumeSpecName: "data") pod "2627e29d-c0db-4c2b-900c-944ad752eb21" (UID: "2627e29d-c0db-4c2b-900c-944ad752eb21"). InnerVolumeSpecName "data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 21:15:45.498335 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:15:45.498307    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2627e29d-c0db-4c2b-900c-944ad752eb21-kube-api-access-6m5n5" (OuterVolumeSpecName: "kube-api-access-6m5n5") pod "2627e29d-c0db-4c2b-900c-944ad752eb21" (UID: "2627e29d-c0db-4c2b-900c-944ad752eb21"). InnerVolumeSpecName "kube-api-access-6m5n5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 21:15:45.596973 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:15:45.596938    2559 reconciler_common.go:299] "Volume detached for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/2627e29d-c0db-4c2b-900c-944ad752eb21-data\") on node \"ip-10-0-131-133.ec2.internal\" DevicePath \"\""
Jul 27 21:15:45.596973 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:15:45.596971    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6m5n5\" (UniqueName: \"kubernetes.io/projected/2627e29d-c0db-4c2b-900c-944ad752eb21-kube-api-access-6m5n5\") on node \"ip-10-0-131-133.ec2.internal\" DevicePath \"\""
Jul 27 21:15:46.116366 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:15:46.116336    2559 generic.go:356] "Generic (PLEG): container finished" podID="2627e29d-c0db-4c2b-900c-944ad752eb21" containerID="6d577f5518d8b78455dafe280f6c4b153c1740872eda15b45645ec10cc489198" exitCode=0
Jul 27 21:15:46.116555 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:15:46.116378    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-tls-custom-6644bf5d6c-th2hc"
Jul 27 21:15:46.116555 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:15:46.116420    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-tls-custom-6644bf5d6c-th2hc" event={"ID":"2627e29d-c0db-4c2b-900c-944ad752eb21","Type":"ContainerDied","Data":"6d577f5518d8b78455dafe280f6c4b153c1740872eda15b45645ec10cc489198"}
Jul 27 21:15:46.116555 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:15:46.116465    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-tls-custom-6644bf5d6c-th2hc" event={"ID":"2627e29d-c0db-4c2b-900c-944ad752eb21","Type":"ContainerDied","Data":"0ab86bde56137409982ad851da75430477c70a237218de85ec4158b2fc51bdc1"}
Jul 27 21:15:46.116555 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:15:46.116487    2559 scope.go:117] "RemoveContainer" containerID="6d577f5518d8b78455dafe280f6c4b153c1740872eda15b45645ec10cc489198"
Jul 27 21:15:46.128625 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:15:46.128598    2559 scope.go:117] "RemoveContainer" containerID="6d577f5518d8b78455dafe280f6c4b153c1740872eda15b45645ec10cc489198"
Jul 27 21:15:46.128998 ip-10-0-131-133 kubenswrapper[2559]: E0727 21:15:46.128979    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6d577f5518d8b78455dafe280f6c4b153c1740872eda15b45645ec10cc489198\": container with ID starting with 6d577f5518d8b78455dafe280f6c4b153c1740872eda15b45645ec10cc489198 not found: ID does not exist" containerID="6d577f5518d8b78455dafe280f6c4b153c1740872eda15b45645ec10cc489198"
Jul 27 21:15:46.129050 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:15:46.129006    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6d577f5518d8b78455dafe280f6c4b153c1740872eda15b45645ec10cc489198"} err="failed to get container status \"6d577f5518d8b78455dafe280f6c4b153c1740872eda15b45645ec10cc489198\": rpc error: code = NotFound desc = could not find container \"6d577f5518d8b78455dafe280f6c4b153c1740872eda15b45645ec10cc489198\": container with ID starting with 6d577f5518d8b78455dafe280f6c4b153c1740872eda15b45645ec10cc489198 not found: ID does not exist"
Jul 27 21:15:46.132567 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:15:46.132539    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve/seaweedfs-tls-custom-6644bf5d6c-th2hc"]
Jul 27 21:15:46.136140 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:15:46.136120    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve/seaweedfs-tls-custom-6644bf5d6c-th2hc"]
Jul 27 21:15:46.161099 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:15:46.161063    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/seaweedfs-tls-custom-8b44d767d-qqzzg"]
Jul 27 21:15:46.161473 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:15:46.161456    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2627e29d-c0db-4c2b-900c-944ad752eb21" containerName="seaweedfs-tls-custom"
Jul 27 21:15:46.161531 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:15:46.161475    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="2627e29d-c0db-4c2b-900c-944ad752eb21" containerName="seaweedfs-tls-custom"
Jul 27 21:15:46.161573 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:15:46.161556    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="2627e29d-c0db-4c2b-900c-944ad752eb21" containerName="seaweedfs-tls-custom"
Jul 27 21:15:46.165767 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:15:46.165747    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-tls-custom-8b44d767d-qqzzg"
Jul 27 21:15:46.167820 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:15:46.167801    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"seaweedfs-tls-custom-artifact\""
Jul 27 21:15:46.167900 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:15:46.167853    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"seaweedfs-tls-custom\""
Jul 27 21:15:46.171035 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:15:46.171009    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-tls-custom-8b44d767d-qqzzg"]
Jul 27 21:15:46.303297 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:15:46.303172    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/2297f5ec-fed9-4ebf-86c7-9470b5c632ae-data\") pod \"seaweedfs-tls-custom-8b44d767d-qqzzg\" (UID: \"2297f5ec-fed9-4ebf-86c7-9470b5c632ae\") " pod="kserve/seaweedfs-tls-custom-8b44d767d-qqzzg"
Jul 27 21:15:46.303473 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:15:46.303332    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"seaweedfs-tls-custom\" (UniqueName: \"kubernetes.io/projected/2297f5ec-fed9-4ebf-86c7-9470b5c632ae-seaweedfs-tls-custom\") pod \"seaweedfs-tls-custom-8b44d767d-qqzzg\" (UID: \"2297f5ec-fed9-4ebf-86c7-9470b5c632ae\") " pod="kserve/seaweedfs-tls-custom-8b44d767d-qqzzg"
Jul 27 21:15:46.303473 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:15:46.303379    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vqj85\" (UniqueName: \"kubernetes.io/projected/2297f5ec-fed9-4ebf-86c7-9470b5c632ae-kube-api-access-vqj85\") pod \"seaweedfs-tls-custom-8b44d767d-qqzzg\" (UID: \"2297f5ec-fed9-4ebf-86c7-9470b5c632ae\") " pod="kserve/seaweedfs-tls-custom-8b44d767d-qqzzg"
Jul 27 21:15:46.404279 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:15:46.404245    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/2297f5ec-fed9-4ebf-86c7-9470b5c632ae-data\") pod \"seaweedfs-tls-custom-8b44d767d-qqzzg\" (UID: \"2297f5ec-fed9-4ebf-86c7-9470b5c632ae\") " pod="kserve/seaweedfs-tls-custom-8b44d767d-qqzzg"
Jul 27 21:15:46.404719 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:15:46.404339    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"seaweedfs-tls-custom\" (UniqueName: \"kubernetes.io/projected/2297f5ec-fed9-4ebf-86c7-9470b5c632ae-seaweedfs-tls-custom\") pod \"seaweedfs-tls-custom-8b44d767d-qqzzg\" (UID: \"2297f5ec-fed9-4ebf-86c7-9470b5c632ae\") " pod="kserve/seaweedfs-tls-custom-8b44d767d-qqzzg"
Jul 27 21:15:46.404719 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:15:46.404358    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vqj85\" (UniqueName: \"kubernetes.io/projected/2297f5ec-fed9-4ebf-86c7-9470b5c632ae-kube-api-access-vqj85\") pod \"seaweedfs-tls-custom-8b44d767d-qqzzg\" (UID: \"2297f5ec-fed9-4ebf-86c7-9470b5c632ae\") " pod="kserve/seaweedfs-tls-custom-8b44d767d-qqzzg"
Jul 27 21:15:46.404719 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:15:46.404692    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/2297f5ec-fed9-4ebf-86c7-9470b5c632ae-data\") pod \"seaweedfs-tls-custom-8b44d767d-qqzzg\" (UID: \"2297f5ec-fed9-4ebf-86c7-9470b5c632ae\") " pod="kserve/seaweedfs-tls-custom-8b44d767d-qqzzg"
Jul 27 21:15:46.406832 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:15:46.406815    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"seaweedfs-tls-custom\" (UniqueName: \"kubernetes.io/projected/2297f5ec-fed9-4ebf-86c7-9470b5c632ae-seaweedfs-tls-custom\") pod \"seaweedfs-tls-custom-8b44d767d-qqzzg\" (UID: \"2297f5ec-fed9-4ebf-86c7-9470b5c632ae\") " pod="kserve/seaweedfs-tls-custom-8b44d767d-qqzzg"
Jul 27 21:15:46.411844 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:15:46.411826    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vqj85\" (UniqueName: \"kubernetes.io/projected/2297f5ec-fed9-4ebf-86c7-9470b5c632ae-kube-api-access-vqj85\") pod \"seaweedfs-tls-custom-8b44d767d-qqzzg\" (UID: \"2297f5ec-fed9-4ebf-86c7-9470b5c632ae\") " pod="kserve/seaweedfs-tls-custom-8b44d767d-qqzzg"
Jul 27 21:15:46.476387 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:15:46.476357    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-tls-custom-8b44d767d-qqzzg"
Jul 27 21:15:46.604817 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:15:46.604793    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-tls-custom-8b44d767d-qqzzg"]
Jul 27 21:15:46.607647 ip-10-0-131-133 kubenswrapper[2559]: W0727 21:15:46.607618    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2297f5ec_fed9_4ebf_86c7_9470b5c632ae.slice/crio-151dd7c6eafe81bd98d87d263aeb51052e1e0f4e40efda1e09de119154e30fd3 WatchSource:0}: Error finding container 151dd7c6eafe81bd98d87d263aeb51052e1e0f4e40efda1e09de119154e30fd3: Status 404 returned error can't find the container with id 151dd7c6eafe81bd98d87d263aeb51052e1e0f4e40efda1e09de119154e30fd3
Jul 27 21:15:47.123388 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:15:47.123303    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-tls-custom-8b44d767d-qqzzg" event={"ID":"2297f5ec-fed9-4ebf-86c7-9470b5c632ae","Type":"ContainerStarted","Data":"d050b63fff585e27761d60dfbacebac4b989cde087d331d2479ab8f8134e783a"}
Jul 27 21:15:47.123388 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:15:47.123348    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-tls-custom-8b44d767d-qqzzg" event={"ID":"2297f5ec-fed9-4ebf-86c7-9470b5c632ae","Type":"ContainerStarted","Data":"151dd7c6eafe81bd98d87d263aeb51052e1e0f4e40efda1e09de119154e30fd3"}
Jul 27 21:15:47.143128 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:15:47.143073    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/seaweedfs-tls-custom-8b44d767d-qqzzg" podStartSLOduration=0.88965419 podStartE2EDuration="1.143055441s" totalImagesPullingTime="253.401251ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 21:15:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 21:15:47.141407975 +0000 UTC m=+675.720592005" watchObservedRunningTime="2026-07-27 21:15:47.143055441 +0000 UTC m=+675.722239448"
Jul 27 21:15:47.982293 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:15:47.982261    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2627e29d-c0db-4c2b-900c-944ad752eb21" path="/var/lib/kubelet/pods/2627e29d-c0db-4c2b-900c-944ad752eb21/volumes"
Jul 27 21:15:59.322578 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:15:59.322537    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/seaweedfs-tls-serving-67b7477c6d-6fgb9"]
Jul 27 21:15:59.325975 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:15:59.325950    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-tls-serving-67b7477c6d-6fgb9"
Jul 27 21:15:59.328862 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:15:59.328837    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"seaweedfs-tls-serving\""
Jul 27 21:15:59.328862 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:15:59.328854    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"seaweedfs-tls-serving-artifact\""
Jul 27 21:15:59.335025 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:15:59.335001    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-tls-serving-67b7477c6d-6fgb9"]
Jul 27 21:15:59.422623 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:15:59.422581    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"seaweedfs-tls-serving\" (UniqueName: \"kubernetes.io/projected/76fc64db-f5ad-464e-b9d7-ec89d17012d5-seaweedfs-tls-serving\") pod \"seaweedfs-tls-serving-67b7477c6d-6fgb9\" (UID: \"76fc64db-f5ad-464e-b9d7-ec89d17012d5\") " pod="kserve/seaweedfs-tls-serving-67b7477c6d-6fgb9"
Jul 27 21:15:59.422793 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:15:59.422638    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/76fc64db-f5ad-464e-b9d7-ec89d17012d5-data\") pod \"seaweedfs-tls-serving-67b7477c6d-6fgb9\" (UID: \"76fc64db-f5ad-464e-b9d7-ec89d17012d5\") " pod="kserve/seaweedfs-tls-serving-67b7477c6d-6fgb9"
Jul 27 21:15:59.422793 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:15:59.422667    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-btlmr\" (UniqueName: \"kubernetes.io/projected/76fc64db-f5ad-464e-b9d7-ec89d17012d5-kube-api-access-btlmr\") pod \"seaweedfs-tls-serving-67b7477c6d-6fgb9\" (UID: \"76fc64db-f5ad-464e-b9d7-ec89d17012d5\") " pod="kserve/seaweedfs-tls-serving-67b7477c6d-6fgb9"
Jul 27 21:15:59.523988 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:15:59.523937    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"seaweedfs-tls-serving\" (UniqueName: \"kubernetes.io/projected/76fc64db-f5ad-464e-b9d7-ec89d17012d5-seaweedfs-tls-serving\") pod \"seaweedfs-tls-serving-67b7477c6d-6fgb9\" (UID: \"76fc64db-f5ad-464e-b9d7-ec89d17012d5\") " pod="kserve/seaweedfs-tls-serving-67b7477c6d-6fgb9"
Jul 27 21:15:59.524190 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:15:59.524015    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/76fc64db-f5ad-464e-b9d7-ec89d17012d5-data\") pod \"seaweedfs-tls-serving-67b7477c6d-6fgb9\" (UID: \"76fc64db-f5ad-464e-b9d7-ec89d17012d5\") " pod="kserve/seaweedfs-tls-serving-67b7477c6d-6fgb9"
Jul 27 21:15:59.524190 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:15:59.524053    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-btlmr\" (UniqueName: \"kubernetes.io/projected/76fc64db-f5ad-464e-b9d7-ec89d17012d5-kube-api-access-btlmr\") pod \"seaweedfs-tls-serving-67b7477c6d-6fgb9\" (UID: \"76fc64db-f5ad-464e-b9d7-ec89d17012d5\") " pod="kserve/seaweedfs-tls-serving-67b7477c6d-6fgb9"
Jul 27 21:15:59.524190 ip-10-0-131-133 kubenswrapper[2559]: E0727 21:15:59.524114    2559 projected.go:266] Couldn't get secret kserve/seaweedfs-tls-serving: secret "seaweedfs-tls-serving" not found
Jul 27 21:15:59.524190 ip-10-0-131-133 kubenswrapper[2559]: E0727 21:15:59.524135    2559 projected.go:196] Error preparing data for projected volume seaweedfs-tls-serving for pod kserve/seaweedfs-tls-serving-67b7477c6d-6fgb9: secret "seaweedfs-tls-serving" not found
Jul 27 21:15:59.524190 ip-10-0-131-133 kubenswrapper[2559]: E0727 21:15:59.524193    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/76fc64db-f5ad-464e-b9d7-ec89d17012d5-seaweedfs-tls-serving podName:76fc64db-f5ad-464e-b9d7-ec89d17012d5 nodeName:}" failed. No retries permitted until 2026-07-27 21:16:00.024177488 +0000 UTC m=+688.603361478 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "seaweedfs-tls-serving" (UniqueName: "kubernetes.io/projected/76fc64db-f5ad-464e-b9d7-ec89d17012d5-seaweedfs-tls-serving") pod "seaweedfs-tls-serving-67b7477c6d-6fgb9" (UID: "76fc64db-f5ad-464e-b9d7-ec89d17012d5") : secret "seaweedfs-tls-serving" not found
Jul 27 21:15:59.524502 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:15:59.524402    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/76fc64db-f5ad-464e-b9d7-ec89d17012d5-data\") pod \"seaweedfs-tls-serving-67b7477c6d-6fgb9\" (UID: \"76fc64db-f5ad-464e-b9d7-ec89d17012d5\") " pod="kserve/seaweedfs-tls-serving-67b7477c6d-6fgb9"
Jul 27 21:15:59.533006 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:15:59.532981    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-btlmr\" (UniqueName: \"kubernetes.io/projected/76fc64db-f5ad-464e-b9d7-ec89d17012d5-kube-api-access-btlmr\") pod \"seaweedfs-tls-serving-67b7477c6d-6fgb9\" (UID: \"76fc64db-f5ad-464e-b9d7-ec89d17012d5\") " pod="kserve/seaweedfs-tls-serving-67b7477c6d-6fgb9"
Jul 27 21:16:00.028770 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:16:00.028732    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"seaweedfs-tls-serving\" (UniqueName: \"kubernetes.io/projected/76fc64db-f5ad-464e-b9d7-ec89d17012d5-seaweedfs-tls-serving\") pod \"seaweedfs-tls-serving-67b7477c6d-6fgb9\" (UID: \"76fc64db-f5ad-464e-b9d7-ec89d17012d5\") " pod="kserve/seaweedfs-tls-serving-67b7477c6d-6fgb9"
Jul 27 21:16:00.031251 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:16:00.031191    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"seaweedfs-tls-serving\" (UniqueName: \"kubernetes.io/projected/76fc64db-f5ad-464e-b9d7-ec89d17012d5-seaweedfs-tls-serving\") pod \"seaweedfs-tls-serving-67b7477c6d-6fgb9\" (UID: \"76fc64db-f5ad-464e-b9d7-ec89d17012d5\") " pod="kserve/seaweedfs-tls-serving-67b7477c6d-6fgb9"
Jul 27 21:16:00.235807 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:16:00.235774    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-tls-serving-67b7477c6d-6fgb9"
Jul 27 21:16:00.368130 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:16:00.368104    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-tls-serving-67b7477c6d-6fgb9"]
Jul 27 21:16:00.369835 ip-10-0-131-133 kubenswrapper[2559]: W0727 21:16:00.369803    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod76fc64db_f5ad_464e_b9d7_ec89d17012d5.slice/crio-42a05bdb4524626e5d51843c1d34458c052846cbbdbafc4eb682fccdb6525927 WatchSource:0}: Error finding container 42a05bdb4524626e5d51843c1d34458c052846cbbdbafc4eb682fccdb6525927: Status 404 returned error can't find the container with id 42a05bdb4524626e5d51843c1d34458c052846cbbdbafc4eb682fccdb6525927
Jul 27 21:16:01.204656 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:16:01.204621    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-tls-serving-67b7477c6d-6fgb9" event={"ID":"76fc64db-f5ad-464e-b9d7-ec89d17012d5","Type":"ContainerStarted","Data":"0f62699ea5e90991a489d68d53ce89bfd475ac68cbf54cf7d60af7e6ddeade63"}
Jul 27 21:16:01.204825 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:16:01.204673    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-tls-serving-67b7477c6d-6fgb9" event={"ID":"76fc64db-f5ad-464e-b9d7-ec89d17012d5","Type":"ContainerStarted","Data":"42a05bdb4524626e5d51843c1d34458c052846cbbdbafc4eb682fccdb6525927"}
Jul 27 21:16:01.221973 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:16:01.221904    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/seaweedfs-tls-serving-67b7477c6d-6fgb9" podStartSLOduration=1.966488834 podStartE2EDuration="2.221881761s" totalImagesPullingTime="255.392927ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 21:15:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 21:16:01.221392938 +0000 UTC m=+689.800576946" watchObservedRunningTime="2026-07-27 21:16:01.221881761 +0000 UTC m=+689.801065769"
Jul 27 21:19:31.915518 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:19:31.915448    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-bjvbc_cba53c9a-64c0-4afa-8731-35521e196622/console-operator/1.log"
Jul 27 21:19:31.917462 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:19:31.917432    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-bjvbc_cba53c9a-64c0-4afa-8731-35521e196622/console-operator/1.log"
Jul 27 21:24:31.947012 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:24:31.946980    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-bjvbc_cba53c9a-64c0-4afa-8731-35521e196622/console-operator/1.log"
Jul 27 21:24:31.947587 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:24:31.947182    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-bjvbc_cba53c9a-64c0-4afa-8731-35521e196622/console-operator/1.log"
Jul 27 21:29:31.983555 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:29:31.983525    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-bjvbc_cba53c9a-64c0-4afa-8731-35521e196622/console-operator/1.log"
Jul 27 21:29:31.983994 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:29:31.983943    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-bjvbc_cba53c9a-64c0-4afa-8731-35521e196622/console-operator/1.log"
Jul 27 21:34:32.010387 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:34:32.010360    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-bjvbc_cba53c9a-64c0-4afa-8731-35521e196622/console-operator/1.log"
Jul 27 21:34:32.011544 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:34:32.011526    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-bjvbc_cba53c9a-64c0-4afa-8731-35521e196622/console-operator/1.log"
Jul 27 21:35:48.896557 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:35:48.896520    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-dmx8m"]
Jul 27 21:35:48.900112 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:35:48.900095    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-dmx8m"
Jul 27 21:35:48.902505 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:35:48.902475    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 27 21:35:48.902505 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:35:48.902490    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 27 21:35:48.902667 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:35:48.902527    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-promote-kube-rbac-proxy-sar-config\""
Jul 27 21:35:48.902667 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:35:48.902593    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-promote-v2-predictor-serving-cert\""
Jul 27 21:35:48.903242 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:35:48.903204    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 27 21:35:48.903242 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:35:48.903238    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 27 21:35:48.903359 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:35:48.903232    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-v7c66\""
Jul 27 21:35:48.908423 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:35:48.908400    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-dmx8m"]
Jul 27 21:35:49.064378 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:35:49.064338    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e6da3ed4-2e22-489d-b6f7-86e70fb5826a-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-v2-predictor-86766c4958-dmx8m\" (UID: \"e6da3ed4-2e22-489d-b6f7-86e70fb5826a\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-dmx8m"
Jul 27 21:35:49.064562 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:35:49.064399    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e6da3ed4-2e22-489d-b6f7-86e70fb5826a-cabundle-cert\") pod \"isvc-canary-promote-v2-predictor-86766c4958-dmx8m\" (UID: \"e6da3ed4-2e22-489d-b6f7-86e70fb5826a\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-dmx8m"
Jul 27 21:35:49.064562 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:35:49.064458    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e6da3ed4-2e22-489d-b6f7-86e70fb5826a-proxy-tls\") pod \"isvc-canary-promote-v2-predictor-86766c4958-dmx8m\" (UID: \"e6da3ed4-2e22-489d-b6f7-86e70fb5826a\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-dmx8m"
Jul 27 21:35:49.064562 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:35:49.064491    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e6da3ed4-2e22-489d-b6f7-86e70fb5826a-kserve-provision-location\") pod \"isvc-canary-promote-v2-predictor-86766c4958-dmx8m\" (UID: \"e6da3ed4-2e22-489d-b6f7-86e70fb5826a\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-dmx8m"
Jul 27 21:35:49.064562 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:35:49.064535    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x2gk8\" (UniqueName: \"kubernetes.io/projected/e6da3ed4-2e22-489d-b6f7-86e70fb5826a-kube-api-access-x2gk8\") pod \"isvc-canary-promote-v2-predictor-86766c4958-dmx8m\" (UID: \"e6da3ed4-2e22-489d-b6f7-86e70fb5826a\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-dmx8m"
Jul 27 21:35:49.165556 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:35:49.165531    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e6da3ed4-2e22-489d-b6f7-86e70fb5826a-cabundle-cert\") pod \"isvc-canary-promote-v2-predictor-86766c4958-dmx8m\" (UID: \"e6da3ed4-2e22-489d-b6f7-86e70fb5826a\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-dmx8m"
Jul 27 21:35:49.165642 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:35:49.165571    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e6da3ed4-2e22-489d-b6f7-86e70fb5826a-proxy-tls\") pod \"isvc-canary-promote-v2-predictor-86766c4958-dmx8m\" (UID: \"e6da3ed4-2e22-489d-b6f7-86e70fb5826a\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-dmx8m"
Jul 27 21:35:49.165642 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:35:49.165590    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e6da3ed4-2e22-489d-b6f7-86e70fb5826a-kserve-provision-location\") pod \"isvc-canary-promote-v2-predictor-86766c4958-dmx8m\" (UID: \"e6da3ed4-2e22-489d-b6f7-86e70fb5826a\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-dmx8m"
Jul 27 21:35:49.165642 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:35:49.165634    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x2gk8\" (UniqueName: \"kubernetes.io/projected/e6da3ed4-2e22-489d-b6f7-86e70fb5826a-kube-api-access-x2gk8\") pod \"isvc-canary-promote-v2-predictor-86766c4958-dmx8m\" (UID: \"e6da3ed4-2e22-489d-b6f7-86e70fb5826a\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-dmx8m"
Jul 27 21:35:49.165760 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:35:49.165703    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e6da3ed4-2e22-489d-b6f7-86e70fb5826a-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-v2-predictor-86766c4958-dmx8m\" (UID: \"e6da3ed4-2e22-489d-b6f7-86e70fb5826a\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-dmx8m"
Jul 27 21:35:49.166285 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:35:49.166259    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e6da3ed4-2e22-489d-b6f7-86e70fb5826a-cabundle-cert\") pod \"isvc-canary-promote-v2-predictor-86766c4958-dmx8m\" (UID: \"e6da3ed4-2e22-489d-b6f7-86e70fb5826a\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-dmx8m"
Jul 27 21:35:49.166396 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:35:49.166284    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e6da3ed4-2e22-489d-b6f7-86e70fb5826a-kserve-provision-location\") pod \"isvc-canary-promote-v2-predictor-86766c4958-dmx8m\" (UID: \"e6da3ed4-2e22-489d-b6f7-86e70fb5826a\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-dmx8m"
Jul 27 21:35:49.166396 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:35:49.166309    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e6da3ed4-2e22-489d-b6f7-86e70fb5826a-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-v2-predictor-86766c4958-dmx8m\" (UID: \"e6da3ed4-2e22-489d-b6f7-86e70fb5826a\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-dmx8m"
Jul 27 21:35:49.168144 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:35:49.168126    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e6da3ed4-2e22-489d-b6f7-86e70fb5826a-proxy-tls\") pod \"isvc-canary-promote-v2-predictor-86766c4958-dmx8m\" (UID: \"e6da3ed4-2e22-489d-b6f7-86e70fb5826a\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-dmx8m"
Jul 27 21:35:49.174829 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:35:49.174797    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x2gk8\" (UniqueName: \"kubernetes.io/projected/e6da3ed4-2e22-489d-b6f7-86e70fb5826a-kube-api-access-x2gk8\") pod \"isvc-canary-promote-v2-predictor-86766c4958-dmx8m\" (UID: \"e6da3ed4-2e22-489d-b6f7-86e70fb5826a\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-dmx8m"
Jul 27 21:35:49.211789 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:35:49.211760    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-dmx8m"
Jul 27 21:35:49.342185 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:35:49.342101    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-dmx8m"]
Jul 27 21:35:49.344315 ip-10-0-131-133 kubenswrapper[2559]: W0727 21:35:49.344283    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode6da3ed4_2e22_489d_b6f7_86e70fb5826a.slice/crio-bdcd03d5abf22fe4269e60bda287d263c24f4753fc0c8d40d2919d3a9c5a17cb WatchSource:0}: Error finding container bdcd03d5abf22fe4269e60bda287d263c24f4753fc0c8d40d2919d3a9c5a17cb: Status 404 returned error can't find the container with id bdcd03d5abf22fe4269e60bda287d263c24f4753fc0c8d40d2919d3a9c5a17cb
Jul 27 21:35:49.346183 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:35:49.346167    2559 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 27 21:35:49.676085 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:35:49.676053    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-dmx8m" event={"ID":"e6da3ed4-2e22-489d-b6f7-86e70fb5826a","Type":"ContainerStarted","Data":"bdcd03d5abf22fe4269e60bda287d263c24f4753fc0c8d40d2919d3a9c5a17cb"}
Jul 27 21:35:53.707638 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:35:53.707596    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-dmx8m" event={"ID":"e6da3ed4-2e22-489d-b6f7-86e70fb5826a","Type":"ContainerStarted","Data":"c89ec63143d063a18149583f494131c0bc3589358c5c30be881661c2536c572b"}
Jul 27 21:35:57.726264 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:35:57.726204    2559 generic.go:356] "Generic (PLEG): container finished" podID="e6da3ed4-2e22-489d-b6f7-86e70fb5826a" containerID="c89ec63143d063a18149583f494131c0bc3589358c5c30be881661c2536c572b" exitCode=0
Jul 27 21:35:57.726635 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:35:57.726285    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-dmx8m" event={"ID":"e6da3ed4-2e22-489d-b6f7-86e70fb5826a","Type":"ContainerDied","Data":"c89ec63143d063a18149583f494131c0bc3589358c5c30be881661c2536c572b"}
Jul 27 21:36:11.816952 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:36:11.816920    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-dmx8m" event={"ID":"e6da3ed4-2e22-489d-b6f7-86e70fb5826a","Type":"ContainerStarted","Data":"caee39126098df85589dc19940472c5c4e1db8d94be332ed672cfff2ed697350"}
Jul 27 21:36:13.830026 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:36:13.829984    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-dmx8m" event={"ID":"e6da3ed4-2e22-489d-b6f7-86e70fb5826a","Type":"ContainerStarted","Data":"72306ba0c2fdfee5fd619bb3164de227ffe32ae649fd224cfa12d5bf6ad30cdf"}
Jul 27 21:36:13.830462 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:36:13.830236    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-dmx8m"
Jul 27 21:36:13.830462 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:36:13.830337    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-dmx8m"
Jul 27 21:36:13.831804 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:36:13.831778    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-dmx8m" podUID="e6da3ed4-2e22-489d-b6f7-86e70fb5826a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.36:8080: connect: connection refused"
Jul 27 21:36:13.849447 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:36:13.849401    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-dmx8m" podStartSLOduration=2.631079131 podStartE2EDuration="25.849383174s" totalImagesPullingTime="19.41276949s" totalInitContainerRuntime="3.805534553s" isStatefulPod=false podCreationTimestamp="2026-07-27 21:35:48 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 21:36:13.848440005 +0000 UTC m=+1902.427624023" watchObservedRunningTime="2026-07-27 21:36:13.849383174 +0000 UTC m=+1902.428567181"
Jul 27 21:36:14.835322 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:36:14.835285    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-dmx8m" podUID="e6da3ed4-2e22-489d-b6f7-86e70fb5826a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.36:8080: connect: connection refused"
Jul 27 21:36:19.839349 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:36:19.839320    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-dmx8m"
Jul 27 21:36:19.839865 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:36:19.839808    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-dmx8m" podUID="e6da3ed4-2e22-489d-b6f7-86e70fb5826a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.36:8080: connect: connection refused"
Jul 27 21:36:29.840490 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:36:29.840443    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-dmx8m" podUID="e6da3ed4-2e22-489d-b6f7-86e70fb5826a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.36:8080: connect: connection refused"
Jul 27 21:36:39.840174 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:36:39.840132    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-dmx8m" podUID="e6da3ed4-2e22-489d-b6f7-86e70fb5826a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.36:8080: connect: connection refused"
Jul 27 21:36:49.840389 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:36:49.840345    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-dmx8m" podUID="e6da3ed4-2e22-489d-b6f7-86e70fb5826a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.36:8080: connect: connection refused"
Jul 27 21:36:59.840705 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:36:59.840657    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-dmx8m" podUID="e6da3ed4-2e22-489d-b6f7-86e70fb5826a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.36:8080: connect: connection refused"
Jul 27 21:37:09.840471 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:37:09.840423    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-dmx8m" podUID="e6da3ed4-2e22-489d-b6f7-86e70fb5826a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.36:8080: connect: connection refused"
Jul 27 21:37:19.840307 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:37:19.840268    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-dmx8m" podUID="e6da3ed4-2e22-489d-b6f7-86e70fb5826a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.36:8080: connect: connection refused"
Jul 27 21:37:29.841078 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:37:29.840996    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-dmx8m"
Jul 27 21:37:39.048619 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:37:39.048583    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-dmx8m"]
Jul 27 21:37:39.049195 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:37:39.048975    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-dmx8m" podUID="e6da3ed4-2e22-489d-b6f7-86e70fb5826a" containerName="kserve-container" containerID="cri-o://caee39126098df85589dc19940472c5c4e1db8d94be332ed672cfff2ed697350" gracePeriod=30
Jul 27 21:37:39.049475 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:37:39.049434    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-dmx8m" podUID="e6da3ed4-2e22-489d-b6f7-86e70fb5826a" containerName="kube-rbac-proxy" containerID="cri-o://72306ba0c2fdfee5fd619bb3164de227ffe32ae649fd224cfa12d5bf6ad30cdf" gracePeriod=30
Jul 27 21:37:39.233920 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:37:39.233887    2559 generic.go:356] "Generic (PLEG): container finished" podID="e6da3ed4-2e22-489d-b6f7-86e70fb5826a" containerID="72306ba0c2fdfee5fd619bb3164de227ffe32ae649fd224cfa12d5bf6ad30cdf" exitCode=2
Jul 27 21:37:39.234080 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:37:39.233954    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-dmx8m" event={"ID":"e6da3ed4-2e22-489d-b6f7-86e70fb5826a","Type":"ContainerDied","Data":"72306ba0c2fdfee5fd619bb3164de227ffe32ae649fd224cfa12d5bf6ad30cdf"}
Jul 27 21:37:39.836619 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:37:39.836572    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-dmx8m" podUID="e6da3ed4-2e22-489d-b6f7-86e70fb5826a" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.36:8643/healthz\": dial tcp 10.132.0.36:8643: connect: connection refused"
Jul 27 21:37:39.839920 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:37:39.839886    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-dmx8m" podUID="e6da3ed4-2e22-489d-b6f7-86e70fb5826a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.36:8080: connect: connection refused"
Jul 27 21:37:44.103968 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:37:44.103940    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-dmx8m"
Jul 27 21:37:44.209315 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:37:44.209201    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e6da3ed4-2e22-489d-b6f7-86e70fb5826a-proxy-tls\") pod \"e6da3ed4-2e22-489d-b6f7-86e70fb5826a\" (UID: \"e6da3ed4-2e22-489d-b6f7-86e70fb5826a\") "
Jul 27 21:37:44.209315 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:37:44.209266    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-x2gk8\" (UniqueName: \"kubernetes.io/projected/e6da3ed4-2e22-489d-b6f7-86e70fb5826a-kube-api-access-x2gk8\") pod \"e6da3ed4-2e22-489d-b6f7-86e70fb5826a\" (UID: \"e6da3ed4-2e22-489d-b6f7-86e70fb5826a\") "
Jul 27 21:37:44.209315 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:37:44.209295    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e6da3ed4-2e22-489d-b6f7-86e70fb5826a-kube-rbac-proxy-sar-config\") pod \"e6da3ed4-2e22-489d-b6f7-86e70fb5826a\" (UID: \"e6da3ed4-2e22-489d-b6f7-86e70fb5826a\") "
Jul 27 21:37:44.209614 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:37:44.209323    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e6da3ed4-2e22-489d-b6f7-86e70fb5826a-kserve-provision-location\") pod \"e6da3ed4-2e22-489d-b6f7-86e70fb5826a\" (UID: \"e6da3ed4-2e22-489d-b6f7-86e70fb5826a\") "
Jul 27 21:37:44.209614 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:37:44.209349    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e6da3ed4-2e22-489d-b6f7-86e70fb5826a-cabundle-cert\") pod \"e6da3ed4-2e22-489d-b6f7-86e70fb5826a\" (UID: \"e6da3ed4-2e22-489d-b6f7-86e70fb5826a\") "
Jul 27 21:37:44.209729 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:37:44.209625    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e6da3ed4-2e22-489d-b6f7-86e70fb5826a-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "e6da3ed4-2e22-489d-b6f7-86e70fb5826a" (UID: "e6da3ed4-2e22-489d-b6f7-86e70fb5826a"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 21:37:44.209729 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:37:44.209693    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e6da3ed4-2e22-489d-b6f7-86e70fb5826a-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "e6da3ed4-2e22-489d-b6f7-86e70fb5826a" (UID: "e6da3ed4-2e22-489d-b6f7-86e70fb5826a"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 21:37:44.209832 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:37:44.209755    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e6da3ed4-2e22-489d-b6f7-86e70fb5826a-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "e6da3ed4-2e22-489d-b6f7-86e70fb5826a" (UID: "e6da3ed4-2e22-489d-b6f7-86e70fb5826a"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 21:37:44.211586 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:37:44.211563    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e6da3ed4-2e22-489d-b6f7-86e70fb5826a-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "e6da3ed4-2e22-489d-b6f7-86e70fb5826a" (UID: "e6da3ed4-2e22-489d-b6f7-86e70fb5826a"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 21:37:44.211692 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:37:44.211613    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e6da3ed4-2e22-489d-b6f7-86e70fb5826a-kube-api-access-x2gk8" (OuterVolumeSpecName: "kube-api-access-x2gk8") pod "e6da3ed4-2e22-489d-b6f7-86e70fb5826a" (UID: "e6da3ed4-2e22-489d-b6f7-86e70fb5826a"). InnerVolumeSpecName "kube-api-access-x2gk8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 21:37:44.230281 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:37:44.230243    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-84f48b7656-xzg7z"]
Jul 27 21:37:44.230626 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:37:44.230613    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e6da3ed4-2e22-489d-b6f7-86e70fb5826a" containerName="kserve-container"
Jul 27 21:37:44.230669 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:37:44.230627    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="e6da3ed4-2e22-489d-b6f7-86e70fb5826a" containerName="kserve-container"
Jul 27 21:37:44.230669 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:37:44.230646    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e6da3ed4-2e22-489d-b6f7-86e70fb5826a" containerName="storage-initializer"
Jul 27 21:37:44.230669 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:37:44.230652    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="e6da3ed4-2e22-489d-b6f7-86e70fb5826a" containerName="storage-initializer"
Jul 27 21:37:44.230669 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:37:44.230661    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e6da3ed4-2e22-489d-b6f7-86e70fb5826a" containerName="kube-rbac-proxy"
Jul 27 21:37:44.230669 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:37:44.230666    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="e6da3ed4-2e22-489d-b6f7-86e70fb5826a" containerName="kube-rbac-proxy"
Jul 27 21:37:44.230821 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:37:44.230725    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="e6da3ed4-2e22-489d-b6f7-86e70fb5826a" containerName="kserve-container"
Jul 27 21:37:44.230821 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:37:44.230735    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="e6da3ed4-2e22-489d-b6f7-86e70fb5826a" containerName="kube-rbac-proxy"
Jul 27 21:37:44.234181 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:37:44.234161    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-84f48b7656-xzg7z"
Jul 27 21:37:44.236363 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:37:44.236341    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-rollback-v2-predictor-serving-cert\""
Jul 27 21:37:44.236555 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:37:44.236538    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-rollback-kube-rbac-proxy-sar-config\""
Jul 27 21:37:44.247639 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:37:44.247612    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-84f48b7656-xzg7z"]
Jul 27 21:37:44.263137 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:37:44.263084    2559 generic.go:356] "Generic (PLEG): container finished" podID="e6da3ed4-2e22-489d-b6f7-86e70fb5826a" containerID="caee39126098df85589dc19940472c5c4e1db8d94be332ed672cfff2ed697350" exitCode=0
Jul 27 21:37:44.263356 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:37:44.263160    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-dmx8m" event={"ID":"e6da3ed4-2e22-489d-b6f7-86e70fb5826a","Type":"ContainerDied","Data":"caee39126098df85589dc19940472c5c4e1db8d94be332ed672cfff2ed697350"}
Jul 27 21:37:44.263356 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:37:44.263258    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-dmx8m" event={"ID":"e6da3ed4-2e22-489d-b6f7-86e70fb5826a","Type":"ContainerDied","Data":"bdcd03d5abf22fe4269e60bda287d263c24f4753fc0c8d40d2919d3a9c5a17cb"}
Jul 27 21:37:44.263356 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:37:44.263289    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-dmx8m"
Jul 27 21:37:44.263532 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:37:44.263299    2559 scope.go:117] "RemoveContainer" containerID="72306ba0c2fdfee5fd619bb3164de227ffe32ae649fd224cfa12d5bf6ad30cdf"
Jul 27 21:37:44.279732 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:37:44.279710    2559 scope.go:117] "RemoveContainer" containerID="caee39126098df85589dc19940472c5c4e1db8d94be332ed672cfff2ed697350"
Jul 27 21:37:44.288312 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:37:44.288287    2559 scope.go:117] "RemoveContainer" containerID="c89ec63143d063a18149583f494131c0bc3589358c5c30be881661c2536c572b"
Jul 27 21:37:44.289780 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:37:44.289757    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-dmx8m"]
Jul 27 21:37:44.293668 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:37:44.293646    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-dmx8m"]
Jul 27 21:37:44.297070 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:37:44.297042    2559 scope.go:117] "RemoveContainer" containerID="72306ba0c2fdfee5fd619bb3164de227ffe32ae649fd224cfa12d5bf6ad30cdf"
Jul 27 21:37:44.297418 ip-10-0-131-133 kubenswrapper[2559]: E0727 21:37:44.297398    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"72306ba0c2fdfee5fd619bb3164de227ffe32ae649fd224cfa12d5bf6ad30cdf\": container with ID starting with 72306ba0c2fdfee5fd619bb3164de227ffe32ae649fd224cfa12d5bf6ad30cdf not found: ID does not exist" containerID="72306ba0c2fdfee5fd619bb3164de227ffe32ae649fd224cfa12d5bf6ad30cdf"
Jul 27 21:37:44.297498 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:37:44.297424    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"72306ba0c2fdfee5fd619bb3164de227ffe32ae649fd224cfa12d5bf6ad30cdf"} err="failed to get container status \"72306ba0c2fdfee5fd619bb3164de227ffe32ae649fd224cfa12d5bf6ad30cdf\": rpc error: code = NotFound desc = could not find container \"72306ba0c2fdfee5fd619bb3164de227ffe32ae649fd224cfa12d5bf6ad30cdf\": container with ID starting with 72306ba0c2fdfee5fd619bb3164de227ffe32ae649fd224cfa12d5bf6ad30cdf not found: ID does not exist"
Jul 27 21:37:44.297498 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:37:44.297444    2559 scope.go:117] "RemoveContainer" containerID="caee39126098df85589dc19940472c5c4e1db8d94be332ed672cfff2ed697350"
Jul 27 21:37:44.297712 ip-10-0-131-133 kubenswrapper[2559]: E0727 21:37:44.297692    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"caee39126098df85589dc19940472c5c4e1db8d94be332ed672cfff2ed697350\": container with ID starting with caee39126098df85589dc19940472c5c4e1db8d94be332ed672cfff2ed697350 not found: ID does not exist" containerID="caee39126098df85589dc19940472c5c4e1db8d94be332ed672cfff2ed697350"
Jul 27 21:37:44.297765 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:37:44.297721    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"caee39126098df85589dc19940472c5c4e1db8d94be332ed672cfff2ed697350"} err="failed to get container status \"caee39126098df85589dc19940472c5c4e1db8d94be332ed672cfff2ed697350\": rpc error: code = NotFound desc = could not find container \"caee39126098df85589dc19940472c5c4e1db8d94be332ed672cfff2ed697350\": container with ID starting with caee39126098df85589dc19940472c5c4e1db8d94be332ed672cfff2ed697350 not found: ID does not exist"
Jul 27 21:37:44.297765 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:37:44.297737    2559 scope.go:117] "RemoveContainer" containerID="c89ec63143d063a18149583f494131c0bc3589358c5c30be881661c2536c572b"
Jul 27 21:37:44.297967 ip-10-0-131-133 kubenswrapper[2559]: E0727 21:37:44.297952    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c89ec63143d063a18149583f494131c0bc3589358c5c30be881661c2536c572b\": container with ID starting with c89ec63143d063a18149583f494131c0bc3589358c5c30be881661c2536c572b not found: ID does not exist" containerID="c89ec63143d063a18149583f494131c0bc3589358c5c30be881661c2536c572b"
Jul 27 21:37:44.298035 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:37:44.297969    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c89ec63143d063a18149583f494131c0bc3589358c5c30be881661c2536c572b"} err="failed to get container status \"c89ec63143d063a18149583f494131c0bc3589358c5c30be881661c2536c572b\": rpc error: code = NotFound desc = could not find container \"c89ec63143d063a18149583f494131c0bc3589358c5c30be881661c2536c572b\": container with ID starting with c89ec63143d063a18149583f494131c0bc3589358c5c30be881661c2536c572b not found: ID does not exist"
Jul 27 21:37:44.310494 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:37:44.310464    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ffmv4\" (UniqueName: \"kubernetes.io/projected/9beaa19c-28c7-4507-8277-ba8a01131e38-kube-api-access-ffmv4\") pod \"isvc-canary-rollback-v2-predictor-84f48b7656-xzg7z\" (UID: \"9beaa19c-28c7-4507-8277-ba8a01131e38\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-84f48b7656-xzg7z"
Jul 27 21:37:44.310615 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:37:44.310506    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9beaa19c-28c7-4507-8277-ba8a01131e38-kserve-provision-location\") pod \"isvc-canary-rollback-v2-predictor-84f48b7656-xzg7z\" (UID: \"9beaa19c-28c7-4507-8277-ba8a01131e38\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-84f48b7656-xzg7z"
Jul 27 21:37:44.310615 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:37:44.310533    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9beaa19c-28c7-4507-8277-ba8a01131e38-cabundle-cert\") pod \"isvc-canary-rollback-v2-predictor-84f48b7656-xzg7z\" (UID: \"9beaa19c-28c7-4507-8277-ba8a01131e38\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-84f48b7656-xzg7z"
Jul 27 21:37:44.310707 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:37:44.310635    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9beaa19c-28c7-4507-8277-ba8a01131e38-proxy-tls\") pod \"isvc-canary-rollback-v2-predictor-84f48b7656-xzg7z\" (UID: \"9beaa19c-28c7-4507-8277-ba8a01131e38\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-84f48b7656-xzg7z"
Jul 27 21:37:44.310707 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:37:44.310683    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9beaa19c-28c7-4507-8277-ba8a01131e38-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-v2-predictor-84f48b7656-xzg7z\" (UID: \"9beaa19c-28c7-4507-8277-ba8a01131e38\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-84f48b7656-xzg7z"
Jul 27 21:37:44.310801 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:37:44.310784    2559 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e6da3ed4-2e22-489d-b6f7-86e70fb5826a-proxy-tls\") on node \"ip-10-0-131-133.ec2.internal\" DevicePath \"\""
Jul 27 21:37:44.310856 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:37:44.310808    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x2gk8\" (UniqueName: \"kubernetes.io/projected/e6da3ed4-2e22-489d-b6f7-86e70fb5826a-kube-api-access-x2gk8\") on node \"ip-10-0-131-133.ec2.internal\" DevicePath \"\""
Jul 27 21:37:44.310856 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:37:44.310825    2559 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e6da3ed4-2e22-489d-b6f7-86e70fb5826a-kube-rbac-proxy-sar-config\") on node \"ip-10-0-131-133.ec2.internal\" DevicePath \"\""
Jul 27 21:37:44.310856 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:37:44.310840    2559 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e6da3ed4-2e22-489d-b6f7-86e70fb5826a-kserve-provision-location\") on node \"ip-10-0-131-133.ec2.internal\" DevicePath \"\""
Jul 27 21:37:44.310957 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:37:44.310855    2559 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e6da3ed4-2e22-489d-b6f7-86e70fb5826a-cabundle-cert\") on node \"ip-10-0-131-133.ec2.internal\" DevicePath \"\""
Jul 27 21:37:44.412145 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:37:44.412106    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9beaa19c-28c7-4507-8277-ba8a01131e38-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-v2-predictor-84f48b7656-xzg7z\" (UID: \"9beaa19c-28c7-4507-8277-ba8a01131e38\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-84f48b7656-xzg7z"
Jul 27 21:37:44.412393 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:37:44.412173    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ffmv4\" (UniqueName: \"kubernetes.io/projected/9beaa19c-28c7-4507-8277-ba8a01131e38-kube-api-access-ffmv4\") pod \"isvc-canary-rollback-v2-predictor-84f48b7656-xzg7z\" (UID: \"9beaa19c-28c7-4507-8277-ba8a01131e38\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-84f48b7656-xzg7z"
Jul 27 21:37:44.412393 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:37:44.412196    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9beaa19c-28c7-4507-8277-ba8a01131e38-kserve-provision-location\") pod \"isvc-canary-rollback-v2-predictor-84f48b7656-xzg7z\" (UID: \"9beaa19c-28c7-4507-8277-ba8a01131e38\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-84f48b7656-xzg7z"
Jul 27 21:37:44.412393 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:37:44.412244    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9beaa19c-28c7-4507-8277-ba8a01131e38-cabundle-cert\") pod \"isvc-canary-rollback-v2-predictor-84f48b7656-xzg7z\" (UID: \"9beaa19c-28c7-4507-8277-ba8a01131e38\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-84f48b7656-xzg7z"
Jul 27 21:37:44.412393 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:37:44.412295    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9beaa19c-28c7-4507-8277-ba8a01131e38-proxy-tls\") pod \"isvc-canary-rollback-v2-predictor-84f48b7656-xzg7z\" (UID: \"9beaa19c-28c7-4507-8277-ba8a01131e38\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-84f48b7656-xzg7z"
Jul 27 21:37:44.412739 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:37:44.412713    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9beaa19c-28c7-4507-8277-ba8a01131e38-kserve-provision-location\") pod \"isvc-canary-rollback-v2-predictor-84f48b7656-xzg7z\" (UID: \"9beaa19c-28c7-4507-8277-ba8a01131e38\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-84f48b7656-xzg7z"
Jul 27 21:37:44.412885 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:37:44.412862    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9beaa19c-28c7-4507-8277-ba8a01131e38-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-v2-predictor-84f48b7656-xzg7z\" (UID: \"9beaa19c-28c7-4507-8277-ba8a01131e38\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-84f48b7656-xzg7z"
Jul 27 21:37:44.412975 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:37:44.412895    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9beaa19c-28c7-4507-8277-ba8a01131e38-cabundle-cert\") pod \"isvc-canary-rollback-v2-predictor-84f48b7656-xzg7z\" (UID: \"9beaa19c-28c7-4507-8277-ba8a01131e38\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-84f48b7656-xzg7z"
Jul 27 21:37:44.414754 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:37:44.414735    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9beaa19c-28c7-4507-8277-ba8a01131e38-proxy-tls\") pod \"isvc-canary-rollback-v2-predictor-84f48b7656-xzg7z\" (UID: \"9beaa19c-28c7-4507-8277-ba8a01131e38\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-84f48b7656-xzg7z"
Jul 27 21:37:44.419932 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:37:44.419910    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ffmv4\" (UniqueName: \"kubernetes.io/projected/9beaa19c-28c7-4507-8277-ba8a01131e38-kube-api-access-ffmv4\") pod \"isvc-canary-rollback-v2-predictor-84f48b7656-xzg7z\" (UID: \"9beaa19c-28c7-4507-8277-ba8a01131e38\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-84f48b7656-xzg7z"
Jul 27 21:37:44.544702 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:37:44.544613    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-84f48b7656-xzg7z"
Jul 27 21:37:44.680642 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:37:44.680618    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-84f48b7656-xzg7z"]
Jul 27 21:37:44.682788 ip-10-0-131-133 kubenswrapper[2559]: W0727 21:37:44.682758    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9beaa19c_28c7_4507_8277_ba8a01131e38.slice/crio-f38fac499022025eb2c22ef390fd23df7e67ad481bdff34f789d884a03779c4e WatchSource:0}: Error finding container f38fac499022025eb2c22ef390fd23df7e67ad481bdff34f789d884a03779c4e: Status 404 returned error can't find the container with id f38fac499022025eb2c22ef390fd23df7e67ad481bdff34f789d884a03779c4e
Jul 27 21:37:45.268895 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:37:45.268854    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-84f48b7656-xzg7z" event={"ID":"9beaa19c-28c7-4507-8277-ba8a01131e38","Type":"ContainerStarted","Data":"1819147993a16d1cb00109478f558ed4a2d15181e9e7a503bc984abb628fb975"}
Jul 27 21:37:45.269381 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:37:45.268926    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-84f48b7656-xzg7z" event={"ID":"9beaa19c-28c7-4507-8277-ba8a01131e38","Type":"ContainerStarted","Data":"f38fac499022025eb2c22ef390fd23df7e67ad481bdff34f789d884a03779c4e"}
Jul 27 21:37:45.982083 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:37:45.982040    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e6da3ed4-2e22-489d-b6f7-86e70fb5826a" path="/var/lib/kubelet/pods/e6da3ed4-2e22-489d-b6f7-86e70fb5826a/volumes"
Jul 27 21:37:49.289336 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:37:49.289304    2559 generic.go:356] "Generic (PLEG): container finished" podID="9beaa19c-28c7-4507-8277-ba8a01131e38" containerID="1819147993a16d1cb00109478f558ed4a2d15181e9e7a503bc984abb628fb975" exitCode=0
Jul 27 21:37:49.289821 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:37:49.289354    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-84f48b7656-xzg7z" event={"ID":"9beaa19c-28c7-4507-8277-ba8a01131e38","Type":"ContainerDied","Data":"1819147993a16d1cb00109478f558ed4a2d15181e9e7a503bc984abb628fb975"}
Jul 27 21:37:50.301898 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:37:50.301865    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-84f48b7656-xzg7z" event={"ID":"9beaa19c-28c7-4507-8277-ba8a01131e38","Type":"ContainerStarted","Data":"381d24c74b5cc3782a9f9ffc16fcd5a8d2fb2ef2ff9048cadeca43d20ce3cf87"}
Jul 27 21:37:50.302310 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:37:50.301954    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-84f48b7656-xzg7z" event={"ID":"9beaa19c-28c7-4507-8277-ba8a01131e38","Type":"ContainerStarted","Data":"2d1ad0edd948428f22053835903801ffa82ea1e7f78b3e46648c9265f57e253f"}
Jul 27 21:37:50.302365 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:37:50.302311    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-84f48b7656-xzg7z"
Jul 27 21:37:50.302407 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:37:50.302382    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-84f48b7656-xzg7z"
Jul 27 21:37:50.303748 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:37:50.303718    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-84f48b7656-xzg7z" podUID="9beaa19c-28c7-4507-8277-ba8a01131e38" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.37:8080: connect: connection refused"
Jul 27 21:37:50.321186 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:37:50.321134    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-84f48b7656-xzg7z" podStartSLOduration=2.573284522 podStartE2EDuration="6.321118941s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.747834419s" isStatefulPod=false podCreationTimestamp="2026-07-27 21:37:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 21:37:50.319126206 +0000 UTC m=+1998.898310234" watchObservedRunningTime="2026-07-27 21:37:50.321118941 +0000 UTC m=+1998.900302947"
Jul 27 21:37:51.307262 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:37:51.307226    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-84f48b7656-xzg7z" podUID="9beaa19c-28c7-4507-8277-ba8a01131e38" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.37:8080: connect: connection refused"
Jul 27 21:37:56.311476 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:37:56.311441    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-84f48b7656-xzg7z"
Jul 27 21:37:56.312145 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:37:56.312116    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-84f48b7656-xzg7z" podUID="9beaa19c-28c7-4507-8277-ba8a01131e38" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.37:8080: connect: connection refused"
Jul 27 21:38:06.312455 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:38:06.312413    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-84f48b7656-xzg7z" podUID="9beaa19c-28c7-4507-8277-ba8a01131e38" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.37:8080: connect: connection refused"
Jul 27 21:38:16.312140 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:38:16.312102    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-84f48b7656-xzg7z" podUID="9beaa19c-28c7-4507-8277-ba8a01131e38" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.37:8080: connect: connection refused"
Jul 27 21:38:26.312079 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:38:26.312039    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-84f48b7656-xzg7z" podUID="9beaa19c-28c7-4507-8277-ba8a01131e38" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.37:8080: connect: connection refused"
Jul 27 21:38:36.312450 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:38:36.312407    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-84f48b7656-xzg7z" podUID="9beaa19c-28c7-4507-8277-ba8a01131e38" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.37:8080: connect: connection refused"
Jul 27 21:38:46.312312 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:38:46.312271    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-84f48b7656-xzg7z" podUID="9beaa19c-28c7-4507-8277-ba8a01131e38" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.37:8080: connect: connection refused"
Jul 27 21:38:56.312456 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:38:56.312416    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-84f48b7656-xzg7z" podUID="9beaa19c-28c7-4507-8277-ba8a01131e38" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.37:8080: connect: connection refused"
Jul 27 21:39:06.313209 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:39:06.313179    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-84f48b7656-xzg7z"
Jul 27 21:39:14.257664 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:39:14.257634    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-84f48b7656-xzg7z"]
Jul 27 21:39:14.258039 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:39:14.257872    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-84f48b7656-xzg7z" podUID="9beaa19c-28c7-4507-8277-ba8a01131e38" containerName="kube-rbac-proxy" containerID="cri-o://381d24c74b5cc3782a9f9ffc16fcd5a8d2fb2ef2ff9048cadeca43d20ce3cf87" gracePeriod=30
Jul 27 21:39:14.258039 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:39:14.257859    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-84f48b7656-xzg7z" podUID="9beaa19c-28c7-4507-8277-ba8a01131e38" containerName="kserve-container" containerID="cri-o://2d1ad0edd948428f22053835903801ffa82ea1e7f78b3e46648c9265f57e253f" gracePeriod=30
Jul 27 21:39:14.324293 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:39:14.324265    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-b8ngh"]
Jul 27 21:39:14.327834 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:39:14.327819    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-b8ngh"
Jul 27 21:39:14.330077 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:39:14.330059    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-rollback-predictor-serving-cert\""
Jul 27 21:39:14.338316 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:39:14.338295    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-b8ngh"]
Jul 27 21:39:14.414769 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:39:14.414730    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9624b8a0-6314-4ed2-ac50-ea171bb73e45-kserve-provision-location\") pod \"isvc-canary-rollback-predictor-5bbb8c8cfc-b8ngh\" (UID: \"9624b8a0-6314-4ed2-ac50-ea171bb73e45\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-b8ngh"
Jul 27 21:39:14.414891 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:39:14.414784    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9624b8a0-6314-4ed2-ac50-ea171bb73e45-cabundle-cert\") pod \"isvc-canary-rollback-predictor-5bbb8c8cfc-b8ngh\" (UID: \"9624b8a0-6314-4ed2-ac50-ea171bb73e45\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-b8ngh"
Jul 27 21:39:14.414891 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:39:14.414878    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9624b8a0-6314-4ed2-ac50-ea171bb73e45-proxy-tls\") pod \"isvc-canary-rollback-predictor-5bbb8c8cfc-b8ngh\" (UID: \"9624b8a0-6314-4ed2-ac50-ea171bb73e45\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-b8ngh"
Jul 27 21:39:14.414966 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:39:14.414910    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hfnqh\" (UniqueName: \"kubernetes.io/projected/9624b8a0-6314-4ed2-ac50-ea171bb73e45-kube-api-access-hfnqh\") pod \"isvc-canary-rollback-predictor-5bbb8c8cfc-b8ngh\" (UID: \"9624b8a0-6314-4ed2-ac50-ea171bb73e45\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-b8ngh"
Jul 27 21:39:14.414966 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:39:14.414958    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9624b8a0-6314-4ed2-ac50-ea171bb73e45-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-predictor-5bbb8c8cfc-b8ngh\" (UID: \"9624b8a0-6314-4ed2-ac50-ea171bb73e45\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-b8ngh"
Jul 27 21:39:14.515830 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:39:14.515753    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9624b8a0-6314-4ed2-ac50-ea171bb73e45-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-predictor-5bbb8c8cfc-b8ngh\" (UID: \"9624b8a0-6314-4ed2-ac50-ea171bb73e45\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-b8ngh"
Jul 27 21:39:14.515830 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:39:14.515802    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9624b8a0-6314-4ed2-ac50-ea171bb73e45-kserve-provision-location\") pod \"isvc-canary-rollback-predictor-5bbb8c8cfc-b8ngh\" (UID: \"9624b8a0-6314-4ed2-ac50-ea171bb73e45\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-b8ngh"
Jul 27 21:39:14.515830 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:39:14.515831    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9624b8a0-6314-4ed2-ac50-ea171bb73e45-cabundle-cert\") pod \"isvc-canary-rollback-predictor-5bbb8c8cfc-b8ngh\" (UID: \"9624b8a0-6314-4ed2-ac50-ea171bb73e45\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-b8ngh"
Jul 27 21:39:14.516057 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:39:14.515871    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9624b8a0-6314-4ed2-ac50-ea171bb73e45-proxy-tls\") pod \"isvc-canary-rollback-predictor-5bbb8c8cfc-b8ngh\" (UID: \"9624b8a0-6314-4ed2-ac50-ea171bb73e45\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-b8ngh"
Jul 27 21:39:14.516057 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:39:14.515893    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hfnqh\" (UniqueName: \"kubernetes.io/projected/9624b8a0-6314-4ed2-ac50-ea171bb73e45-kube-api-access-hfnqh\") pod \"isvc-canary-rollback-predictor-5bbb8c8cfc-b8ngh\" (UID: \"9624b8a0-6314-4ed2-ac50-ea171bb73e45\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-b8ngh"
Jul 27 21:39:14.516259 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:39:14.516199    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9624b8a0-6314-4ed2-ac50-ea171bb73e45-kserve-provision-location\") pod \"isvc-canary-rollback-predictor-5bbb8c8cfc-b8ngh\" (UID: \"9624b8a0-6314-4ed2-ac50-ea171bb73e45\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-b8ngh"
Jul 27 21:39:14.516487 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:39:14.516467    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9624b8a0-6314-4ed2-ac50-ea171bb73e45-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-predictor-5bbb8c8cfc-b8ngh\" (UID: \"9624b8a0-6314-4ed2-ac50-ea171bb73e45\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-b8ngh"
Jul 27 21:39:14.516557 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:39:14.516505    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9624b8a0-6314-4ed2-ac50-ea171bb73e45-cabundle-cert\") pod \"isvc-canary-rollback-predictor-5bbb8c8cfc-b8ngh\" (UID: \"9624b8a0-6314-4ed2-ac50-ea171bb73e45\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-b8ngh"
Jul 27 21:39:14.518384 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:39:14.518363    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9624b8a0-6314-4ed2-ac50-ea171bb73e45-proxy-tls\") pod \"isvc-canary-rollback-predictor-5bbb8c8cfc-b8ngh\" (UID: \"9624b8a0-6314-4ed2-ac50-ea171bb73e45\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-b8ngh"
Jul 27 21:39:14.525241 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:39:14.525191    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hfnqh\" (UniqueName: \"kubernetes.io/projected/9624b8a0-6314-4ed2-ac50-ea171bb73e45-kube-api-access-hfnqh\") pod \"isvc-canary-rollback-predictor-5bbb8c8cfc-b8ngh\" (UID: \"9624b8a0-6314-4ed2-ac50-ea171bb73e45\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-b8ngh"
Jul 27 21:39:14.638150 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:39:14.638116    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-b8ngh"
Jul 27 21:39:14.704810 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:39:14.704783    2559 generic.go:356] "Generic (PLEG): container finished" podID="9beaa19c-28c7-4507-8277-ba8a01131e38" containerID="381d24c74b5cc3782a9f9ffc16fcd5a8d2fb2ef2ff9048cadeca43d20ce3cf87" exitCode=2
Jul 27 21:39:14.704930 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:39:14.704863    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-84f48b7656-xzg7z" event={"ID":"9beaa19c-28c7-4507-8277-ba8a01131e38","Type":"ContainerDied","Data":"381d24c74b5cc3782a9f9ffc16fcd5a8d2fb2ef2ff9048cadeca43d20ce3cf87"}
Jul 27 21:39:14.764407 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:39:14.764384    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-b8ngh"]
Jul 27 21:39:14.766355 ip-10-0-131-133 kubenswrapper[2559]: W0727 21:39:14.766302    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9624b8a0_6314_4ed2_ac50_ea171bb73e45.slice/crio-451e5f60ae33fd58ed8e69890a13bacda53abcf6372764015f5d726043d5b3f5 WatchSource:0}: Error finding container 451e5f60ae33fd58ed8e69890a13bacda53abcf6372764015f5d726043d5b3f5: Status 404 returned error can't find the container with id 451e5f60ae33fd58ed8e69890a13bacda53abcf6372764015f5d726043d5b3f5
Jul 27 21:39:15.710904 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:39:15.710869    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-b8ngh" event={"ID":"9624b8a0-6314-4ed2-ac50-ea171bb73e45","Type":"ContainerStarted","Data":"b70c81686b15b7ea8d1e5c298b91b0345c139f4401ed4b13f8b3e194fb4070fc"}
Jul 27 21:39:15.711326 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:39:15.710934    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-b8ngh" event={"ID":"9624b8a0-6314-4ed2-ac50-ea171bb73e45","Type":"ContainerStarted","Data":"451e5f60ae33fd58ed8e69890a13bacda53abcf6372764015f5d726043d5b3f5"}
Jul 27 21:39:16.308469 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:39:16.308426    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-84f48b7656-xzg7z" podUID="9beaa19c-28c7-4507-8277-ba8a01131e38" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.37:8643/healthz\": dial tcp 10.132.0.37:8643: connect: connection refused"
Jul 27 21:39:16.312755 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:39:16.312731    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-84f48b7656-xzg7z" podUID="9beaa19c-28c7-4507-8277-ba8a01131e38" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.37:8080: connect: connection refused"
Jul 27 21:39:18.726737 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:39:18.726651    2559 generic.go:356] "Generic (PLEG): container finished" podID="9624b8a0-6314-4ed2-ac50-ea171bb73e45" containerID="b70c81686b15b7ea8d1e5c298b91b0345c139f4401ed4b13f8b3e194fb4070fc" exitCode=0
Jul 27 21:39:18.726737 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:39:18.726700    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-b8ngh" event={"ID":"9624b8a0-6314-4ed2-ac50-ea171bb73e45","Type":"ContainerDied","Data":"b70c81686b15b7ea8d1e5c298b91b0345c139f4401ed4b13f8b3e194fb4070fc"}
Jul 27 21:39:19.005278 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:39:19.005256    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-84f48b7656-xzg7z"
Jul 27 21:39:19.163098 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:39:19.163065    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9beaa19c-28c7-4507-8277-ba8a01131e38-kube-rbac-proxy-sar-config\") pod \"9beaa19c-28c7-4507-8277-ba8a01131e38\" (UID: \"9beaa19c-28c7-4507-8277-ba8a01131e38\") "
Jul 27 21:39:19.163323 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:39:19.163117    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9beaa19c-28c7-4507-8277-ba8a01131e38-kserve-provision-location\") pod \"9beaa19c-28c7-4507-8277-ba8a01131e38\" (UID: \"9beaa19c-28c7-4507-8277-ba8a01131e38\") "
Jul 27 21:39:19.163323 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:39:19.163158    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9beaa19c-28c7-4507-8277-ba8a01131e38-cabundle-cert\") pod \"9beaa19c-28c7-4507-8277-ba8a01131e38\" (UID: \"9beaa19c-28c7-4507-8277-ba8a01131e38\") "
Jul 27 21:39:19.163323 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:39:19.163185    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ffmv4\" (UniqueName: \"kubernetes.io/projected/9beaa19c-28c7-4507-8277-ba8a01131e38-kube-api-access-ffmv4\") pod \"9beaa19c-28c7-4507-8277-ba8a01131e38\" (UID: \"9beaa19c-28c7-4507-8277-ba8a01131e38\") "
Jul 27 21:39:19.163323 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:39:19.163265    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9beaa19c-28c7-4507-8277-ba8a01131e38-proxy-tls\") pod \"9beaa19c-28c7-4507-8277-ba8a01131e38\" (UID: \"9beaa19c-28c7-4507-8277-ba8a01131e38\") "
Jul 27 21:39:19.163548 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:39:19.163460    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9beaa19c-28c7-4507-8277-ba8a01131e38-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "9beaa19c-28c7-4507-8277-ba8a01131e38" (UID: "9beaa19c-28c7-4507-8277-ba8a01131e38"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 21:39:19.163548 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:39:19.163478    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9beaa19c-28c7-4507-8277-ba8a01131e38-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "9beaa19c-28c7-4507-8277-ba8a01131e38" (UID: "9beaa19c-28c7-4507-8277-ba8a01131e38"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 21:39:19.163548 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:39:19.163532    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9beaa19c-28c7-4507-8277-ba8a01131e38-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "9beaa19c-28c7-4507-8277-ba8a01131e38" (UID: "9beaa19c-28c7-4507-8277-ba8a01131e38"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 21:39:19.163672 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:39:19.163540    2559 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9beaa19c-28c7-4507-8277-ba8a01131e38-kube-rbac-proxy-sar-config\") on node \"ip-10-0-131-133.ec2.internal\" DevicePath \"\""
Jul 27 21:39:19.163672 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:39:19.163574    2559 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9beaa19c-28c7-4507-8277-ba8a01131e38-kserve-provision-location\") on node \"ip-10-0-131-133.ec2.internal\" DevicePath \"\""
Jul 27 21:39:19.165353 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:39:19.165329    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9beaa19c-28c7-4507-8277-ba8a01131e38-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "9beaa19c-28c7-4507-8277-ba8a01131e38" (UID: "9beaa19c-28c7-4507-8277-ba8a01131e38"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 21:39:19.165442 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:39:19.165401    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9beaa19c-28c7-4507-8277-ba8a01131e38-kube-api-access-ffmv4" (OuterVolumeSpecName: "kube-api-access-ffmv4") pod "9beaa19c-28c7-4507-8277-ba8a01131e38" (UID: "9beaa19c-28c7-4507-8277-ba8a01131e38"). InnerVolumeSpecName "kube-api-access-ffmv4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 21:39:19.264646 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:39:19.264568    2559 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9beaa19c-28c7-4507-8277-ba8a01131e38-cabundle-cert\") on node \"ip-10-0-131-133.ec2.internal\" DevicePath \"\""
Jul 27 21:39:19.264646 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:39:19.264595    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ffmv4\" (UniqueName: \"kubernetes.io/projected/9beaa19c-28c7-4507-8277-ba8a01131e38-kube-api-access-ffmv4\") on node \"ip-10-0-131-133.ec2.internal\" DevicePath \"\""
Jul 27 21:39:19.264646 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:39:19.264604    2559 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9beaa19c-28c7-4507-8277-ba8a01131e38-proxy-tls\") on node \"ip-10-0-131-133.ec2.internal\" DevicePath \"\""
Jul 27 21:39:19.733636 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:39:19.733602    2559 generic.go:356] "Generic (PLEG): container finished" podID="9beaa19c-28c7-4507-8277-ba8a01131e38" containerID="2d1ad0edd948428f22053835903801ffa82ea1e7f78b3e46648c9265f57e253f" exitCode=0
Jul 27 21:39:19.734068 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:39:19.733662    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-84f48b7656-xzg7z"
Jul 27 21:39:19.734068 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:39:19.733680    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-84f48b7656-xzg7z" event={"ID":"9beaa19c-28c7-4507-8277-ba8a01131e38","Type":"ContainerDied","Data":"2d1ad0edd948428f22053835903801ffa82ea1e7f78b3e46648c9265f57e253f"}
Jul 27 21:39:19.734068 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:39:19.733737    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-84f48b7656-xzg7z" event={"ID":"9beaa19c-28c7-4507-8277-ba8a01131e38","Type":"ContainerDied","Data":"f38fac499022025eb2c22ef390fd23df7e67ad481bdff34f789d884a03779c4e"}
Jul 27 21:39:19.734068 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:39:19.733766    2559 scope.go:117] "RemoveContainer" containerID="381d24c74b5cc3782a9f9ffc16fcd5a8d2fb2ef2ff9048cadeca43d20ce3cf87"
Jul 27 21:39:19.736077 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:39:19.736053    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-b8ngh" event={"ID":"9624b8a0-6314-4ed2-ac50-ea171bb73e45","Type":"ContainerStarted","Data":"42eef4c0be333017c78d233bd93a18c72ae712741a6c91cee729884107a85bca"}
Jul 27 21:39:19.736174 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:39:19.736104    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-b8ngh" event={"ID":"9624b8a0-6314-4ed2-ac50-ea171bb73e45","Type":"ContainerStarted","Data":"c4b4661e54a2bb624210769d0525ad4714a96ee041a8c61ffe733dc4d7e12cd7"}
Jul 27 21:39:19.736297 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:39:19.736278    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-b8ngh"
Jul 27 21:39:19.736364 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:39:19.736334    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-b8ngh"
Jul 27 21:39:19.738086 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:39:19.738060    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-b8ngh" podUID="9624b8a0-6314-4ed2-ac50-ea171bb73e45" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.38:8080: connect: connection refused"
Jul 27 21:39:19.742768 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:39:19.742753    2559 scope.go:117] "RemoveContainer" containerID="2d1ad0edd948428f22053835903801ffa82ea1e7f78b3e46648c9265f57e253f"
Jul 27 21:39:19.750331 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:39:19.750312    2559 scope.go:117] "RemoveContainer" containerID="1819147993a16d1cb00109478f558ed4a2d15181e9e7a503bc984abb628fb975"
Jul 27 21:39:19.756312 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:39:19.756234    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-b8ngh" podStartSLOduration=2.182925331 podStartE2EDuration="5.756206863s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.573281532s" isStatefulPod=false podCreationTimestamp="2026-07-27 21:39:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 21:39:19.755560896 +0000 UTC m=+2088.334744902" watchObservedRunningTime="2026-07-27 21:39:19.756206863 +0000 UTC m=+2088.335390871"
Jul 27 21:39:19.757902 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:39:19.757880    2559 scope.go:117] "RemoveContainer" containerID="381d24c74b5cc3782a9f9ffc16fcd5a8d2fb2ef2ff9048cadeca43d20ce3cf87"
Jul 27 21:39:19.758146 ip-10-0-131-133 kubenswrapper[2559]: E0727 21:39:19.758121    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"381d24c74b5cc3782a9f9ffc16fcd5a8d2fb2ef2ff9048cadeca43d20ce3cf87\": container with ID starting with 381d24c74b5cc3782a9f9ffc16fcd5a8d2fb2ef2ff9048cadeca43d20ce3cf87 not found: ID does not exist" containerID="381d24c74b5cc3782a9f9ffc16fcd5a8d2fb2ef2ff9048cadeca43d20ce3cf87"
Jul 27 21:39:19.758191 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:39:19.758145    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"381d24c74b5cc3782a9f9ffc16fcd5a8d2fb2ef2ff9048cadeca43d20ce3cf87"} err="failed to get container status \"381d24c74b5cc3782a9f9ffc16fcd5a8d2fb2ef2ff9048cadeca43d20ce3cf87\": rpc error: code = NotFound desc = could not find container \"381d24c74b5cc3782a9f9ffc16fcd5a8d2fb2ef2ff9048cadeca43d20ce3cf87\": container with ID starting with 381d24c74b5cc3782a9f9ffc16fcd5a8d2fb2ef2ff9048cadeca43d20ce3cf87 not found: ID does not exist"
Jul 27 21:39:19.758191 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:39:19.758161    2559 scope.go:117] "RemoveContainer" containerID="2d1ad0edd948428f22053835903801ffa82ea1e7f78b3e46648c9265f57e253f"
Jul 27 21:39:19.758494 ip-10-0-131-133 kubenswrapper[2559]: E0727 21:39:19.758466    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2d1ad0edd948428f22053835903801ffa82ea1e7f78b3e46648c9265f57e253f\": container with ID starting with 2d1ad0edd948428f22053835903801ffa82ea1e7f78b3e46648c9265f57e253f not found: ID does not exist" containerID="2d1ad0edd948428f22053835903801ffa82ea1e7f78b3e46648c9265f57e253f"
Jul 27 21:39:19.758574 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:39:19.758492    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2d1ad0edd948428f22053835903801ffa82ea1e7f78b3e46648c9265f57e253f"} err="failed to get container status \"2d1ad0edd948428f22053835903801ffa82ea1e7f78b3e46648c9265f57e253f\": rpc error: code = NotFound desc = could not find container \"2d1ad0edd948428f22053835903801ffa82ea1e7f78b3e46648c9265f57e253f\": container with ID starting with 2d1ad0edd948428f22053835903801ffa82ea1e7f78b3e46648c9265f57e253f not found: ID does not exist"
Jul 27 21:39:19.758574 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:39:19.758517    2559 scope.go:117] "RemoveContainer" containerID="1819147993a16d1cb00109478f558ed4a2d15181e9e7a503bc984abb628fb975"
Jul 27 21:39:19.758717 ip-10-0-131-133 kubenswrapper[2559]: E0727 21:39:19.758697    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1819147993a16d1cb00109478f558ed4a2d15181e9e7a503bc984abb628fb975\": container with ID starting with 1819147993a16d1cb00109478f558ed4a2d15181e9e7a503bc984abb628fb975 not found: ID does not exist" containerID="1819147993a16d1cb00109478f558ed4a2d15181e9e7a503bc984abb628fb975"
Jul 27 21:39:19.758764 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:39:19.758726    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1819147993a16d1cb00109478f558ed4a2d15181e9e7a503bc984abb628fb975"} err="failed to get container status \"1819147993a16d1cb00109478f558ed4a2d15181e9e7a503bc984abb628fb975\": rpc error: code = NotFound desc = could not find container \"1819147993a16d1cb00109478f558ed4a2d15181e9e7a503bc984abb628fb975\": container with ID starting with 1819147993a16d1cb00109478f558ed4a2d15181e9e7a503bc984abb628fb975 not found: ID does not exist"
Jul 27 21:39:19.768722 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:39:19.768702    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-84f48b7656-xzg7z"]
Jul 27 21:39:19.773071 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:39:19.773050    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-84f48b7656-xzg7z"]
Jul 27 21:39:19.982117 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:39:19.982076    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9beaa19c-28c7-4507-8277-ba8a01131e38" path="/var/lib/kubelet/pods/9beaa19c-28c7-4507-8277-ba8a01131e38/volumes"
Jul 27 21:39:20.741857 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:39:20.741816    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-b8ngh" podUID="9624b8a0-6314-4ed2-ac50-ea171bb73e45" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.38:8080: connect: connection refused"
Jul 27 21:39:25.746943 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:39:25.746912    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-b8ngh"
Jul 27 21:39:25.747503 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:39:25.747475    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-b8ngh" podUID="9624b8a0-6314-4ed2-ac50-ea171bb73e45" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.38:8080: connect: connection refused"
Jul 27 21:39:32.039675 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:39:32.039643    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-bjvbc_cba53c9a-64c0-4afa-8731-35521e196622/console-operator/1.log"
Jul 27 21:39:32.042597 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:39:32.042574    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-bjvbc_cba53c9a-64c0-4afa-8731-35521e196622/console-operator/1.log"
Jul 27 21:39:35.748295 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:39:35.748252    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-b8ngh" podUID="9624b8a0-6314-4ed2-ac50-ea171bb73e45" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.38:8080: connect: connection refused"
Jul 27 21:39:45.748152 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:39:45.748103    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-b8ngh" podUID="9624b8a0-6314-4ed2-ac50-ea171bb73e45" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.38:8080: connect: connection refused"
Jul 27 21:39:55.747514 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:39:55.747467    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-b8ngh" podUID="9624b8a0-6314-4ed2-ac50-ea171bb73e45" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.38:8080: connect: connection refused"
Jul 27 21:40:05.748018 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:40:05.747972    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-b8ngh" podUID="9624b8a0-6314-4ed2-ac50-ea171bb73e45" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.38:8080: connect: connection refused"
Jul 27 21:40:15.748045 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:40:15.748003    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-b8ngh" podUID="9624b8a0-6314-4ed2-ac50-ea171bb73e45" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.38:8080: connect: connection refused"
Jul 27 21:40:25.747432 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:40:25.747338    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-b8ngh" podUID="9624b8a0-6314-4ed2-ac50-ea171bb73e45" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.38:8080: connect: connection refused"
Jul 27 21:40:35.747915 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:40:35.747886    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-b8ngh"
Jul 27 21:40:44.448954 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:40:44.448911    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-b8ngh"]
Jul 27 21:40:44.449506 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:40:44.449209    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-b8ngh" podUID="9624b8a0-6314-4ed2-ac50-ea171bb73e45" containerName="kserve-container" containerID="cri-o://c4b4661e54a2bb624210769d0525ad4714a96ee041a8c61ffe733dc4d7e12cd7" gracePeriod=30
Jul 27 21:40:44.449506 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:40:44.449286    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-b8ngh" podUID="9624b8a0-6314-4ed2-ac50-ea171bb73e45" containerName="kube-rbac-proxy" containerID="cri-o://42eef4c0be333017c78d233bd93a18c72ae712741a6c91cee729884107a85bca" gracePeriod=30
Jul 27 21:40:45.130032 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:40:45.130001    2559 generic.go:356] "Generic (PLEG): container finished" podID="9624b8a0-6314-4ed2-ac50-ea171bb73e45" containerID="42eef4c0be333017c78d233bd93a18c72ae712741a6c91cee729884107a85bca" exitCode=2
Jul 27 21:40:45.130194 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:40:45.130070    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-b8ngh" event={"ID":"9624b8a0-6314-4ed2-ac50-ea171bb73e45","Type":"ContainerDied","Data":"42eef4c0be333017c78d233bd93a18c72ae712741a6c91cee729884107a85bca"}
Jul 27 21:40:45.742909 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:40:45.742864    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-b8ngh" podUID="9624b8a0-6314-4ed2-ac50-ea171bb73e45" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.38:8643/healthz\": dial tcp 10.132.0.38:8643: connect: connection refused"
Jul 27 21:40:45.747923 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:40:45.747895    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-b8ngh" podUID="9624b8a0-6314-4ed2-ac50-ea171bb73e45" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.38:8080: connect: connection refused"
Jul 27 21:40:49.007613 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:40:49.007586    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-b8ngh"
Jul 27 21:40:49.139389 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:40:49.139358    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9624b8a0-6314-4ed2-ac50-ea171bb73e45-proxy-tls\") pod \"9624b8a0-6314-4ed2-ac50-ea171bb73e45\" (UID: \"9624b8a0-6314-4ed2-ac50-ea171bb73e45\") "
Jul 27 21:40:49.139558 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:40:49.139403    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hfnqh\" (UniqueName: \"kubernetes.io/projected/9624b8a0-6314-4ed2-ac50-ea171bb73e45-kube-api-access-hfnqh\") pod \"9624b8a0-6314-4ed2-ac50-ea171bb73e45\" (UID: \"9624b8a0-6314-4ed2-ac50-ea171bb73e45\") "
Jul 27 21:40:49.139558 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:40:49.139484    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9624b8a0-6314-4ed2-ac50-ea171bb73e45-kserve-provision-location\") pod \"9624b8a0-6314-4ed2-ac50-ea171bb73e45\" (UID: \"9624b8a0-6314-4ed2-ac50-ea171bb73e45\") "
Jul 27 21:40:49.139558 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:40:49.139510    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9624b8a0-6314-4ed2-ac50-ea171bb73e45-cabundle-cert\") pod \"9624b8a0-6314-4ed2-ac50-ea171bb73e45\" (UID: \"9624b8a0-6314-4ed2-ac50-ea171bb73e45\") "
Jul 27 21:40:49.139558 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:40:49.139527    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9624b8a0-6314-4ed2-ac50-ea171bb73e45-kube-rbac-proxy-sar-config\") pod \"9624b8a0-6314-4ed2-ac50-ea171bb73e45\" (UID: \"9624b8a0-6314-4ed2-ac50-ea171bb73e45\") "
Jul 27 21:40:49.139805 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:40:49.139779    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9624b8a0-6314-4ed2-ac50-ea171bb73e45-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "9624b8a0-6314-4ed2-ac50-ea171bb73e45" (UID: "9624b8a0-6314-4ed2-ac50-ea171bb73e45"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 21:40:49.139898 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:40:49.139824    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9624b8a0-6314-4ed2-ac50-ea171bb73e45-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "9624b8a0-6314-4ed2-ac50-ea171bb73e45" (UID: "9624b8a0-6314-4ed2-ac50-ea171bb73e45"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 21:40:49.139978 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:40:49.139956    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9624b8a0-6314-4ed2-ac50-ea171bb73e45-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "9624b8a0-6314-4ed2-ac50-ea171bb73e45" (UID: "9624b8a0-6314-4ed2-ac50-ea171bb73e45"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 21:40:49.141539 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:40:49.141508    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9624b8a0-6314-4ed2-ac50-ea171bb73e45-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "9624b8a0-6314-4ed2-ac50-ea171bb73e45" (UID: "9624b8a0-6314-4ed2-ac50-ea171bb73e45"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 21:40:49.141659 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:40:49.141584    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9624b8a0-6314-4ed2-ac50-ea171bb73e45-kube-api-access-hfnqh" (OuterVolumeSpecName: "kube-api-access-hfnqh") pod "9624b8a0-6314-4ed2-ac50-ea171bb73e45" (UID: "9624b8a0-6314-4ed2-ac50-ea171bb73e45"). InnerVolumeSpecName "kube-api-access-hfnqh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 21:40:49.149680 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:40:49.149653    2559 generic.go:356] "Generic (PLEG): container finished" podID="9624b8a0-6314-4ed2-ac50-ea171bb73e45" containerID="c4b4661e54a2bb624210769d0525ad4714a96ee041a8c61ffe733dc4d7e12cd7" exitCode=0
Jul 27 21:40:49.149770 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:40:49.149703    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-b8ngh"
Jul 27 21:40:49.149770 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:40:49.149716    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-b8ngh" event={"ID":"9624b8a0-6314-4ed2-ac50-ea171bb73e45","Type":"ContainerDied","Data":"c4b4661e54a2bb624210769d0525ad4714a96ee041a8c61ffe733dc4d7e12cd7"}
Jul 27 21:40:49.149858 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:40:49.149778    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-b8ngh" event={"ID":"9624b8a0-6314-4ed2-ac50-ea171bb73e45","Type":"ContainerDied","Data":"451e5f60ae33fd58ed8e69890a13bacda53abcf6372764015f5d726043d5b3f5"}
Jul 27 21:40:49.149858 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:40:49.149817    2559 scope.go:117] "RemoveContainer" containerID="42eef4c0be333017c78d233bd93a18c72ae712741a6c91cee729884107a85bca"
Jul 27 21:40:49.161398 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:40:49.161373    2559 scope.go:117] "RemoveContainer" containerID="c4b4661e54a2bb624210769d0525ad4714a96ee041a8c61ffe733dc4d7e12cd7"
Jul 27 21:40:49.168690 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:40:49.168670    2559 scope.go:117] "RemoveContainer" containerID="b70c81686b15b7ea8d1e5c298b91b0345c139f4401ed4b13f8b3e194fb4070fc"
Jul 27 21:40:49.175676 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:40:49.175661    2559 scope.go:117] "RemoveContainer" containerID="42eef4c0be333017c78d233bd93a18c72ae712741a6c91cee729884107a85bca"
Jul 27 21:40:49.175935 ip-10-0-131-133 kubenswrapper[2559]: E0727 21:40:49.175917    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"42eef4c0be333017c78d233bd93a18c72ae712741a6c91cee729884107a85bca\": container with ID starting with 42eef4c0be333017c78d233bd93a18c72ae712741a6c91cee729884107a85bca not found: ID does not exist" containerID="42eef4c0be333017c78d233bd93a18c72ae712741a6c91cee729884107a85bca"
Jul 27 21:40:49.175975 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:40:49.175943    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"42eef4c0be333017c78d233bd93a18c72ae712741a6c91cee729884107a85bca"} err="failed to get container status \"42eef4c0be333017c78d233bd93a18c72ae712741a6c91cee729884107a85bca\": rpc error: code = NotFound desc = could not find container \"42eef4c0be333017c78d233bd93a18c72ae712741a6c91cee729884107a85bca\": container with ID starting with 42eef4c0be333017c78d233bd93a18c72ae712741a6c91cee729884107a85bca not found: ID does not exist"
Jul 27 21:40:49.175975 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:40:49.175961    2559 scope.go:117] "RemoveContainer" containerID="c4b4661e54a2bb624210769d0525ad4714a96ee041a8c61ffe733dc4d7e12cd7"
Jul 27 21:40:49.176187 ip-10-0-131-133 kubenswrapper[2559]: E0727 21:40:49.176169    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c4b4661e54a2bb624210769d0525ad4714a96ee041a8c61ffe733dc4d7e12cd7\": container with ID starting with c4b4661e54a2bb624210769d0525ad4714a96ee041a8c61ffe733dc4d7e12cd7 not found: ID does not exist" containerID="c4b4661e54a2bb624210769d0525ad4714a96ee041a8c61ffe733dc4d7e12cd7"
Jul 27 21:40:49.176273 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:40:49.176194    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c4b4661e54a2bb624210769d0525ad4714a96ee041a8c61ffe733dc4d7e12cd7"} err="failed to get container status \"c4b4661e54a2bb624210769d0525ad4714a96ee041a8c61ffe733dc4d7e12cd7\": rpc error: code = NotFound desc = could not find container \"c4b4661e54a2bb624210769d0525ad4714a96ee041a8c61ffe733dc4d7e12cd7\": container with ID starting with c4b4661e54a2bb624210769d0525ad4714a96ee041a8c61ffe733dc4d7e12cd7 not found: ID does not exist"
Jul 27 21:40:49.176273 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:40:49.176209    2559 scope.go:117] "RemoveContainer" containerID="b70c81686b15b7ea8d1e5c298b91b0345c139f4401ed4b13f8b3e194fb4070fc"
Jul 27 21:40:49.176448 ip-10-0-131-133 kubenswrapper[2559]: E0727 21:40:49.176433    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b70c81686b15b7ea8d1e5c298b91b0345c139f4401ed4b13f8b3e194fb4070fc\": container with ID starting with b70c81686b15b7ea8d1e5c298b91b0345c139f4401ed4b13f8b3e194fb4070fc not found: ID does not exist" containerID="b70c81686b15b7ea8d1e5c298b91b0345c139f4401ed4b13f8b3e194fb4070fc"
Jul 27 21:40:49.176496 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:40:49.176451    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b70c81686b15b7ea8d1e5c298b91b0345c139f4401ed4b13f8b3e194fb4070fc"} err="failed to get container status \"b70c81686b15b7ea8d1e5c298b91b0345c139f4401ed4b13f8b3e194fb4070fc\": rpc error: code = NotFound desc = could not find container \"b70c81686b15b7ea8d1e5c298b91b0345c139f4401ed4b13f8b3e194fb4070fc\": container with ID starting with b70c81686b15b7ea8d1e5c298b91b0345c139f4401ed4b13f8b3e194fb4070fc not found: ID does not exist"
Jul 27 21:40:49.188340 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:40:49.188319    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-b8ngh"]
Jul 27 21:40:49.191933 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:40:49.191916    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-b8ngh"]
Jul 27 21:40:49.240544 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:40:49.240514    2559 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9624b8a0-6314-4ed2-ac50-ea171bb73e45-kserve-provision-location\") on node \"ip-10-0-131-133.ec2.internal\" DevicePath \"\""
Jul 27 21:40:49.240544 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:40:49.240542    2559 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9624b8a0-6314-4ed2-ac50-ea171bb73e45-cabundle-cert\") on node \"ip-10-0-131-133.ec2.internal\" DevicePath \"\""
Jul 27 21:40:49.240701 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:40:49.240558    2559 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9624b8a0-6314-4ed2-ac50-ea171bb73e45-kube-rbac-proxy-sar-config\") on node \"ip-10-0-131-133.ec2.internal\" DevicePath \"\""
Jul 27 21:40:49.240701 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:40:49.240570    2559 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9624b8a0-6314-4ed2-ac50-ea171bb73e45-proxy-tls\") on node \"ip-10-0-131-133.ec2.internal\" DevicePath \"\""
Jul 27 21:40:49.240701 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:40:49.240582    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hfnqh\" (UniqueName: \"kubernetes.io/projected/9624b8a0-6314-4ed2-ac50-ea171bb73e45-kube-api-access-hfnqh\") on node \"ip-10-0-131-133.ec2.internal\" DevicePath \"\""
Jul 27 21:40:49.622385 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:40:49.622301    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-86dc9fdb65-m4b6h"]
Jul 27 21:40:49.622739 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:40:49.622721    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9624b8a0-6314-4ed2-ac50-ea171bb73e45" containerName="storage-initializer"
Jul 27 21:40:49.622824 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:40:49.622742    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="9624b8a0-6314-4ed2-ac50-ea171bb73e45" containerName="storage-initializer"
Jul 27 21:40:49.622824 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:40:49.622759    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9624b8a0-6314-4ed2-ac50-ea171bb73e45" containerName="kube-rbac-proxy"
Jul 27 21:40:49.622824 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:40:49.622767    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="9624b8a0-6314-4ed2-ac50-ea171bb73e45" containerName="kube-rbac-proxy"
Jul 27 21:40:49.622824 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:40:49.622784    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9beaa19c-28c7-4507-8277-ba8a01131e38" containerName="kube-rbac-proxy"
Jul 27 21:40:49.622824 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:40:49.622793    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="9beaa19c-28c7-4507-8277-ba8a01131e38" containerName="kube-rbac-proxy"
Jul 27 21:40:49.622824 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:40:49.622808    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9beaa19c-28c7-4507-8277-ba8a01131e38" containerName="kserve-container"
Jul 27 21:40:49.622824 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:40:49.622816    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="9beaa19c-28c7-4507-8277-ba8a01131e38" containerName="kserve-container"
Jul 27 21:40:49.623155 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:40:49.622829    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9beaa19c-28c7-4507-8277-ba8a01131e38" containerName="storage-initializer"
Jul 27 21:40:49.623155 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:40:49.622837    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="9beaa19c-28c7-4507-8277-ba8a01131e38" containerName="storage-initializer"
Jul 27 21:40:49.623155 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:40:49.622858    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9624b8a0-6314-4ed2-ac50-ea171bb73e45" containerName="kserve-container"
Jul 27 21:40:49.623155 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:40:49.622867    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="9624b8a0-6314-4ed2-ac50-ea171bb73e45" containerName="kserve-container"
Jul 27 21:40:49.623155 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:40:49.622952    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="9beaa19c-28c7-4507-8277-ba8a01131e38" containerName="kube-rbac-proxy"
Jul 27 21:40:49.623155 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:40:49.622965    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="9624b8a0-6314-4ed2-ac50-ea171bb73e45" containerName="kube-rbac-proxy"
Jul 27 21:40:49.623155 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:40:49.622975    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="9beaa19c-28c7-4507-8277-ba8a01131e38" containerName="kserve-container"
Jul 27 21:40:49.623155 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:40:49.622988    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="9624b8a0-6314-4ed2-ac50-ea171bb73e45" containerName="kserve-container"
Jul 27 21:40:49.627890 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:40:49.627870    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-86dc9fdb65-m4b6h"
Jul 27 21:40:49.630458 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:40:49.630436    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 27 21:40:49.630594 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:40:49.630567    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-stop-kube-rbac-proxy-sar-config\""
Jul 27 21:40:49.631111 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:40:49.631090    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-v7c66\""
Jul 27 21:40:49.631416 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:40:49.631144    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 27 21:40:49.631581 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:40:49.631455    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-stop-predictor-serving-cert\""
Jul 27 21:40:49.631581 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:40:49.631464    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 27 21:40:49.631581 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:40:49.631459    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 27 21:40:49.640023 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:40:49.640002    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-86dc9fdb65-m4b6h"]
Jul 27 21:40:49.743970 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:40:49.743927    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/68d2f1f7-ae21-44ae-b207-c18a4adaea28-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-predictor-86dc9fdb65-m4b6h\" (UID: \"68d2f1f7-ae21-44ae-b207-c18a4adaea28\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-86dc9fdb65-m4b6h"
Jul 27 21:40:49.744160 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:40:49.743982    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m4h4w\" (UniqueName: \"kubernetes.io/projected/68d2f1f7-ae21-44ae-b207-c18a4adaea28-kube-api-access-m4h4w\") pod \"isvc-canary-stop-predictor-86dc9fdb65-m4b6h\" (UID: \"68d2f1f7-ae21-44ae-b207-c18a4adaea28\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-86dc9fdb65-m4b6h"
Jul 27 21:40:49.744160 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:40:49.744007    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/68d2f1f7-ae21-44ae-b207-c18a4adaea28-kserve-provision-location\") pod \"isvc-canary-stop-predictor-86dc9fdb65-m4b6h\" (UID: \"68d2f1f7-ae21-44ae-b207-c18a4adaea28\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-86dc9fdb65-m4b6h"
Jul 27 21:40:49.744160 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:40:49.744031    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/68d2f1f7-ae21-44ae-b207-c18a4adaea28-cabundle-cert\") pod \"isvc-canary-stop-predictor-86dc9fdb65-m4b6h\" (UID: \"68d2f1f7-ae21-44ae-b207-c18a4adaea28\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-86dc9fdb65-m4b6h"
Jul 27 21:40:49.744160 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:40:49.744096    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/68d2f1f7-ae21-44ae-b207-c18a4adaea28-proxy-tls\") pod \"isvc-canary-stop-predictor-86dc9fdb65-m4b6h\" (UID: \"68d2f1f7-ae21-44ae-b207-c18a4adaea28\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-86dc9fdb65-m4b6h"
Jul 27 21:40:49.844712 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:40:49.844679    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/68d2f1f7-ae21-44ae-b207-c18a4adaea28-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-predictor-86dc9fdb65-m4b6h\" (UID: \"68d2f1f7-ae21-44ae-b207-c18a4adaea28\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-86dc9fdb65-m4b6h"
Jul 27 21:40:49.844907 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:40:49.844726    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m4h4w\" (UniqueName: \"kubernetes.io/projected/68d2f1f7-ae21-44ae-b207-c18a4adaea28-kube-api-access-m4h4w\") pod \"isvc-canary-stop-predictor-86dc9fdb65-m4b6h\" (UID: \"68d2f1f7-ae21-44ae-b207-c18a4adaea28\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-86dc9fdb65-m4b6h"
Jul 27 21:40:49.844907 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:40:49.844749    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/68d2f1f7-ae21-44ae-b207-c18a4adaea28-kserve-provision-location\") pod \"isvc-canary-stop-predictor-86dc9fdb65-m4b6h\" (UID: \"68d2f1f7-ae21-44ae-b207-c18a4adaea28\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-86dc9fdb65-m4b6h"
Jul 27 21:40:49.844907 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:40:49.844777    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/68d2f1f7-ae21-44ae-b207-c18a4adaea28-cabundle-cert\") pod \"isvc-canary-stop-predictor-86dc9fdb65-m4b6h\" (UID: \"68d2f1f7-ae21-44ae-b207-c18a4adaea28\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-86dc9fdb65-m4b6h"
Jul 27 21:40:49.844907 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:40:49.844821    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/68d2f1f7-ae21-44ae-b207-c18a4adaea28-proxy-tls\") pod \"isvc-canary-stop-predictor-86dc9fdb65-m4b6h\" (UID: \"68d2f1f7-ae21-44ae-b207-c18a4adaea28\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-86dc9fdb65-m4b6h"
Jul 27 21:40:49.845124 ip-10-0-131-133 kubenswrapper[2559]: E0727 21:40:49.844951    2559 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-canary-stop-predictor-serving-cert: secret "isvc-canary-stop-predictor-serving-cert" not found
Jul 27 21:40:49.845124 ip-10-0-131-133 kubenswrapper[2559]: E0727 21:40:49.845007    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/68d2f1f7-ae21-44ae-b207-c18a4adaea28-proxy-tls podName:68d2f1f7-ae21-44ae-b207-c18a4adaea28 nodeName:}" failed. No retries permitted until 2026-07-27 21:40:50.344989091 +0000 UTC m=+2178.924173084 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/68d2f1f7-ae21-44ae-b207-c18a4adaea28-proxy-tls") pod "isvc-canary-stop-predictor-86dc9fdb65-m4b6h" (UID: "68d2f1f7-ae21-44ae-b207-c18a4adaea28") : secret "isvc-canary-stop-predictor-serving-cert" not found
Jul 27 21:40:49.845270 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:40:49.845133    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/68d2f1f7-ae21-44ae-b207-c18a4adaea28-kserve-provision-location\") pod \"isvc-canary-stop-predictor-86dc9fdb65-m4b6h\" (UID: \"68d2f1f7-ae21-44ae-b207-c18a4adaea28\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-86dc9fdb65-m4b6h"
Jul 27 21:40:49.845480 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:40:49.845461    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/68d2f1f7-ae21-44ae-b207-c18a4adaea28-cabundle-cert\") pod \"isvc-canary-stop-predictor-86dc9fdb65-m4b6h\" (UID: \"68d2f1f7-ae21-44ae-b207-c18a4adaea28\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-86dc9fdb65-m4b6h"
Jul 27 21:40:49.845553 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:40:49.845493    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/68d2f1f7-ae21-44ae-b207-c18a4adaea28-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-predictor-86dc9fdb65-m4b6h\" (UID: \"68d2f1f7-ae21-44ae-b207-c18a4adaea28\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-86dc9fdb65-m4b6h"
Jul 27 21:40:49.853508 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:40:49.853487    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m4h4w\" (UniqueName: \"kubernetes.io/projected/68d2f1f7-ae21-44ae-b207-c18a4adaea28-kube-api-access-m4h4w\") pod \"isvc-canary-stop-predictor-86dc9fdb65-m4b6h\" (UID: \"68d2f1f7-ae21-44ae-b207-c18a4adaea28\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-86dc9fdb65-m4b6h"
Jul 27 21:40:49.981468 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:40:49.981431    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9624b8a0-6314-4ed2-ac50-ea171bb73e45" path="/var/lib/kubelet/pods/9624b8a0-6314-4ed2-ac50-ea171bb73e45/volumes"
Jul 27 21:40:50.350465 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:40:50.350383    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/68d2f1f7-ae21-44ae-b207-c18a4adaea28-proxy-tls\") pod \"isvc-canary-stop-predictor-86dc9fdb65-m4b6h\" (UID: \"68d2f1f7-ae21-44ae-b207-c18a4adaea28\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-86dc9fdb65-m4b6h"
Jul 27 21:40:50.352783 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:40:50.352765    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/68d2f1f7-ae21-44ae-b207-c18a4adaea28-proxy-tls\") pod \"isvc-canary-stop-predictor-86dc9fdb65-m4b6h\" (UID: \"68d2f1f7-ae21-44ae-b207-c18a4adaea28\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-86dc9fdb65-m4b6h"
Jul 27 21:40:50.538010 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:40:50.537969    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-86dc9fdb65-m4b6h"
Jul 27 21:40:50.665690 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:40:50.665661    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-86dc9fdb65-m4b6h"]
Jul 27 21:40:50.668289 ip-10-0-131-133 kubenswrapper[2559]: W0727 21:40:50.668264    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod68d2f1f7_ae21_44ae_b207_c18a4adaea28.slice/crio-05d58ea40b59c7078d49f2d4e77be78d4cc526bca280066aab336995b0a6ea83 WatchSource:0}: Error finding container 05d58ea40b59c7078d49f2d4e77be78d4cc526bca280066aab336995b0a6ea83: Status 404 returned error can't find the container with id 05d58ea40b59c7078d49f2d4e77be78d4cc526bca280066aab336995b0a6ea83
Jul 27 21:40:51.162246 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:40:51.162196    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-86dc9fdb65-m4b6h" event={"ID":"68d2f1f7-ae21-44ae-b207-c18a4adaea28","Type":"ContainerStarted","Data":"65d28181321705488c58e831f70ba0e17e4e5cf1408d2826f1c7d79b6f509caf"}
Jul 27 21:40:51.162417 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:40:51.162275    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-86dc9fdb65-m4b6h" event={"ID":"68d2f1f7-ae21-44ae-b207-c18a4adaea28","Type":"ContainerStarted","Data":"05d58ea40b59c7078d49f2d4e77be78d4cc526bca280066aab336995b0a6ea83"}
Jul 27 21:40:55.182519 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:40:55.182484    2559 generic.go:356] "Generic (PLEG): container finished" podID="68d2f1f7-ae21-44ae-b207-c18a4adaea28" containerID="65d28181321705488c58e831f70ba0e17e4e5cf1408d2826f1c7d79b6f509caf" exitCode=0
Jul 27 21:40:55.182953 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:40:55.182528    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-86dc9fdb65-m4b6h" event={"ID":"68d2f1f7-ae21-44ae-b207-c18a4adaea28","Type":"ContainerDied","Data":"65d28181321705488c58e831f70ba0e17e4e5cf1408d2826f1c7d79b6f509caf"}
Jul 27 21:40:56.189283 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:40:56.189242    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-86dc9fdb65-m4b6h" event={"ID":"68d2f1f7-ae21-44ae-b207-c18a4adaea28","Type":"ContainerStarted","Data":"ac82f10f730ff892de45c75428e399183e3fe72e6f9b7b3cd356f307a1a2e21a"}
Jul 27 21:40:56.189668 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:40:56.189308    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-86dc9fdb65-m4b6h" event={"ID":"68d2f1f7-ae21-44ae-b207-c18a4adaea28","Type":"ContainerStarted","Data":"81d5d8df8a7398c8b18685dd0af89d9e04fe2b112c114c8147e5bae3e2290c4a"}
Jul 27 21:40:56.189668 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:40:56.189622    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-86dc9fdb65-m4b6h"
Jul 27 21:40:56.189770 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:40:56.189750    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-86dc9fdb65-m4b6h"
Jul 27 21:40:56.191153 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:40:56.191127    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-86dc9fdb65-m4b6h" podUID="68d2f1f7-ae21-44ae-b207-c18a4adaea28" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.39:8080: connect: connection refused"
Jul 27 21:40:56.207986 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:40:56.207945    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-86dc9fdb65-m4b6h" podStartSLOduration=3.45146485 podStartE2EDuration="7.207932051s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.756467201s" isStatefulPod=false podCreationTimestamp="2026-07-27 21:40:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 21:40:56.2064321 +0000 UTC m=+2184.785616107" watchObservedRunningTime="2026-07-27 21:40:56.207932051 +0000 UTC m=+2184.787116057"
Jul 27 21:40:57.194027 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:40:57.193987    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-86dc9fdb65-m4b6h" podUID="68d2f1f7-ae21-44ae-b207-c18a4adaea28" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.39:8080: connect: connection refused"
Jul 27 21:41:02.199067 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:41:02.199030    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-86dc9fdb65-m4b6h"
Jul 27 21:41:02.199632 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:41:02.199607    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-86dc9fdb65-m4b6h" podUID="68d2f1f7-ae21-44ae-b207-c18a4adaea28" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.39:8080: connect: connection refused"
Jul 27 21:41:12.199752 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:41:12.199712    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-86dc9fdb65-m4b6h" podUID="68d2f1f7-ae21-44ae-b207-c18a4adaea28" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.39:8080: connect: connection refused"
Jul 27 21:41:22.199730 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:41:22.199687    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-86dc9fdb65-m4b6h" podUID="68d2f1f7-ae21-44ae-b207-c18a4adaea28" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.39:8080: connect: connection refused"
Jul 27 21:41:32.199986 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:41:32.199946    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-86dc9fdb65-m4b6h" podUID="68d2f1f7-ae21-44ae-b207-c18a4adaea28" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.39:8080: connect: connection refused"
Jul 27 21:41:42.199960 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:41:42.199918    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-86dc9fdb65-m4b6h" podUID="68d2f1f7-ae21-44ae-b207-c18a4adaea28" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.39:8080: connect: connection refused"
Jul 27 21:41:52.200191 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:41:52.200149    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-86dc9fdb65-m4b6h" podUID="68d2f1f7-ae21-44ae-b207-c18a4adaea28" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.39:8080: connect: connection refused"
Jul 27 21:42:02.199952 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:42:02.199904    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-86dc9fdb65-m4b6h" podUID="68d2f1f7-ae21-44ae-b207-c18a4adaea28" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.39:8080: connect: connection refused"
Jul 27 21:42:03.976982 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:42:03.976926    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-86dc9fdb65-m4b6h" podUID="68d2f1f7-ae21-44ae-b207-c18a4adaea28" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.39:8080: connect: connection refused"
Jul 27 21:42:13.981775 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:42:13.981738    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-86dc9fdb65-m4b6h"
Jul 27 21:42:19.606382 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:42:19.606285    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-86dc9fdb65-m4b6h"]
Jul 27 21:42:19.606904 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:42:19.606768    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-86dc9fdb65-m4b6h" podUID="68d2f1f7-ae21-44ae-b207-c18a4adaea28" containerName="kserve-container" containerID="cri-o://81d5d8df8a7398c8b18685dd0af89d9e04fe2b112c114c8147e5bae3e2290c4a" gracePeriod=30
Jul 27 21:42:19.607259 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:42:19.607206    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-86dc9fdb65-m4b6h" podUID="68d2f1f7-ae21-44ae-b207-c18a4adaea28" containerName="kube-rbac-proxy" containerID="cri-o://ac82f10f730ff892de45c75428e399183e3fe72e6f9b7b3cd356f307a1a2e21a" gracePeriod=30
Jul 27 21:42:20.585123 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:42:20.585090    2559 generic.go:356] "Generic (PLEG): container finished" podID="68d2f1f7-ae21-44ae-b207-c18a4adaea28" containerID="ac82f10f730ff892de45c75428e399183e3fe72e6f9b7b3cd356f307a1a2e21a" exitCode=2
Jul 27 21:42:20.585333 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:42:20.585141    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-86dc9fdb65-m4b6h" event={"ID":"68d2f1f7-ae21-44ae-b207-c18a4adaea28","Type":"ContainerDied","Data":"ac82f10f730ff892de45c75428e399183e3fe72e6f9b7b3cd356f307a1a2e21a"}
Jul 27 21:42:22.194553 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:42:22.194508    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-86dc9fdb65-m4b6h" podUID="68d2f1f7-ae21-44ae-b207-c18a4adaea28" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.39:8643/healthz\": dial tcp 10.132.0.39:8643: connect: connection refused"
Jul 27 21:42:23.977613 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:42:23.977561    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-86dc9fdb65-m4b6h" podUID="68d2f1f7-ae21-44ae-b207-c18a4adaea28" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.39:8080: connect: connection refused"
Jul 27 21:42:24.564752 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:42:24.564725    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-86dc9fdb65-m4b6h"
Jul 27 21:42:24.606487 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:42:24.606292    2559 generic.go:356] "Generic (PLEG): container finished" podID="68d2f1f7-ae21-44ae-b207-c18a4adaea28" containerID="81d5d8df8a7398c8b18685dd0af89d9e04fe2b112c114c8147e5bae3e2290c4a" exitCode=0
Jul 27 21:42:24.606487 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:42:24.606350    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-86dc9fdb65-m4b6h" event={"ID":"68d2f1f7-ae21-44ae-b207-c18a4adaea28","Type":"ContainerDied","Data":"81d5d8df8a7398c8b18685dd0af89d9e04fe2b112c114c8147e5bae3e2290c4a"}
Jul 27 21:42:24.606487 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:42:24.606372    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-86dc9fdb65-m4b6h"
Jul 27 21:42:24.606487 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:42:24.606438    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-86dc9fdb65-m4b6h" event={"ID":"68d2f1f7-ae21-44ae-b207-c18a4adaea28","Type":"ContainerDied","Data":"05d58ea40b59c7078d49f2d4e77be78d4cc526bca280066aab336995b0a6ea83"}
Jul 27 21:42:24.606487 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:42:24.606477    2559 scope.go:117] "RemoveContainer" containerID="ac82f10f730ff892de45c75428e399183e3fe72e6f9b7b3cd356f307a1a2e21a"
Jul 27 21:42:24.615081 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:42:24.615056    2559 scope.go:117] "RemoveContainer" containerID="81d5d8df8a7398c8b18685dd0af89d9e04fe2b112c114c8147e5bae3e2290c4a"
Jul 27 21:42:24.623291 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:42:24.623269    2559 scope.go:117] "RemoveContainer" containerID="65d28181321705488c58e831f70ba0e17e4e5cf1408d2826f1c7d79b6f509caf"
Jul 27 21:42:24.631388 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:42:24.631363    2559 scope.go:117] "RemoveContainer" containerID="ac82f10f730ff892de45c75428e399183e3fe72e6f9b7b3cd356f307a1a2e21a"
Jul 27 21:42:24.631697 ip-10-0-131-133 kubenswrapper[2559]: E0727 21:42:24.631679    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ac82f10f730ff892de45c75428e399183e3fe72e6f9b7b3cd356f307a1a2e21a\": container with ID starting with ac82f10f730ff892de45c75428e399183e3fe72e6f9b7b3cd356f307a1a2e21a not found: ID does not exist" containerID="ac82f10f730ff892de45c75428e399183e3fe72e6f9b7b3cd356f307a1a2e21a"
Jul 27 21:42:24.631755 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:42:24.631707    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ac82f10f730ff892de45c75428e399183e3fe72e6f9b7b3cd356f307a1a2e21a"} err="failed to get container status \"ac82f10f730ff892de45c75428e399183e3fe72e6f9b7b3cd356f307a1a2e21a\": rpc error: code = NotFound desc = could not find container \"ac82f10f730ff892de45c75428e399183e3fe72e6f9b7b3cd356f307a1a2e21a\": container with ID starting with ac82f10f730ff892de45c75428e399183e3fe72e6f9b7b3cd356f307a1a2e21a not found: ID does not exist"
Jul 27 21:42:24.631755 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:42:24.631725    2559 scope.go:117] "RemoveContainer" containerID="81d5d8df8a7398c8b18685dd0af89d9e04fe2b112c114c8147e5bae3e2290c4a"
Jul 27 21:42:24.631958 ip-10-0-131-133 kubenswrapper[2559]: E0727 21:42:24.631938    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"81d5d8df8a7398c8b18685dd0af89d9e04fe2b112c114c8147e5bae3e2290c4a\": container with ID starting with 81d5d8df8a7398c8b18685dd0af89d9e04fe2b112c114c8147e5bae3e2290c4a not found: ID does not exist" containerID="81d5d8df8a7398c8b18685dd0af89d9e04fe2b112c114c8147e5bae3e2290c4a"
Jul 27 21:42:24.632105 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:42:24.631965    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"81d5d8df8a7398c8b18685dd0af89d9e04fe2b112c114c8147e5bae3e2290c4a"} err="failed to get container status \"81d5d8df8a7398c8b18685dd0af89d9e04fe2b112c114c8147e5bae3e2290c4a\": rpc error: code = NotFound desc = could not find container \"81d5d8df8a7398c8b18685dd0af89d9e04fe2b112c114c8147e5bae3e2290c4a\": container with ID starting with 81d5d8df8a7398c8b18685dd0af89d9e04fe2b112c114c8147e5bae3e2290c4a not found: ID does not exist"
Jul 27 21:42:24.632105 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:42:24.631982    2559 scope.go:117] "RemoveContainer" containerID="65d28181321705488c58e831f70ba0e17e4e5cf1408d2826f1c7d79b6f509caf"
Jul 27 21:42:24.632258 ip-10-0-131-133 kubenswrapper[2559]: E0727 21:42:24.632242    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"65d28181321705488c58e831f70ba0e17e4e5cf1408d2826f1c7d79b6f509caf\": container with ID starting with 65d28181321705488c58e831f70ba0e17e4e5cf1408d2826f1c7d79b6f509caf not found: ID does not exist" containerID="65d28181321705488c58e831f70ba0e17e4e5cf1408d2826f1c7d79b6f509caf"
Jul 27 21:42:24.632305 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:42:24.632263    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"65d28181321705488c58e831f70ba0e17e4e5cf1408d2826f1c7d79b6f509caf"} err="failed to get container status \"65d28181321705488c58e831f70ba0e17e4e5cf1408d2826f1c7d79b6f509caf\": rpc error: code = NotFound desc = could not find container \"65d28181321705488c58e831f70ba0e17e4e5cf1408d2826f1c7d79b6f509caf\": container with ID starting with 65d28181321705488c58e831f70ba0e17e4e5cf1408d2826f1c7d79b6f509caf not found: ID does not exist"
Jul 27 21:42:24.690502 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:42:24.690459    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/68d2f1f7-ae21-44ae-b207-c18a4adaea28-kserve-provision-location\") pod \"68d2f1f7-ae21-44ae-b207-c18a4adaea28\" (UID: \"68d2f1f7-ae21-44ae-b207-c18a4adaea28\") "
Jul 27 21:42:24.690684 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:42:24.690517    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/68d2f1f7-ae21-44ae-b207-c18a4adaea28-proxy-tls\") pod \"68d2f1f7-ae21-44ae-b207-c18a4adaea28\" (UID: \"68d2f1f7-ae21-44ae-b207-c18a4adaea28\") "
Jul 27 21:42:24.690684 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:42:24.690581    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/68d2f1f7-ae21-44ae-b207-c18a4adaea28-cabundle-cert\") pod \"68d2f1f7-ae21-44ae-b207-c18a4adaea28\" (UID: \"68d2f1f7-ae21-44ae-b207-c18a4adaea28\") "
Jul 27 21:42:24.690684 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:42:24.690633    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/68d2f1f7-ae21-44ae-b207-c18a4adaea28-kube-rbac-proxy-sar-config\") pod \"68d2f1f7-ae21-44ae-b207-c18a4adaea28\" (UID: \"68d2f1f7-ae21-44ae-b207-c18a4adaea28\") "
Jul 27 21:42:24.690684 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:42:24.690659    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-m4h4w\" (UniqueName: \"kubernetes.io/projected/68d2f1f7-ae21-44ae-b207-c18a4adaea28-kube-api-access-m4h4w\") pod \"68d2f1f7-ae21-44ae-b207-c18a4adaea28\" (UID: \"68d2f1f7-ae21-44ae-b207-c18a4adaea28\") "
Jul 27 21:42:24.690973 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:42:24.690866    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/68d2f1f7-ae21-44ae-b207-c18a4adaea28-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "68d2f1f7-ae21-44ae-b207-c18a4adaea28" (UID: "68d2f1f7-ae21-44ae-b207-c18a4adaea28"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 21:42:24.691102 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:42:24.691044    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/68d2f1f7-ae21-44ae-b207-c18a4adaea28-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "68d2f1f7-ae21-44ae-b207-c18a4adaea28" (UID: "68d2f1f7-ae21-44ae-b207-c18a4adaea28"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 21:42:24.691102 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:42:24.691050    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/68d2f1f7-ae21-44ae-b207-c18a4adaea28-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "68d2f1f7-ae21-44ae-b207-c18a4adaea28" (UID: "68d2f1f7-ae21-44ae-b207-c18a4adaea28"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 21:42:24.692733 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:42:24.692700    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/68d2f1f7-ae21-44ae-b207-c18a4adaea28-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "68d2f1f7-ae21-44ae-b207-c18a4adaea28" (UID: "68d2f1f7-ae21-44ae-b207-c18a4adaea28"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 21:42:24.692978 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:42:24.692953    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/68d2f1f7-ae21-44ae-b207-c18a4adaea28-kube-api-access-m4h4w" (OuterVolumeSpecName: "kube-api-access-m4h4w") pod "68d2f1f7-ae21-44ae-b207-c18a4adaea28" (UID: "68d2f1f7-ae21-44ae-b207-c18a4adaea28"). InnerVolumeSpecName "kube-api-access-m4h4w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 21:42:24.791794 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:42:24.791754    2559 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/68d2f1f7-ae21-44ae-b207-c18a4adaea28-kube-rbac-proxy-sar-config\") on node \"ip-10-0-131-133.ec2.internal\" DevicePath \"\""
Jul 27 21:42:24.791794 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:42:24.791787    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m4h4w\" (UniqueName: \"kubernetes.io/projected/68d2f1f7-ae21-44ae-b207-c18a4adaea28-kube-api-access-m4h4w\") on node \"ip-10-0-131-133.ec2.internal\" DevicePath \"\""
Jul 27 21:42:24.791794 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:42:24.791797    2559 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/68d2f1f7-ae21-44ae-b207-c18a4adaea28-kserve-provision-location\") on node \"ip-10-0-131-133.ec2.internal\" DevicePath \"\""
Jul 27 21:42:24.791794 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:42:24.791806    2559 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/68d2f1f7-ae21-44ae-b207-c18a4adaea28-proxy-tls\") on node \"ip-10-0-131-133.ec2.internal\" DevicePath \"\""
Jul 27 21:42:24.792067 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:42:24.791815    2559 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/68d2f1f7-ae21-44ae-b207-c18a4adaea28-cabundle-cert\") on node \"ip-10-0-131-133.ec2.internal\" DevicePath \"\""
Jul 27 21:42:24.927975 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:42:24.927935    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-86dc9fdb65-m4b6h"]
Jul 27 21:42:24.931915 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:42:24.931885    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-86dc9fdb65-m4b6h"]
Jul 27 21:42:25.982416 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:42:25.982383    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="68d2f1f7-ae21-44ae-b207-c18a4adaea28" path="/var/lib/kubelet/pods/68d2f1f7-ae21-44ae-b207-c18a4adaea28/volumes"
Jul 27 21:42:29.918637 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:42:29.918599    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-h6rrk"]
Jul 27 21:42:29.919029 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:42:29.918981    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="68d2f1f7-ae21-44ae-b207-c18a4adaea28" containerName="storage-initializer"
Jul 27 21:42:29.919029 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:42:29.918993    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="68d2f1f7-ae21-44ae-b207-c18a4adaea28" containerName="storage-initializer"
Jul 27 21:42:29.919029 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:42:29.919009    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="68d2f1f7-ae21-44ae-b207-c18a4adaea28" containerName="kserve-container"
Jul 27 21:42:29.919029 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:42:29.919015    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="68d2f1f7-ae21-44ae-b207-c18a4adaea28" containerName="kserve-container"
Jul 27 21:42:29.919029 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:42:29.919023    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="68d2f1f7-ae21-44ae-b207-c18a4adaea28" containerName="kube-rbac-proxy"
Jul 27 21:42:29.919029 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:42:29.919029    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="68d2f1f7-ae21-44ae-b207-c18a4adaea28" containerName="kube-rbac-proxy"
Jul 27 21:42:29.919238 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:42:29.919086    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="68d2f1f7-ae21-44ae-b207-c18a4adaea28" containerName="kube-rbac-proxy"
Jul 27 21:42:29.919238 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:42:29.919094    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="68d2f1f7-ae21-44ae-b207-c18a4adaea28" containerName="kserve-container"
Jul 27 21:42:29.923785 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:42:29.923763    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-h6rrk"
Jul 27 21:42:29.928170 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:42:29.928150    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-lightgbm-predictor-serving-cert\""
Jul 27 21:42:29.928439 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:42:29.928417    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 27 21:42:29.928526 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:42:29.928469    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 27 21:42:29.928526 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:42:29.928482    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-v7c66\""
Jul 27 21:42:29.928884 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:42:29.928865    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-lightgbm-kube-rbac-proxy-sar-config\""
Jul 27 21:42:29.929007 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:42:29.928949    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 27 21:42:29.929250 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:42:29.929208    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 27 21:42:29.940633 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:42:29.940605    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-h6rrk"]
Jul 27 21:42:30.038782 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:42:30.038719    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/62c1b0c8-63e5-4109-bb7d-aab7d848b982-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-predictor-54f85ddf88-h6rrk\" (UID: \"62c1b0c8-63e5-4109-bb7d-aab7d848b982\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-h6rrk"
Jul 27 21:42:30.038782 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:42:30.038794    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/62c1b0c8-63e5-4109-bb7d-aab7d848b982-kserve-provision-location\") pod \"isvc-lightgbm-predictor-54f85ddf88-h6rrk\" (UID: \"62c1b0c8-63e5-4109-bb7d-aab7d848b982\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-h6rrk"
Jul 27 21:42:30.039025 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:42:30.038849    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/62c1b0c8-63e5-4109-bb7d-aab7d848b982-cabundle-cert\") pod \"isvc-lightgbm-predictor-54f85ddf88-h6rrk\" (UID: \"62c1b0c8-63e5-4109-bb7d-aab7d848b982\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-h6rrk"
Jul 27 21:42:30.039025 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:42:30.038893    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/62c1b0c8-63e5-4109-bb7d-aab7d848b982-proxy-tls\") pod \"isvc-lightgbm-predictor-54f85ddf88-h6rrk\" (UID: \"62c1b0c8-63e5-4109-bb7d-aab7d848b982\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-h6rrk"
Jul 27 21:42:30.039025 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:42:30.038963    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v5rv5\" (UniqueName: \"kubernetes.io/projected/62c1b0c8-63e5-4109-bb7d-aab7d848b982-kube-api-access-v5rv5\") pod \"isvc-lightgbm-predictor-54f85ddf88-h6rrk\" (UID: \"62c1b0c8-63e5-4109-bb7d-aab7d848b982\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-h6rrk"
Jul 27 21:42:30.140331 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:42:30.140295    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/62c1b0c8-63e5-4109-bb7d-aab7d848b982-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-predictor-54f85ddf88-h6rrk\" (UID: \"62c1b0c8-63e5-4109-bb7d-aab7d848b982\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-h6rrk"
Jul 27 21:42:30.140521 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:42:30.140342    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/62c1b0c8-63e5-4109-bb7d-aab7d848b982-kserve-provision-location\") pod \"isvc-lightgbm-predictor-54f85ddf88-h6rrk\" (UID: \"62c1b0c8-63e5-4109-bb7d-aab7d848b982\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-h6rrk"
Jul 27 21:42:30.140521 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:42:30.140372    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/62c1b0c8-63e5-4109-bb7d-aab7d848b982-cabundle-cert\") pod \"isvc-lightgbm-predictor-54f85ddf88-h6rrk\" (UID: \"62c1b0c8-63e5-4109-bb7d-aab7d848b982\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-h6rrk"
Jul 27 21:42:30.140521 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:42:30.140414    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/62c1b0c8-63e5-4109-bb7d-aab7d848b982-proxy-tls\") pod \"isvc-lightgbm-predictor-54f85ddf88-h6rrk\" (UID: \"62c1b0c8-63e5-4109-bb7d-aab7d848b982\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-h6rrk"
Jul 27 21:42:30.140521 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:42:30.140449    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v5rv5\" (UniqueName: \"kubernetes.io/projected/62c1b0c8-63e5-4109-bb7d-aab7d848b982-kube-api-access-v5rv5\") pod \"isvc-lightgbm-predictor-54f85ddf88-h6rrk\" (UID: \"62c1b0c8-63e5-4109-bb7d-aab7d848b982\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-h6rrk"
Jul 27 21:42:30.140880 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:42:30.140858    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/62c1b0c8-63e5-4109-bb7d-aab7d848b982-kserve-provision-location\") pod \"isvc-lightgbm-predictor-54f85ddf88-h6rrk\" (UID: \"62c1b0c8-63e5-4109-bb7d-aab7d848b982\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-h6rrk"
Jul 27 21:42:30.141053 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:42:30.141034    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/62c1b0c8-63e5-4109-bb7d-aab7d848b982-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-predictor-54f85ddf88-h6rrk\" (UID: \"62c1b0c8-63e5-4109-bb7d-aab7d848b982\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-h6rrk"
Jul 27 21:42:30.141106 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:42:30.141053    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/62c1b0c8-63e5-4109-bb7d-aab7d848b982-cabundle-cert\") pod \"isvc-lightgbm-predictor-54f85ddf88-h6rrk\" (UID: \"62c1b0c8-63e5-4109-bb7d-aab7d848b982\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-h6rrk"
Jul 27 21:42:30.142987 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:42:30.142957    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/62c1b0c8-63e5-4109-bb7d-aab7d848b982-proxy-tls\") pod \"isvc-lightgbm-predictor-54f85ddf88-h6rrk\" (UID: \"62c1b0c8-63e5-4109-bb7d-aab7d848b982\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-h6rrk"
Jul 27 21:42:30.147461 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:42:30.147434    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v5rv5\" (UniqueName: \"kubernetes.io/projected/62c1b0c8-63e5-4109-bb7d-aab7d848b982-kube-api-access-v5rv5\") pod \"isvc-lightgbm-predictor-54f85ddf88-h6rrk\" (UID: \"62c1b0c8-63e5-4109-bb7d-aab7d848b982\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-h6rrk"
Jul 27 21:42:30.233794 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:42:30.233709    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-h6rrk"
Jul 27 21:42:30.370624 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:42:30.370591    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-h6rrk"]
Jul 27 21:42:30.374387 ip-10-0-131-133 kubenswrapper[2559]: W0727 21:42:30.374360    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod62c1b0c8_63e5_4109_bb7d_aab7d848b982.slice/crio-982b2c3ad8e614ba6ec23817aae3c36cf0f60a8b6e834a519c00c120756341c2 WatchSource:0}: Error finding container 982b2c3ad8e614ba6ec23817aae3c36cf0f60a8b6e834a519c00c120756341c2: Status 404 returned error can't find the container with id 982b2c3ad8e614ba6ec23817aae3c36cf0f60a8b6e834a519c00c120756341c2
Jul 27 21:42:30.637070 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:42:30.637030    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-h6rrk" event={"ID":"62c1b0c8-63e5-4109-bb7d-aab7d848b982","Type":"ContainerStarted","Data":"49843ff5fecd1790a6be01d18f72691a5c230a2b1aba46a02db3dc57fcd4be12"}
Jul 27 21:42:30.637287 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:42:30.637117    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-h6rrk" event={"ID":"62c1b0c8-63e5-4109-bb7d-aab7d848b982","Type":"ContainerStarted","Data":"982b2c3ad8e614ba6ec23817aae3c36cf0f60a8b6e834a519c00c120756341c2"}
Jul 27 21:42:34.657480 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:42:34.657450    2559 generic.go:356] "Generic (PLEG): container finished" podID="62c1b0c8-63e5-4109-bb7d-aab7d848b982" containerID="49843ff5fecd1790a6be01d18f72691a5c230a2b1aba46a02db3dc57fcd4be12" exitCode=0
Jul 27 21:42:34.657871 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:42:34.657522    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-h6rrk" event={"ID":"62c1b0c8-63e5-4109-bb7d-aab7d848b982","Type":"ContainerDied","Data":"49843ff5fecd1790a6be01d18f72691a5c230a2b1aba46a02db3dc57fcd4be12"}
Jul 27 21:42:34.658856 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:42:34.658834    2559 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 27 21:42:41.702000 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:42:41.701966    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-h6rrk" event={"ID":"62c1b0c8-63e5-4109-bb7d-aab7d848b982","Type":"ContainerStarted","Data":"41ec63e2d3f23407bff0628d8b923d40f494da9c81723f1449b636b642a2886e"}
Jul 27 21:42:41.702498 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:42:41.702072    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-h6rrk" event={"ID":"62c1b0c8-63e5-4109-bb7d-aab7d848b982","Type":"ContainerStarted","Data":"e2c4c088212c96b8580e0bf64231dccf3e3fad87b40f8e2c02b8c27f31ec7031"}
Jul 27 21:42:41.702498 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:42:41.702330    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-h6rrk"
Jul 27 21:42:41.702498 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:42:41.702374    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-h6rrk"
Jul 27 21:42:41.703769 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:42:41.703744    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-h6rrk" podUID="62c1b0c8-63e5-4109-bb7d-aab7d848b982" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.40:8080: connect: connection refused"
Jul 27 21:42:41.720369 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:42:41.720325    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-h6rrk" podStartSLOduration=2.062000437 podStartE2EDuration="12.720311829s" totalImagesPullingTime="6.678168494s" totalInitContainerRuntime="3.980142898s" isStatefulPod=false podCreationTimestamp="2026-07-27 21:42:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 21:42:41.719437362 +0000 UTC m=+2290.298621372" watchObservedRunningTime="2026-07-27 21:42:41.720311829 +0000 UTC m=+2290.299495836"
Jul 27 21:42:42.707325 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:42:42.707286    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-h6rrk" podUID="62c1b0c8-63e5-4109-bb7d-aab7d848b982" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.40:8080: connect: connection refused"
Jul 27 21:42:47.711459 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:42:47.711430    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-h6rrk"
Jul 27 21:42:47.711913 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:42:47.711890    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-h6rrk" podUID="62c1b0c8-63e5-4109-bb7d-aab7d848b982" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.40:8080: connect: connection refused"
Jul 27 21:42:57.712546 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:42:57.712505    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-h6rrk" podUID="62c1b0c8-63e5-4109-bb7d-aab7d848b982" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.40:8080: connect: connection refused"
Jul 27 21:43:07.712313 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:43:07.712269    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-h6rrk" podUID="62c1b0c8-63e5-4109-bb7d-aab7d848b982" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.40:8080: connect: connection refused"
Jul 27 21:43:17.711855 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:43:17.711817    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-h6rrk" podUID="62c1b0c8-63e5-4109-bb7d-aab7d848b982" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.40:8080: connect: connection refused"
Jul 27 21:43:27.712463 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:43:27.712380    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-h6rrk" podUID="62c1b0c8-63e5-4109-bb7d-aab7d848b982" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.40:8080: connect: connection refused"
Jul 27 21:43:37.711993 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:43:37.711954    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-h6rrk" podUID="62c1b0c8-63e5-4109-bb7d-aab7d848b982" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.40:8080: connect: connection refused"
Jul 27 21:43:47.712600 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:43:47.712559    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-h6rrk" podUID="62c1b0c8-63e5-4109-bb7d-aab7d848b982" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.40:8080: connect: connection refused"
Jul 27 21:43:57.712419 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:43:57.712382    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-h6rrk"
Jul 27 21:44:00.116681 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:44:00.116649    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-h6rrk"]
Jul 27 21:44:00.117129 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:44:00.116878    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-h6rrk" podUID="62c1b0c8-63e5-4109-bb7d-aab7d848b982" containerName="kserve-container" containerID="cri-o://e2c4c088212c96b8580e0bf64231dccf3e3fad87b40f8e2c02b8c27f31ec7031" gracePeriod=30
Jul 27 21:44:00.117129 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:44:00.116944    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-h6rrk" podUID="62c1b0c8-63e5-4109-bb7d-aab7d848b982" containerName="kube-rbac-proxy" containerID="cri-o://41ec63e2d3f23407bff0628d8b923d40f494da9c81723f1449b636b642a2886e" gracePeriod=30
Jul 27 21:44:01.085399 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:44:01.085362    2559 generic.go:356] "Generic (PLEG): container finished" podID="62c1b0c8-63e5-4109-bb7d-aab7d848b982" containerID="41ec63e2d3f23407bff0628d8b923d40f494da9c81723f1449b636b642a2886e" exitCode=2
Jul 27 21:44:01.085603 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:44:01.085440    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-h6rrk" event={"ID":"62c1b0c8-63e5-4109-bb7d-aab7d848b982","Type":"ContainerDied","Data":"41ec63e2d3f23407bff0628d8b923d40f494da9c81723f1449b636b642a2886e"}
Jul 27 21:44:02.708283 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:44:02.708238    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-h6rrk" podUID="62c1b0c8-63e5-4109-bb7d-aab7d848b982" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.40:8643/healthz\": dial tcp 10.132.0.40:8643: connect: connection refused"
Jul 27 21:44:04.975301 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:44:04.975272    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-h6rrk"
Jul 27 21:44:05.106406 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:44:05.106312    2559 generic.go:356] "Generic (PLEG): container finished" podID="62c1b0c8-63e5-4109-bb7d-aab7d848b982" containerID="e2c4c088212c96b8580e0bf64231dccf3e3fad87b40f8e2c02b8c27f31ec7031" exitCode=0
Jul 27 21:44:05.106406 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:44:05.106377    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-h6rrk"
Jul 27 21:44:05.106597 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:44:05.106403    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-h6rrk" event={"ID":"62c1b0c8-63e5-4109-bb7d-aab7d848b982","Type":"ContainerDied","Data":"e2c4c088212c96b8580e0bf64231dccf3e3fad87b40f8e2c02b8c27f31ec7031"}
Jul 27 21:44:05.106597 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:44:05.106466    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-h6rrk" event={"ID":"62c1b0c8-63e5-4109-bb7d-aab7d848b982","Type":"ContainerDied","Data":"982b2c3ad8e614ba6ec23817aae3c36cf0f60a8b6e834a519c00c120756341c2"}
Jul 27 21:44:05.106597 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:44:05.106494    2559 scope.go:117] "RemoveContainer" containerID="41ec63e2d3f23407bff0628d8b923d40f494da9c81723f1449b636b642a2886e"
Jul 27 21:44:05.106897 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:44:05.106876    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/62c1b0c8-63e5-4109-bb7d-aab7d848b982-kube-rbac-proxy-sar-config\") pod \"62c1b0c8-63e5-4109-bb7d-aab7d848b982\" (UID: \"62c1b0c8-63e5-4109-bb7d-aab7d848b982\") "
Jul 27 21:44:05.106960 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:44:05.106912    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/62c1b0c8-63e5-4109-bb7d-aab7d848b982-proxy-tls\") pod \"62c1b0c8-63e5-4109-bb7d-aab7d848b982\" (UID: \"62c1b0c8-63e5-4109-bb7d-aab7d848b982\") "
Jul 27 21:44:05.107004 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:44:05.106988    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-v5rv5\" (UniqueName: \"kubernetes.io/projected/62c1b0c8-63e5-4109-bb7d-aab7d848b982-kube-api-access-v5rv5\") pod \"62c1b0c8-63e5-4109-bb7d-aab7d848b982\" (UID: \"62c1b0c8-63e5-4109-bb7d-aab7d848b982\") "
Jul 27 21:44:05.107079 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:44:05.107056    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/62c1b0c8-63e5-4109-bb7d-aab7d848b982-kserve-provision-location\") pod \"62c1b0c8-63e5-4109-bb7d-aab7d848b982\" (UID: \"62c1b0c8-63e5-4109-bb7d-aab7d848b982\") "
Jul 27 21:44:05.107226 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:44:05.107108    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/62c1b0c8-63e5-4109-bb7d-aab7d848b982-cabundle-cert\") pod \"62c1b0c8-63e5-4109-bb7d-aab7d848b982\" (UID: \"62c1b0c8-63e5-4109-bb7d-aab7d848b982\") "
Jul 27 21:44:05.107424 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:44:05.107339    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/62c1b0c8-63e5-4109-bb7d-aab7d848b982-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "62c1b0c8-63e5-4109-bb7d-aab7d848b982" (UID: "62c1b0c8-63e5-4109-bb7d-aab7d848b982"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 21:44:05.107571 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:44:05.107546    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/62c1b0c8-63e5-4109-bb7d-aab7d848b982-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "62c1b0c8-63e5-4109-bb7d-aab7d848b982" (UID: "62c1b0c8-63e5-4109-bb7d-aab7d848b982"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 21:44:05.107644 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:44:05.107567    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/62c1b0c8-63e5-4109-bb7d-aab7d848b982-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "62c1b0c8-63e5-4109-bb7d-aab7d848b982" (UID: "62c1b0c8-63e5-4109-bb7d-aab7d848b982"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 21:44:05.109332 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:44:05.109302    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/62c1b0c8-63e5-4109-bb7d-aab7d848b982-kube-api-access-v5rv5" (OuterVolumeSpecName: "kube-api-access-v5rv5") pod "62c1b0c8-63e5-4109-bb7d-aab7d848b982" (UID: "62c1b0c8-63e5-4109-bb7d-aab7d848b982"). InnerVolumeSpecName "kube-api-access-v5rv5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 21:44:05.109450 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:44:05.109331    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/62c1b0c8-63e5-4109-bb7d-aab7d848b982-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "62c1b0c8-63e5-4109-bb7d-aab7d848b982" (UID: "62c1b0c8-63e5-4109-bb7d-aab7d848b982"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 21:44:05.126353 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:44:05.126327    2559 scope.go:117] "RemoveContainer" containerID="e2c4c088212c96b8580e0bf64231dccf3e3fad87b40f8e2c02b8c27f31ec7031"
Jul 27 21:44:05.134768 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:44:05.134744    2559 scope.go:117] "RemoveContainer" containerID="49843ff5fecd1790a6be01d18f72691a5c230a2b1aba46a02db3dc57fcd4be12"
Jul 27 21:44:05.144233 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:44:05.144194    2559 scope.go:117] "RemoveContainer" containerID="41ec63e2d3f23407bff0628d8b923d40f494da9c81723f1449b636b642a2886e"
Jul 27 21:44:05.144948 ip-10-0-131-133 kubenswrapper[2559]: E0727 21:44:05.144926    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"41ec63e2d3f23407bff0628d8b923d40f494da9c81723f1449b636b642a2886e\": container with ID starting with 41ec63e2d3f23407bff0628d8b923d40f494da9c81723f1449b636b642a2886e not found: ID does not exist" containerID="41ec63e2d3f23407bff0628d8b923d40f494da9c81723f1449b636b642a2886e"
Jul 27 21:44:05.145021 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:44:05.144957    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"41ec63e2d3f23407bff0628d8b923d40f494da9c81723f1449b636b642a2886e"} err="failed to get container status \"41ec63e2d3f23407bff0628d8b923d40f494da9c81723f1449b636b642a2886e\": rpc error: code = NotFound desc = could not find container \"41ec63e2d3f23407bff0628d8b923d40f494da9c81723f1449b636b642a2886e\": container with ID starting with 41ec63e2d3f23407bff0628d8b923d40f494da9c81723f1449b636b642a2886e not found: ID does not exist"
Jul 27 21:44:05.145021 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:44:05.144976    2559 scope.go:117] "RemoveContainer" containerID="e2c4c088212c96b8580e0bf64231dccf3e3fad87b40f8e2c02b8c27f31ec7031"
Jul 27 21:44:05.145231 ip-10-0-131-133 kubenswrapper[2559]: E0727 21:44:05.145195    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e2c4c088212c96b8580e0bf64231dccf3e3fad87b40f8e2c02b8c27f31ec7031\": container with ID starting with e2c4c088212c96b8580e0bf64231dccf3e3fad87b40f8e2c02b8c27f31ec7031 not found: ID does not exist" containerID="e2c4c088212c96b8580e0bf64231dccf3e3fad87b40f8e2c02b8c27f31ec7031"
Jul 27 21:44:05.145283 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:44:05.145240    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e2c4c088212c96b8580e0bf64231dccf3e3fad87b40f8e2c02b8c27f31ec7031"} err="failed to get container status \"e2c4c088212c96b8580e0bf64231dccf3e3fad87b40f8e2c02b8c27f31ec7031\": rpc error: code = NotFound desc = could not find container \"e2c4c088212c96b8580e0bf64231dccf3e3fad87b40f8e2c02b8c27f31ec7031\": container with ID starting with e2c4c088212c96b8580e0bf64231dccf3e3fad87b40f8e2c02b8c27f31ec7031 not found: ID does not exist"
Jul 27 21:44:05.145283 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:44:05.145258    2559 scope.go:117] "RemoveContainer" containerID="49843ff5fecd1790a6be01d18f72691a5c230a2b1aba46a02db3dc57fcd4be12"
Jul 27 21:44:05.145537 ip-10-0-131-133 kubenswrapper[2559]: E0727 21:44:05.145521    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"49843ff5fecd1790a6be01d18f72691a5c230a2b1aba46a02db3dc57fcd4be12\": container with ID starting with 49843ff5fecd1790a6be01d18f72691a5c230a2b1aba46a02db3dc57fcd4be12 not found: ID does not exist" containerID="49843ff5fecd1790a6be01d18f72691a5c230a2b1aba46a02db3dc57fcd4be12"
Jul 27 21:44:05.145607 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:44:05.145539    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"49843ff5fecd1790a6be01d18f72691a5c230a2b1aba46a02db3dc57fcd4be12"} err="failed to get container status \"49843ff5fecd1790a6be01d18f72691a5c230a2b1aba46a02db3dc57fcd4be12\": rpc error: code = NotFound desc = could not find container \"49843ff5fecd1790a6be01d18f72691a5c230a2b1aba46a02db3dc57fcd4be12\": container with ID starting with 49843ff5fecd1790a6be01d18f72691a5c230a2b1aba46a02db3dc57fcd4be12 not found: ID does not exist"
Jul 27 21:44:05.208048 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:44:05.208012    2559 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/62c1b0c8-63e5-4109-bb7d-aab7d848b982-kserve-provision-location\") on node \"ip-10-0-131-133.ec2.internal\" DevicePath \"\""
Jul 27 21:44:05.208048 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:44:05.208042    2559 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/62c1b0c8-63e5-4109-bb7d-aab7d848b982-cabundle-cert\") on node \"ip-10-0-131-133.ec2.internal\" DevicePath \"\""
Jul 27 21:44:05.208048 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:44:05.208056    2559 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/62c1b0c8-63e5-4109-bb7d-aab7d848b982-kube-rbac-proxy-sar-config\") on node \"ip-10-0-131-133.ec2.internal\" DevicePath \"\""
Jul 27 21:44:05.208327 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:44:05.208065    2559 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/62c1b0c8-63e5-4109-bb7d-aab7d848b982-proxy-tls\") on node \"ip-10-0-131-133.ec2.internal\" DevicePath \"\""
Jul 27 21:44:05.208327 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:44:05.208074    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v5rv5\" (UniqueName: \"kubernetes.io/projected/62c1b0c8-63e5-4109-bb7d-aab7d848b982-kube-api-access-v5rv5\") on node \"ip-10-0-131-133.ec2.internal\" DevicePath \"\""
Jul 27 21:44:05.260499 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:44:05.260465    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-75dmc"]
Jul 27 21:44:05.260859 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:44:05.260846    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="62c1b0c8-63e5-4109-bb7d-aab7d848b982" containerName="kserve-container"
Jul 27 21:44:05.260904 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:44:05.260861    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="62c1b0c8-63e5-4109-bb7d-aab7d848b982" containerName="kserve-container"
Jul 27 21:44:05.260904 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:44:05.260868    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="62c1b0c8-63e5-4109-bb7d-aab7d848b982" containerName="kube-rbac-proxy"
Jul 27 21:44:05.260904 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:44:05.260875    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="62c1b0c8-63e5-4109-bb7d-aab7d848b982" containerName="kube-rbac-proxy"
Jul 27 21:44:05.260904 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:44:05.260889    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="62c1b0c8-63e5-4109-bb7d-aab7d848b982" containerName="storage-initializer"
Jul 27 21:44:05.260904 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:44:05.260896    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="62c1b0c8-63e5-4109-bb7d-aab7d848b982" containerName="storage-initializer"
Jul 27 21:44:05.261075 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:44:05.260963    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="62c1b0c8-63e5-4109-bb7d-aab7d848b982" containerName="kserve-container"
Jul 27 21:44:05.261075 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:44:05.260970    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="62c1b0c8-63e5-4109-bb7d-aab7d848b982" containerName="kube-rbac-proxy"
Jul 27 21:44:05.265591 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:44:05.265561    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-75dmc"
Jul 27 21:44:05.267793 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:44:05.267773    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-lightgbm-runtime-predictor-serving-cert\""
Jul 27 21:44:05.267893 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:44:05.267775    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-lightgbm-runtime-kube-rbac-proxy-sar-config\""
Jul 27 21:44:05.274078 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:44:05.274049    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-75dmc"]
Jul 27 21:44:05.409014 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:44:05.408975    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/641c5f41-5e1f-4904-9938-207ea8535f85-proxy-tls\") pod \"isvc-lightgbm-runtime-predictor-577f7db7cb-75dmc\" (UID: \"641c5f41-5e1f-4904-9938-207ea8535f85\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-75dmc"
Jul 27 21:44:05.409014 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:44:05.409012    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5gf8w\" (UniqueName: \"kubernetes.io/projected/641c5f41-5e1f-4904-9938-207ea8535f85-kube-api-access-5gf8w\") pod \"isvc-lightgbm-runtime-predictor-577f7db7cb-75dmc\" (UID: \"641c5f41-5e1f-4904-9938-207ea8535f85\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-75dmc"
Jul 27 21:44:05.409283 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:44:05.409037    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/641c5f41-5e1f-4904-9938-207ea8535f85-cabundle-cert\") pod \"isvc-lightgbm-runtime-predictor-577f7db7cb-75dmc\" (UID: \"641c5f41-5e1f-4904-9938-207ea8535f85\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-75dmc"
Jul 27 21:44:05.409283 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:44:05.409200    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/641c5f41-5e1f-4904-9938-207ea8535f85-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-runtime-predictor-577f7db7cb-75dmc\" (UID: \"641c5f41-5e1f-4904-9938-207ea8535f85\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-75dmc"
Jul 27 21:44:05.409283 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:44:05.409274    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/641c5f41-5e1f-4904-9938-207ea8535f85-kserve-provision-location\") pod \"isvc-lightgbm-runtime-predictor-577f7db7cb-75dmc\" (UID: \"641c5f41-5e1f-4904-9938-207ea8535f85\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-75dmc"
Jul 27 21:44:05.430332 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:44:05.430296    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-h6rrk"]
Jul 27 21:44:05.436457 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:44:05.436425    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-h6rrk"]
Jul 27 21:44:05.510018 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:44:05.509978    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/641c5f41-5e1f-4904-9938-207ea8535f85-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-runtime-predictor-577f7db7cb-75dmc\" (UID: \"641c5f41-5e1f-4904-9938-207ea8535f85\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-75dmc"
Jul 27 21:44:05.510018 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:44:05.510018    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/641c5f41-5e1f-4904-9938-207ea8535f85-kserve-provision-location\") pod \"isvc-lightgbm-runtime-predictor-577f7db7cb-75dmc\" (UID: \"641c5f41-5e1f-4904-9938-207ea8535f85\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-75dmc"
Jul 27 21:44:05.510321 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:44:05.510149    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/641c5f41-5e1f-4904-9938-207ea8535f85-proxy-tls\") pod \"isvc-lightgbm-runtime-predictor-577f7db7cb-75dmc\" (UID: \"641c5f41-5e1f-4904-9938-207ea8535f85\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-75dmc"
Jul 27 21:44:05.510321 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:44:05.510188    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5gf8w\" (UniqueName: \"kubernetes.io/projected/641c5f41-5e1f-4904-9938-207ea8535f85-kube-api-access-5gf8w\") pod \"isvc-lightgbm-runtime-predictor-577f7db7cb-75dmc\" (UID: \"641c5f41-5e1f-4904-9938-207ea8535f85\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-75dmc"
Jul 27 21:44:05.510321 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:44:05.510259    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/641c5f41-5e1f-4904-9938-207ea8535f85-cabundle-cert\") pod \"isvc-lightgbm-runtime-predictor-577f7db7cb-75dmc\" (UID: \"641c5f41-5e1f-4904-9938-207ea8535f85\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-75dmc"
Jul 27 21:44:05.510557 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:44:05.510433    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/641c5f41-5e1f-4904-9938-207ea8535f85-kserve-provision-location\") pod \"isvc-lightgbm-runtime-predictor-577f7db7cb-75dmc\" (UID: \"641c5f41-5e1f-4904-9938-207ea8535f85\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-75dmc"
Jul 27 21:44:05.510733 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:44:05.510712    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/641c5f41-5e1f-4904-9938-207ea8535f85-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-runtime-predictor-577f7db7cb-75dmc\" (UID: \"641c5f41-5e1f-4904-9938-207ea8535f85\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-75dmc"
Jul 27 21:44:05.510859 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:44:05.510840    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/641c5f41-5e1f-4904-9938-207ea8535f85-cabundle-cert\") pod \"isvc-lightgbm-runtime-predictor-577f7db7cb-75dmc\" (UID: \"641c5f41-5e1f-4904-9938-207ea8535f85\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-75dmc"
Jul 27 21:44:05.512638 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:44:05.512621    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/641c5f41-5e1f-4904-9938-207ea8535f85-proxy-tls\") pod \"isvc-lightgbm-runtime-predictor-577f7db7cb-75dmc\" (UID: \"641c5f41-5e1f-4904-9938-207ea8535f85\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-75dmc"
Jul 27 21:44:05.517985 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:44:05.517962    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5gf8w\" (UniqueName: \"kubernetes.io/projected/641c5f41-5e1f-4904-9938-207ea8535f85-kube-api-access-5gf8w\") pod \"isvc-lightgbm-runtime-predictor-577f7db7cb-75dmc\" (UID: \"641c5f41-5e1f-4904-9938-207ea8535f85\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-75dmc"
Jul 27 21:44:05.576974 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:44:05.576933    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-75dmc"
Jul 27 21:44:05.711136 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:44:05.711105    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-75dmc"]
Jul 27 21:44:05.713066 ip-10-0-131-133 kubenswrapper[2559]: W0727 21:44:05.713040    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod641c5f41_5e1f_4904_9938_207ea8535f85.slice/crio-2ce2e833d7714c0a6a5ce0058128fa250f2e2f9cf56853776799274ccfd092a1 WatchSource:0}: Error finding container 2ce2e833d7714c0a6a5ce0058128fa250f2e2f9cf56853776799274ccfd092a1: Status 404 returned error can't find the container with id 2ce2e833d7714c0a6a5ce0058128fa250f2e2f9cf56853776799274ccfd092a1
Jul 27 21:44:05.982496 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:44:05.982418    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="62c1b0c8-63e5-4109-bb7d-aab7d848b982" path="/var/lib/kubelet/pods/62c1b0c8-63e5-4109-bb7d-aab7d848b982/volumes"
Jul 27 21:44:06.114393 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:44:06.114355    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-75dmc" event={"ID":"641c5f41-5e1f-4904-9938-207ea8535f85","Type":"ContainerStarted","Data":"789ff491b86bead5566388689c66d0f0373368d9429a542cb785a1ee45ce8c01"}
Jul 27 21:44:06.114621 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:44:06.114432    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-75dmc" event={"ID":"641c5f41-5e1f-4904-9938-207ea8535f85","Type":"ContainerStarted","Data":"2ce2e833d7714c0a6a5ce0058128fa250f2e2f9cf56853776799274ccfd092a1"}
Jul 27 21:44:10.143615 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:44:10.143585    2559 generic.go:356] "Generic (PLEG): container finished" podID="641c5f41-5e1f-4904-9938-207ea8535f85" containerID="789ff491b86bead5566388689c66d0f0373368d9429a542cb785a1ee45ce8c01" exitCode=0
Jul 27 21:44:10.144057 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:44:10.143655    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-75dmc" event={"ID":"641c5f41-5e1f-4904-9938-207ea8535f85","Type":"ContainerDied","Data":"789ff491b86bead5566388689c66d0f0373368d9429a542cb785a1ee45ce8c01"}
Jul 27 21:44:11.151115 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:44:11.151080    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-75dmc" event={"ID":"641c5f41-5e1f-4904-9938-207ea8535f85","Type":"ContainerStarted","Data":"7595410aad6fc9d6d28836c8a58cb01ee0d63739fb8000d0a41db1aeb56b9eda"}
Jul 27 21:44:11.151541 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:44:11.151144    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-75dmc" event={"ID":"641c5f41-5e1f-4904-9938-207ea8535f85","Type":"ContainerStarted","Data":"dce211518749a0e079880de83ce8769d3a49c1f32dbe356a9936942effde2009"}
Jul 27 21:44:11.151541 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:44:11.151435    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-75dmc"
Jul 27 21:44:11.151641 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:44:11.151576    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-75dmc"
Jul 27 21:44:11.153124 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:44:11.153098    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-75dmc" podUID="641c5f41-5e1f-4904-9938-207ea8535f85" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.41:8080: connect: connection refused"
Jul 27 21:44:11.169265 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:44:11.169189    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-75dmc" podStartSLOduration=2.251909113 podStartE2EDuration="6.169167451s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.917258338s" isStatefulPod=false podCreationTimestamp="2026-07-27 21:44:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 21:44:11.16833933 +0000 UTC m=+2379.747523336" watchObservedRunningTime="2026-07-27 21:44:11.169167451 +0000 UTC m=+2379.748351458"
Jul 27 21:44:12.156444 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:44:12.156403    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-75dmc" podUID="641c5f41-5e1f-4904-9938-207ea8535f85" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.41:8080: connect: connection refused"
Jul 27 21:44:17.161002 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:44:17.160965    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-75dmc"
Jul 27 21:44:17.161729 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:44:17.161703    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-75dmc" podUID="641c5f41-5e1f-4904-9938-207ea8535f85" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.41:8080: connect: connection refused"
Jul 27 21:44:27.161785 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:44:27.161742    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-75dmc" podUID="641c5f41-5e1f-4904-9938-207ea8535f85" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.41:8080: connect: connection refused"
Jul 27 21:44:32.069944 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:44:32.069916    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-bjvbc_cba53c9a-64c0-4afa-8731-35521e196622/console-operator/1.log"
Jul 27 21:44:32.073615 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:44:32.073588    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-bjvbc_cba53c9a-64c0-4afa-8731-35521e196622/console-operator/1.log"
Jul 27 21:44:37.162464 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:44:37.162425    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-75dmc" podUID="641c5f41-5e1f-4904-9938-207ea8535f85" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.41:8080: connect: connection refused"
Jul 27 21:44:47.162270 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:44:47.162232    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-75dmc" podUID="641c5f41-5e1f-4904-9938-207ea8535f85" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.41:8080: connect: connection refused"
Jul 27 21:44:57.162496 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:44:57.162447    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-75dmc" podUID="641c5f41-5e1f-4904-9938-207ea8535f85" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.41:8080: connect: connection refused"
Jul 27 21:45:07.161814 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:45:07.161757    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-75dmc" podUID="641c5f41-5e1f-4904-9938-207ea8535f85" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.41:8080: connect: connection refused"
Jul 27 21:45:17.165637 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:45:17.165590    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-75dmc" podUID="641c5f41-5e1f-4904-9938-207ea8535f85" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.41:8080: connect: connection refused"
Jul 27 21:45:24.976544 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:45:24.976494    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-75dmc" podUID="641c5f41-5e1f-4904-9938-207ea8535f85" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.41:8080: connect: connection refused"
Jul 27 21:45:34.977127 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:45:34.977092    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-75dmc"
Jul 27 21:45:35.872596 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:45:35.872560    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-75dmc"]
Jul 27 21:45:35.872820 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:45:35.872781    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-75dmc" podUID="641c5f41-5e1f-4904-9938-207ea8535f85" containerName="kserve-container" containerID="cri-o://dce211518749a0e079880de83ce8769d3a49c1f32dbe356a9936942effde2009" gracePeriod=30
Jul 27 21:45:35.872872 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:45:35.872828    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-75dmc" podUID="641c5f41-5e1f-4904-9938-207ea8535f85" containerName="kube-rbac-proxy" containerID="cri-o://7595410aad6fc9d6d28836c8a58cb01ee0d63739fb8000d0a41db1aeb56b9eda" gracePeriod=30
Jul 27 21:45:36.566102 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:45:36.566067    2559 generic.go:356] "Generic (PLEG): container finished" podID="641c5f41-5e1f-4904-9938-207ea8535f85" containerID="7595410aad6fc9d6d28836c8a58cb01ee0d63739fb8000d0a41db1aeb56b9eda" exitCode=2
Jul 27 21:45:36.566525 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:45:36.566136    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-75dmc" event={"ID":"641c5f41-5e1f-4904-9938-207ea8535f85","Type":"ContainerDied","Data":"7595410aad6fc9d6d28836c8a58cb01ee0d63739fb8000d0a41db1aeb56b9eda"}
Jul 27 21:45:37.157280 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:45:37.157240    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-75dmc" podUID="641c5f41-5e1f-4904-9938-207ea8535f85" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.41:8643/healthz\": dial tcp 10.132.0.41:8643: connect: connection refused"
Jul 27 21:45:40.588476 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:45:40.588441    2559 generic.go:356] "Generic (PLEG): container finished" podID="641c5f41-5e1f-4904-9938-207ea8535f85" containerID="dce211518749a0e079880de83ce8769d3a49c1f32dbe356a9936942effde2009" exitCode=0
Jul 27 21:45:40.588887 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:45:40.588506    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-75dmc" event={"ID":"641c5f41-5e1f-4904-9938-207ea8535f85","Type":"ContainerDied","Data":"dce211518749a0e079880de83ce8769d3a49c1f32dbe356a9936942effde2009"}
Jul 27 21:45:40.721232 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:45:40.721197    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-75dmc"
Jul 27 21:45:40.804751 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:45:40.804649    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/641c5f41-5e1f-4904-9938-207ea8535f85-kube-rbac-proxy-sar-config\") pod \"641c5f41-5e1f-4904-9938-207ea8535f85\" (UID: \"641c5f41-5e1f-4904-9938-207ea8535f85\") "
Jul 27 21:45:40.804751 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:45:40.804725    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/641c5f41-5e1f-4904-9938-207ea8535f85-cabundle-cert\") pod \"641c5f41-5e1f-4904-9938-207ea8535f85\" (UID: \"641c5f41-5e1f-4904-9938-207ea8535f85\") "
Jul 27 21:45:40.805010 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:45:40.804813    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/641c5f41-5e1f-4904-9938-207ea8535f85-kserve-provision-location\") pod \"641c5f41-5e1f-4904-9938-207ea8535f85\" (UID: \"641c5f41-5e1f-4904-9938-207ea8535f85\") "
Jul 27 21:45:40.805010 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:45:40.804851    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/641c5f41-5e1f-4904-9938-207ea8535f85-proxy-tls\") pod \"641c5f41-5e1f-4904-9938-207ea8535f85\" (UID: \"641c5f41-5e1f-4904-9938-207ea8535f85\") "
Jul 27 21:45:40.805010 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:45:40.804900    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5gf8w\" (UniqueName: \"kubernetes.io/projected/641c5f41-5e1f-4904-9938-207ea8535f85-kube-api-access-5gf8w\") pod \"641c5f41-5e1f-4904-9938-207ea8535f85\" (UID: \"641c5f41-5e1f-4904-9938-207ea8535f85\") "
Jul 27 21:45:40.805146 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:45:40.805032    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/641c5f41-5e1f-4904-9938-207ea8535f85-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "641c5f41-5e1f-4904-9938-207ea8535f85" (UID: "641c5f41-5e1f-4904-9938-207ea8535f85"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 21:45:40.805146 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:45:40.805058    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/641c5f41-5e1f-4904-9938-207ea8535f85-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "641c5f41-5e1f-4904-9938-207ea8535f85" (UID: "641c5f41-5e1f-4904-9938-207ea8535f85"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 21:45:40.805146 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:45:40.805124    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/641c5f41-5e1f-4904-9938-207ea8535f85-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "641c5f41-5e1f-4904-9938-207ea8535f85" (UID: "641c5f41-5e1f-4904-9938-207ea8535f85"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 21:45:40.805285 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:45:40.805207    2559 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/641c5f41-5e1f-4904-9938-207ea8535f85-kserve-provision-location\") on node \"ip-10-0-131-133.ec2.internal\" DevicePath \"\""
Jul 27 21:45:40.805285 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:45:40.805248    2559 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/641c5f41-5e1f-4904-9938-207ea8535f85-kube-rbac-proxy-sar-config\") on node \"ip-10-0-131-133.ec2.internal\" DevicePath \"\""
Jul 27 21:45:40.805285 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:45:40.805257    2559 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/641c5f41-5e1f-4904-9938-207ea8535f85-cabundle-cert\") on node \"ip-10-0-131-133.ec2.internal\" DevicePath \"\""
Jul 27 21:45:40.807007 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:45:40.806981    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/641c5f41-5e1f-4904-9938-207ea8535f85-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "641c5f41-5e1f-4904-9938-207ea8535f85" (UID: "641c5f41-5e1f-4904-9938-207ea8535f85"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 21:45:40.807124 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:45:40.807041    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/641c5f41-5e1f-4904-9938-207ea8535f85-kube-api-access-5gf8w" (OuterVolumeSpecName: "kube-api-access-5gf8w") pod "641c5f41-5e1f-4904-9938-207ea8535f85" (UID: "641c5f41-5e1f-4904-9938-207ea8535f85"). InnerVolumeSpecName "kube-api-access-5gf8w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 21:45:40.906375 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:45:40.906337    2559 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/641c5f41-5e1f-4904-9938-207ea8535f85-proxy-tls\") on node \"ip-10-0-131-133.ec2.internal\" DevicePath \"\""
Jul 27 21:45:40.906375 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:45:40.906365    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5gf8w\" (UniqueName: \"kubernetes.io/projected/641c5f41-5e1f-4904-9938-207ea8535f85-kube-api-access-5gf8w\") on node \"ip-10-0-131-133.ec2.internal\" DevicePath \"\""
Jul 27 21:45:41.594656 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:45:41.594616    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-75dmc" event={"ID":"641c5f41-5e1f-4904-9938-207ea8535f85","Type":"ContainerDied","Data":"2ce2e833d7714c0a6a5ce0058128fa250f2e2f9cf56853776799274ccfd092a1"}
Jul 27 21:45:41.595116 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:45:41.594696    2559 scope.go:117] "RemoveContainer" containerID="7595410aad6fc9d6d28836c8a58cb01ee0d63739fb8000d0a41db1aeb56b9eda"
Jul 27 21:45:41.595116 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:45:41.594634    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-75dmc"
Jul 27 21:45:41.603777 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:45:41.603760    2559 scope.go:117] "RemoveContainer" containerID="dce211518749a0e079880de83ce8769d3a49c1f32dbe356a9936942effde2009"
Jul 27 21:45:41.611572 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:45:41.611547    2559 scope.go:117] "RemoveContainer" containerID="789ff491b86bead5566388689c66d0f0373368d9429a542cb785a1ee45ce8c01"
Jul 27 21:45:41.617017 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:45:41.616993    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-75dmc"]
Jul 27 21:45:41.620704 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:45:41.620682    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-75dmc"]
Jul 27 21:45:41.982122 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:45:41.982088    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="641c5f41-5e1f-4904-9938-207ea8535f85" path="/var/lib/kubelet/pods/641c5f41-5e1f-4904-9938-207ea8535f85/volumes"
Jul 27 21:49:32.098386 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:49:32.098314    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-bjvbc_cba53c9a-64c0-4afa-8731-35521e196622/console-operator/1.log"
Jul 27 21:49:32.103662 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:49:32.103638    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-bjvbc_cba53c9a-64c0-4afa-8731-35521e196622/console-operator/1.log"
Jul 27 21:50:06.784377 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:50:06.784335    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5996957568-7dskx"]
Jul 27 21:50:06.784993 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:50:06.784866    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="641c5f41-5e1f-4904-9938-207ea8535f85" containerName="kube-rbac-proxy"
Jul 27 21:50:06.784993 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:50:06.784886    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="641c5f41-5e1f-4904-9938-207ea8535f85" containerName="kube-rbac-proxy"
Jul 27 21:50:06.784993 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:50:06.784902    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="641c5f41-5e1f-4904-9938-207ea8535f85" containerName="kserve-container"
Jul 27 21:50:06.784993 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:50:06.784910    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="641c5f41-5e1f-4904-9938-207ea8535f85" containerName="kserve-container"
Jul 27 21:50:06.784993 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:50:06.784937    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="641c5f41-5e1f-4904-9938-207ea8535f85" containerName="storage-initializer"
Jul 27 21:50:06.784993 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:50:06.784948    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="641c5f41-5e1f-4904-9938-207ea8535f85" containerName="storage-initializer"
Jul 27 21:50:06.785353 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:50:06.785047    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="641c5f41-5e1f-4904-9938-207ea8535f85" containerName="kserve-container"
Jul 27 21:50:06.785353 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:50:06.785065    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="641c5f41-5e1f-4904-9938-207ea8535f85" containerName="kube-rbac-proxy"
Jul 27 21:50:06.788608 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:50:06.788586    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5996957568-7dskx"
Jul 27 21:50:06.790733 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:50:06.790711    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-mcp-predictor-serving-cert\""
Jul 27 21:50:06.791265 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:50:06.791247    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 27 21:50:06.791367 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:50:06.791247    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 27 21:50:06.791432 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:50:06.791397    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-mcp-kube-rbac-proxy-sar-config\""
Jul 27 21:50:06.791620 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:50:06.791605    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 27 21:50:06.791720 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:50:06.791649    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-v7c66\""
Jul 27 21:50:06.791720 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:50:06.791704    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 27 21:50:06.799592 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:50:06.799566    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5996957568-7dskx"]
Jul 27 21:50:06.976655 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:50:06.976616    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/508f765d-e01a-4506-9862-7fab9968614c-proxy-tls\") pod \"isvc-sklearn-mcp-predictor-5996957568-7dskx\" (UID: \"508f765d-e01a-4506-9862-7fab9968614c\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5996957568-7dskx"
Jul 27 21:50:06.976846 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:50:06.976673    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gqlgf\" (UniqueName: \"kubernetes.io/projected/508f765d-e01a-4506-9862-7fab9968614c-kube-api-access-gqlgf\") pod \"isvc-sklearn-mcp-predictor-5996957568-7dskx\" (UID: \"508f765d-e01a-4506-9862-7fab9968614c\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5996957568-7dskx"
Jul 27 21:50:06.976846 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:50:06.976714    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/508f765d-e01a-4506-9862-7fab9968614c-kserve-provision-location\") pod \"isvc-sklearn-mcp-predictor-5996957568-7dskx\" (UID: \"508f765d-e01a-4506-9862-7fab9968614c\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5996957568-7dskx"
Jul 27 21:50:06.976846 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:50:06.976803    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/508f765d-e01a-4506-9862-7fab9968614c-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-mcp-predictor-5996957568-7dskx\" (UID: \"508f765d-e01a-4506-9862-7fab9968614c\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5996957568-7dskx"
Jul 27 21:50:06.976846 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:50:06.976831    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/508f765d-e01a-4506-9862-7fab9968614c-cabundle-cert\") pod \"isvc-sklearn-mcp-predictor-5996957568-7dskx\" (UID: \"508f765d-e01a-4506-9862-7fab9968614c\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5996957568-7dskx"
Jul 27 21:50:07.077882 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:50:07.077776    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gqlgf\" (UniqueName: \"kubernetes.io/projected/508f765d-e01a-4506-9862-7fab9968614c-kube-api-access-gqlgf\") pod \"isvc-sklearn-mcp-predictor-5996957568-7dskx\" (UID: \"508f765d-e01a-4506-9862-7fab9968614c\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5996957568-7dskx"
Jul 27 21:50:07.077882 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:50:07.077819    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/508f765d-e01a-4506-9862-7fab9968614c-kserve-provision-location\") pod \"isvc-sklearn-mcp-predictor-5996957568-7dskx\" (UID: \"508f765d-e01a-4506-9862-7fab9968614c\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5996957568-7dskx"
Jul 27 21:50:07.078123 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:50:07.077902    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/508f765d-e01a-4506-9862-7fab9968614c-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-mcp-predictor-5996957568-7dskx\" (UID: \"508f765d-e01a-4506-9862-7fab9968614c\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5996957568-7dskx"
Jul 27 21:50:07.078123 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:50:07.077921    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/508f765d-e01a-4506-9862-7fab9968614c-cabundle-cert\") pod \"isvc-sklearn-mcp-predictor-5996957568-7dskx\" (UID: \"508f765d-e01a-4506-9862-7fab9968614c\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5996957568-7dskx"
Jul 27 21:50:07.078123 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:50:07.077949    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/508f765d-e01a-4506-9862-7fab9968614c-proxy-tls\") pod \"isvc-sklearn-mcp-predictor-5996957568-7dskx\" (UID: \"508f765d-e01a-4506-9862-7fab9968614c\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5996957568-7dskx"
Jul 27 21:50:07.078313 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:50:07.078257    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/508f765d-e01a-4506-9862-7fab9968614c-kserve-provision-location\") pod \"isvc-sklearn-mcp-predictor-5996957568-7dskx\" (UID: \"508f765d-e01a-4506-9862-7fab9968614c\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5996957568-7dskx"
Jul 27 21:50:07.078701 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:50:07.078678    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/508f765d-e01a-4506-9862-7fab9968614c-cabundle-cert\") pod \"isvc-sklearn-mcp-predictor-5996957568-7dskx\" (UID: \"508f765d-e01a-4506-9862-7fab9968614c\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5996957568-7dskx"
Jul 27 21:50:07.078746 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:50:07.078678    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/508f765d-e01a-4506-9862-7fab9968614c-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-mcp-predictor-5996957568-7dskx\" (UID: \"508f765d-e01a-4506-9862-7fab9968614c\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5996957568-7dskx"
Jul 27 21:50:07.080363 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:50:07.080345    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/508f765d-e01a-4506-9862-7fab9968614c-proxy-tls\") pod \"isvc-sklearn-mcp-predictor-5996957568-7dskx\" (UID: \"508f765d-e01a-4506-9862-7fab9968614c\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5996957568-7dskx"
Jul 27 21:50:07.085626 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:50:07.085603    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gqlgf\" (UniqueName: \"kubernetes.io/projected/508f765d-e01a-4506-9862-7fab9968614c-kube-api-access-gqlgf\") pod \"isvc-sklearn-mcp-predictor-5996957568-7dskx\" (UID: \"508f765d-e01a-4506-9862-7fab9968614c\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5996957568-7dskx"
Jul 27 21:50:07.099563 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:50:07.099531    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5996957568-7dskx"
Jul 27 21:50:07.235285 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:50:07.235259    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5996957568-7dskx"]
Jul 27 21:50:07.237431 ip-10-0-131-133 kubenswrapper[2559]: W0727 21:50:07.237401    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod508f765d_e01a_4506_9862_7fab9968614c.slice/crio-5013594ee133c601e26df73d25bb16b8697c17c0eb42236589b2f269e34eca2d WatchSource:0}: Error finding container 5013594ee133c601e26df73d25bb16b8697c17c0eb42236589b2f269e34eca2d: Status 404 returned error can't find the container with id 5013594ee133c601e26df73d25bb16b8697c17c0eb42236589b2f269e34eca2d
Jul 27 21:50:07.844132 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:50:07.844095    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5996957568-7dskx" event={"ID":"508f765d-e01a-4506-9862-7fab9968614c","Type":"ContainerStarted","Data":"20af4bb2cec8b29e07afd4b9b23698cfd665cdbbce88354a572f6ff9845f73c3"}
Jul 27 21:50:07.844547 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:50:07.844185    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5996957568-7dskx" event={"ID":"508f765d-e01a-4506-9862-7fab9968614c","Type":"ContainerStarted","Data":"5013594ee133c601e26df73d25bb16b8697c17c0eb42236589b2f269e34eca2d"}
Jul 27 21:50:11.865147 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:50:11.865110    2559 generic.go:356] "Generic (PLEG): container finished" podID="508f765d-e01a-4506-9862-7fab9968614c" containerID="20af4bb2cec8b29e07afd4b9b23698cfd665cdbbce88354a572f6ff9845f73c3" exitCode=0
Jul 27 21:50:11.865571 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:50:11.865186    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5996957568-7dskx" event={"ID":"508f765d-e01a-4506-9862-7fab9968614c","Type":"ContainerDied","Data":"20af4bb2cec8b29e07afd4b9b23698cfd665cdbbce88354a572f6ff9845f73c3"}
Jul 27 21:50:11.934233 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:50:11.934199    2559 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 27 21:50:12.875291 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:50:12.875246    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5996957568-7dskx" event={"ID":"508f765d-e01a-4506-9862-7fab9968614c","Type":"ContainerStarted","Data":"ed9aaa3f36b729e5dd481a3bbe8efff93ef52a44b0917b6e10bdaf367af1103b"}
Jul 27 21:50:14.890030 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:50:14.889986    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5996957568-7dskx" event={"ID":"508f765d-e01a-4506-9862-7fab9968614c","Type":"ContainerStarted","Data":"c01b25026b4b706a938d780d3df4f192e3ad67d3eaae34c1741674c9ffd4d4d7"}
Jul 27 21:50:14.890414 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:50:14.890058    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5996957568-7dskx" event={"ID":"508f765d-e01a-4506-9862-7fab9968614c","Type":"ContainerStarted","Data":"8ac61dda00c63ac7466d7b235c0dead4cb223f726fa937346db7da6dd390d724"}
Jul 27 21:50:14.890414 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:50:14.890192    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5996957568-7dskx"
Jul 27 21:50:14.890414 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:50:14.890285    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5996957568-7dskx"
Jul 27 21:50:14.911162 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:50:14.911110    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5996957568-7dskx" podStartSLOduration=2.600369713 podStartE2EDuration="8.911093408s" totalImagesPullingTime="2.358089767s" totalInitContainerRuntime="3.952633928s" isStatefulPod=false podCreationTimestamp="2026-07-27 21:50:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 21:50:14.909572663 +0000 UTC m=+2743.488756668" watchObservedRunningTime="2026-07-27 21:50:14.911093408 +0000 UTC m=+2743.490277413"
Jul 27 21:50:15.894662 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:50:15.894631    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5996957568-7dskx"
Jul 27 21:50:21.904020 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:50:21.903987    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5996957568-7dskx"
Jul 27 21:50:41.905323 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:50:41.905282    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5996957568-7dskx" podUID="508f765d-e01a-4506-9862-7fab9968614c" containerName="kserve-container" probeResult="failure" output="Get \"http://10.132.0.42:8080/v1/models/isvc-sklearn-mcp\": dial tcp 10.132.0.42:8080: connect: connection refused"
Jul 27 21:50:51.906303 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:50:51.906264    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5996957568-7dskx"
Jul 27 21:51:21.907396 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:51:21.907364    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5996957568-7dskx"
Jul 27 21:51:26.860390 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:51:26.860354    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5996957568-7dskx"]
Jul 27 21:51:26.860795 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:51:26.860599    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5996957568-7dskx" podUID="508f765d-e01a-4506-9862-7fab9968614c" containerName="kserve-container" containerID="cri-o://ed9aaa3f36b729e5dd481a3bbe8efff93ef52a44b0917b6e10bdaf367af1103b" gracePeriod=30
Jul 27 21:51:26.860795 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:51:26.860646    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5996957568-7dskx" podUID="508f765d-e01a-4506-9862-7fab9968614c" containerName="kube-rbac-proxy" containerID="cri-o://c01b25026b4b706a938d780d3df4f192e3ad67d3eaae34c1741674c9ffd4d4d7" gracePeriod=30
Jul 27 21:51:26.860795 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:51:26.860654    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5996957568-7dskx" podUID="508f765d-e01a-4506-9862-7fab9968614c" containerName="kserve-agent" containerID="cri-o://8ac61dda00c63ac7466d7b235c0dead4cb223f726fa937346db7da6dd390d724" gracePeriod=30
Jul 27 21:51:26.899005 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:51:26.898964    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5996957568-7dskx" podUID="508f765d-e01a-4506-9862-7fab9968614c" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.42:8643/healthz\": dial tcp 10.132.0.42:8643: connect: connection refused"
Jul 27 21:51:27.230490 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:51:27.230457    2559 generic.go:356] "Generic (PLEG): container finished" podID="508f765d-e01a-4506-9862-7fab9968614c" containerID="c01b25026b4b706a938d780d3df4f192e3ad67d3eaae34c1741674c9ffd4d4d7" exitCode=2
Jul 27 21:51:27.230659 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:51:27.230521    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5996957568-7dskx" event={"ID":"508f765d-e01a-4506-9862-7fab9968614c","Type":"ContainerDied","Data":"c01b25026b4b706a938d780d3df4f192e3ad67d3eaae34c1741674c9ffd4d4d7"}
Jul 27 21:51:29.242581 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:51:29.242552    2559 generic.go:356] "Generic (PLEG): container finished" podID="508f765d-e01a-4506-9862-7fab9968614c" containerID="ed9aaa3f36b729e5dd481a3bbe8efff93ef52a44b0917b6e10bdaf367af1103b" exitCode=0
Jul 27 21:51:29.242939 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:51:29.242618    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5996957568-7dskx" event={"ID":"508f765d-e01a-4506-9862-7fab9968614c","Type":"ContainerDied","Data":"ed9aaa3f36b729e5dd481a3bbe8efff93ef52a44b0917b6e10bdaf367af1103b"}
Jul 27 21:51:31.899194 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:51:31.899160    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5996957568-7dskx" podUID="508f765d-e01a-4506-9862-7fab9968614c" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.42:8643/healthz\": dial tcp 10.132.0.42:8643: connect: connection refused"
Jul 27 21:51:31.904595 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:51:31.904572    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5996957568-7dskx" podUID="508f765d-e01a-4506-9862-7fab9968614c" containerName="kserve-container" probeResult="failure" output="Get \"http://10.132.0.42:8080/v1/models/isvc-sklearn-mcp\": dial tcp 10.132.0.42:8080: connect: connection refused"
Jul 27 21:51:36.899149 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:51:36.899108    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5996957568-7dskx" podUID="508f765d-e01a-4506-9862-7fab9968614c" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.42:8643/healthz\": dial tcp 10.132.0.42:8643: connect: connection refused"
Jul 27 21:51:36.901404 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:51:36.899273    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5996957568-7dskx"
Jul 27 21:51:41.898987 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:51:41.898948    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5996957568-7dskx" podUID="508f765d-e01a-4506-9862-7fab9968614c" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.42:8643/healthz\": dial tcp 10.132.0.42:8643: connect: connection refused"
Jul 27 21:51:41.905427 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:51:41.905400    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5996957568-7dskx" podUID="508f765d-e01a-4506-9862-7fab9968614c" containerName="kserve-container" probeResult="failure" output="Get \"http://10.132.0.42:8080/v1/models/isvc-sklearn-mcp\": dial tcp 10.132.0.42:8080: connect: connection refused"
Jul 27 21:51:46.899295 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:51:46.899253    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5996957568-7dskx" podUID="508f765d-e01a-4506-9862-7fab9968614c" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.42:8643/healthz\": dial tcp 10.132.0.42:8643: connect: connection refused"
Jul 27 21:51:51.899529 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:51:51.899485    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5996957568-7dskx" podUID="508f765d-e01a-4506-9862-7fab9968614c" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.42:8643/healthz\": dial tcp 10.132.0.42:8643: connect: connection refused"
Jul 27 21:51:51.904960 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:51:51.904934    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5996957568-7dskx" podUID="508f765d-e01a-4506-9862-7fab9968614c" containerName="kserve-container" probeResult="failure" output="Get \"http://10.132.0.42:8080/v1/models/isvc-sklearn-mcp\": dial tcp 10.132.0.42:8080: connect: connection refused"
Jul 27 21:51:51.905082 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:51:51.905069    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5996957568-7dskx"
Jul 27 21:51:56.899049 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:51:56.899011    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5996957568-7dskx" podUID="508f765d-e01a-4506-9862-7fab9968614c" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.42:8643/healthz\": dial tcp 10.132.0.42:8643: connect: connection refused"
Jul 27 21:51:57.007619 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:51:57.007590    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5996957568-7dskx"
Jul 27 21:51:57.049006 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:51:57.048963    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/508f765d-e01a-4506-9862-7fab9968614c-proxy-tls\") pod \"508f765d-e01a-4506-9862-7fab9968614c\" (UID: \"508f765d-e01a-4506-9862-7fab9968614c\") "
Jul 27 21:51:57.049201 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:51:57.049022    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/508f765d-e01a-4506-9862-7fab9968614c-kube-rbac-proxy-sar-config\") pod \"508f765d-e01a-4506-9862-7fab9968614c\" (UID: \"508f765d-e01a-4506-9862-7fab9968614c\") "
Jul 27 21:51:57.049201 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:51:57.049053    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/508f765d-e01a-4506-9862-7fab9968614c-cabundle-cert\") pod \"508f765d-e01a-4506-9862-7fab9968614c\" (UID: \"508f765d-e01a-4506-9862-7fab9968614c\") "
Jul 27 21:51:57.049201 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:51:57.049110    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/508f765d-e01a-4506-9862-7fab9968614c-kserve-provision-location\") pod \"508f765d-e01a-4506-9862-7fab9968614c\" (UID: \"508f765d-e01a-4506-9862-7fab9968614c\") "
Jul 27 21:51:57.049201 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:51:57.049192    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gqlgf\" (UniqueName: \"kubernetes.io/projected/508f765d-e01a-4506-9862-7fab9968614c-kube-api-access-gqlgf\") pod \"508f765d-e01a-4506-9862-7fab9968614c\" (UID: \"508f765d-e01a-4506-9862-7fab9968614c\") "
Jul 27 21:51:57.049468 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:51:57.049417    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/508f765d-e01a-4506-9862-7fab9968614c-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "508f765d-e01a-4506-9862-7fab9968614c" (UID: "508f765d-e01a-4506-9862-7fab9968614c"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 21:51:57.049536 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:51:57.049435    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/508f765d-e01a-4506-9862-7fab9968614c-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "508f765d-e01a-4506-9862-7fab9968614c" (UID: "508f765d-e01a-4506-9862-7fab9968614c"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 21:51:57.049536 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:51:57.049449    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/508f765d-e01a-4506-9862-7fab9968614c-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "508f765d-e01a-4506-9862-7fab9968614c" (UID: "508f765d-e01a-4506-9862-7fab9968614c"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 21:51:57.049536 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:51:57.049503    2559 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/508f765d-e01a-4506-9862-7fab9968614c-kserve-provision-location\") on node \"ip-10-0-131-133.ec2.internal\" DevicePath \"\""
Jul 27 21:51:57.051155 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:51:57.051121    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/508f765d-e01a-4506-9862-7fab9968614c-kube-api-access-gqlgf" (OuterVolumeSpecName: "kube-api-access-gqlgf") pod "508f765d-e01a-4506-9862-7fab9968614c" (UID: "508f765d-e01a-4506-9862-7fab9968614c"). InnerVolumeSpecName "kube-api-access-gqlgf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 21:51:57.051155 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:51:57.051149    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/508f765d-e01a-4506-9862-7fab9968614c-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "508f765d-e01a-4506-9862-7fab9968614c" (UID: "508f765d-e01a-4506-9862-7fab9968614c"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 21:51:57.150824 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:51:57.150794    2559 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/508f765d-e01a-4506-9862-7fab9968614c-kube-rbac-proxy-sar-config\") on node \"ip-10-0-131-133.ec2.internal\" DevicePath \"\""
Jul 27 21:51:57.150824 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:51:57.150823    2559 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/508f765d-e01a-4506-9862-7fab9968614c-cabundle-cert\") on node \"ip-10-0-131-133.ec2.internal\" DevicePath \"\""
Jul 27 21:51:57.151010 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:51:57.150835    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gqlgf\" (UniqueName: \"kubernetes.io/projected/508f765d-e01a-4506-9862-7fab9968614c-kube-api-access-gqlgf\") on node \"ip-10-0-131-133.ec2.internal\" DevicePath \"\""
Jul 27 21:51:57.151010 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:51:57.150849    2559 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/508f765d-e01a-4506-9862-7fab9968614c-proxy-tls\") on node \"ip-10-0-131-133.ec2.internal\" DevicePath \"\""
Jul 27 21:51:57.369350 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:51:57.369316    2559 generic.go:356] "Generic (PLEG): container finished" podID="508f765d-e01a-4506-9862-7fab9968614c" containerID="8ac61dda00c63ac7466d7b235c0dead4cb223f726fa937346db7da6dd390d724" exitCode=0
Jul 27 21:51:57.369519 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:51:57.369354    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5996957568-7dskx" event={"ID":"508f765d-e01a-4506-9862-7fab9968614c","Type":"ContainerDied","Data":"8ac61dda00c63ac7466d7b235c0dead4cb223f726fa937346db7da6dd390d724"}
Jul 27 21:51:57.369519 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:51:57.369375    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5996957568-7dskx"
Jul 27 21:51:57.369519 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:51:57.369418    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5996957568-7dskx" event={"ID":"508f765d-e01a-4506-9862-7fab9968614c","Type":"ContainerDied","Data":"5013594ee133c601e26df73d25bb16b8697c17c0eb42236589b2f269e34eca2d"}
Jul 27 21:51:57.369519 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:51:57.369450    2559 scope.go:117] "RemoveContainer" containerID="c01b25026b4b706a938d780d3df4f192e3ad67d3eaae34c1741674c9ffd4d4d7"
Jul 27 21:51:57.385045 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:51:57.385028    2559 scope.go:117] "RemoveContainer" containerID="8ac61dda00c63ac7466d7b235c0dead4cb223f726fa937346db7da6dd390d724"
Jul 27 21:51:57.392619 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:51:57.392600    2559 scope.go:117] "RemoveContainer" containerID="ed9aaa3f36b729e5dd481a3bbe8efff93ef52a44b0917b6e10bdaf367af1103b"
Jul 27 21:51:57.396204 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:51:57.396176    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5996957568-7dskx"]
Jul 27 21:51:57.400288 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:51:57.400208    2559 scope.go:117] "RemoveContainer" containerID="20af4bb2cec8b29e07afd4b9b23698cfd665cdbbce88354a572f6ff9845f73c3"
Jul 27 21:51:57.402104 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:51:57.402086    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5996957568-7dskx"]
Jul 27 21:51:57.407392 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:51:57.407371    2559 scope.go:117] "RemoveContainer" containerID="c01b25026b4b706a938d780d3df4f192e3ad67d3eaae34c1741674c9ffd4d4d7"
Jul 27 21:51:57.407629 ip-10-0-131-133 kubenswrapper[2559]: E0727 21:51:57.407609    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c01b25026b4b706a938d780d3df4f192e3ad67d3eaae34c1741674c9ffd4d4d7\": container with ID starting with c01b25026b4b706a938d780d3df4f192e3ad67d3eaae34c1741674c9ffd4d4d7 not found: ID does not exist" containerID="c01b25026b4b706a938d780d3df4f192e3ad67d3eaae34c1741674c9ffd4d4d7"
Jul 27 21:51:57.407700 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:51:57.407642    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c01b25026b4b706a938d780d3df4f192e3ad67d3eaae34c1741674c9ffd4d4d7"} err="failed to get container status \"c01b25026b4b706a938d780d3df4f192e3ad67d3eaae34c1741674c9ffd4d4d7\": rpc error: code = NotFound desc = could not find container \"c01b25026b4b706a938d780d3df4f192e3ad67d3eaae34c1741674c9ffd4d4d7\": container with ID starting with c01b25026b4b706a938d780d3df4f192e3ad67d3eaae34c1741674c9ffd4d4d7 not found: ID does not exist"
Jul 27 21:51:57.407700 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:51:57.407667    2559 scope.go:117] "RemoveContainer" containerID="8ac61dda00c63ac7466d7b235c0dead4cb223f726fa937346db7da6dd390d724"
Jul 27 21:51:57.407903 ip-10-0-131-133 kubenswrapper[2559]: E0727 21:51:57.407886    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8ac61dda00c63ac7466d7b235c0dead4cb223f726fa937346db7da6dd390d724\": container with ID starting with 8ac61dda00c63ac7466d7b235c0dead4cb223f726fa937346db7da6dd390d724 not found: ID does not exist" containerID="8ac61dda00c63ac7466d7b235c0dead4cb223f726fa937346db7da6dd390d724"
Jul 27 21:51:57.407942 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:51:57.407909    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8ac61dda00c63ac7466d7b235c0dead4cb223f726fa937346db7da6dd390d724"} err="failed to get container status \"8ac61dda00c63ac7466d7b235c0dead4cb223f726fa937346db7da6dd390d724\": rpc error: code = NotFound desc = could not find container \"8ac61dda00c63ac7466d7b235c0dead4cb223f726fa937346db7da6dd390d724\": container with ID starting with 8ac61dda00c63ac7466d7b235c0dead4cb223f726fa937346db7da6dd390d724 not found: ID does not exist"
Jul 27 21:51:57.407942 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:51:57.407924    2559 scope.go:117] "RemoveContainer" containerID="ed9aaa3f36b729e5dd481a3bbe8efff93ef52a44b0917b6e10bdaf367af1103b"
Jul 27 21:51:57.408116 ip-10-0-131-133 kubenswrapper[2559]: E0727 21:51:57.408101    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ed9aaa3f36b729e5dd481a3bbe8efff93ef52a44b0917b6e10bdaf367af1103b\": container with ID starting with ed9aaa3f36b729e5dd481a3bbe8efff93ef52a44b0917b6e10bdaf367af1103b not found: ID does not exist" containerID="ed9aaa3f36b729e5dd481a3bbe8efff93ef52a44b0917b6e10bdaf367af1103b"
Jul 27 21:51:57.408158 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:51:57.408123    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ed9aaa3f36b729e5dd481a3bbe8efff93ef52a44b0917b6e10bdaf367af1103b"} err="failed to get container status \"ed9aaa3f36b729e5dd481a3bbe8efff93ef52a44b0917b6e10bdaf367af1103b\": rpc error: code = NotFound desc = could not find container \"ed9aaa3f36b729e5dd481a3bbe8efff93ef52a44b0917b6e10bdaf367af1103b\": container with ID starting with ed9aaa3f36b729e5dd481a3bbe8efff93ef52a44b0917b6e10bdaf367af1103b not found: ID does not exist"
Jul 27 21:51:57.408158 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:51:57.408137    2559 scope.go:117] "RemoveContainer" containerID="20af4bb2cec8b29e07afd4b9b23698cfd665cdbbce88354a572f6ff9845f73c3"
Jul 27 21:51:57.408337 ip-10-0-131-133 kubenswrapper[2559]: E0727 21:51:57.408319    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"20af4bb2cec8b29e07afd4b9b23698cfd665cdbbce88354a572f6ff9845f73c3\": container with ID starting with 20af4bb2cec8b29e07afd4b9b23698cfd665cdbbce88354a572f6ff9845f73c3 not found: ID does not exist" containerID="20af4bb2cec8b29e07afd4b9b23698cfd665cdbbce88354a572f6ff9845f73c3"
Jul 27 21:51:57.408384 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:51:57.408341    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"20af4bb2cec8b29e07afd4b9b23698cfd665cdbbce88354a572f6ff9845f73c3"} err="failed to get container status \"20af4bb2cec8b29e07afd4b9b23698cfd665cdbbce88354a572f6ff9845f73c3\": rpc error: code = NotFound desc = could not find container \"20af4bb2cec8b29e07afd4b9b23698cfd665cdbbce88354a572f6ff9845f73c3\": container with ID starting with 20af4bb2cec8b29e07afd4b9b23698cfd665cdbbce88354a572f6ff9845f73c3 not found: ID does not exist"
Jul 27 21:51:57.981307 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:51:57.981280    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="508f765d-e01a-4506-9862-7fab9968614c" path="/var/lib/kubelet/pods/508f765d-e01a-4506-9862-7fab9968614c/volumes"
Jul 27 21:52:48.817897 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:52:48.817858    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-5zfkc"]
Jul 27 21:52:48.818354 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:52:48.818235    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="508f765d-e01a-4506-9862-7fab9968614c" containerName="kserve-agent"
Jul 27 21:52:48.818354 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:52:48.818251    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="508f765d-e01a-4506-9862-7fab9968614c" containerName="kserve-agent"
Jul 27 21:52:48.818354 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:52:48.818267    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="508f765d-e01a-4506-9862-7fab9968614c" containerName="kserve-container"
Jul 27 21:52:48.818354 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:52:48.818273    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="508f765d-e01a-4506-9862-7fab9968614c" containerName="kserve-container"
Jul 27 21:52:48.818354 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:52:48.818280    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="508f765d-e01a-4506-9862-7fab9968614c" containerName="kube-rbac-proxy"
Jul 27 21:52:48.818354 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:52:48.818286    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="508f765d-e01a-4506-9862-7fab9968614c" containerName="kube-rbac-proxy"
Jul 27 21:52:48.818354 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:52:48.818301    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="508f765d-e01a-4506-9862-7fab9968614c" containerName="storage-initializer"
Jul 27 21:52:48.818354 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:52:48.818307    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="508f765d-e01a-4506-9862-7fab9968614c" containerName="storage-initializer"
Jul 27 21:52:48.818605 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:52:48.818363    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="508f765d-e01a-4506-9862-7fab9968614c" containerName="kube-rbac-proxy"
Jul 27 21:52:48.818605 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:52:48.818372    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="508f765d-e01a-4506-9862-7fab9968614c" containerName="kserve-agent"
Jul 27 21:52:48.818605 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:52:48.818377    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="508f765d-e01a-4506-9862-7fab9968614c" containerName="kserve-container"
Jul 27 21:52:48.821809 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:52:48.821788    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-5zfkc"
Jul 27 21:52:48.824114 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:52:48.824090    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 27 21:52:48.824318 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:52:48.824228    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 27 21:52:48.824318 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:52:48.824238    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-paddle-runtime-kube-rbac-proxy-sar-config\""
Jul 27 21:52:48.824318 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:52:48.824246    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-v7c66\""
Jul 27 21:52:48.824318 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:52:48.824282    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-paddle-runtime-predictor-serving-cert\""
Jul 27 21:52:48.824616 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:52:48.824599    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 27 21:52:48.824756 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:52:48.824730    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 27 21:52:48.832073 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:52:48.832051    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-5zfkc"]
Jul 27 21:52:48.915350 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:52:48.915308    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gn6lp\" (UniqueName: \"kubernetes.io/projected/c2876ecd-92c1-46fd-8960-efc2df67f2d5-kube-api-access-gn6lp\") pod \"isvc-paddle-runtime-predictor-58b99fff76-5zfkc\" (UID: \"c2876ecd-92c1-46fd-8960-efc2df67f2d5\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-5zfkc"
Jul 27 21:52:48.915350 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:52:48.915356    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c2876ecd-92c1-46fd-8960-efc2df67f2d5-proxy-tls\") pod \"isvc-paddle-runtime-predictor-58b99fff76-5zfkc\" (UID: \"c2876ecd-92c1-46fd-8960-efc2df67f2d5\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-5zfkc"
Jul 27 21:52:48.915596 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:52:48.915382    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c2876ecd-92c1-46fd-8960-efc2df67f2d5-cabundle-cert\") pod \"isvc-paddle-runtime-predictor-58b99fff76-5zfkc\" (UID: \"c2876ecd-92c1-46fd-8960-efc2df67f2d5\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-5zfkc"
Jul 27 21:52:48.915596 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:52:48.915455    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c2876ecd-92c1-46fd-8960-efc2df67f2d5-kube-rbac-proxy-sar-config\") pod \"isvc-paddle-runtime-predictor-58b99fff76-5zfkc\" (UID: \"c2876ecd-92c1-46fd-8960-efc2df67f2d5\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-5zfkc"
Jul 27 21:52:48.915596 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:52:48.915506    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c2876ecd-92c1-46fd-8960-efc2df67f2d5-kserve-provision-location\") pod \"isvc-paddle-runtime-predictor-58b99fff76-5zfkc\" (UID: \"c2876ecd-92c1-46fd-8960-efc2df67f2d5\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-5zfkc"
Jul 27 21:52:49.016152 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:52:49.016109    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c2876ecd-92c1-46fd-8960-efc2df67f2d5-kserve-provision-location\") pod \"isvc-paddle-runtime-predictor-58b99fff76-5zfkc\" (UID: \"c2876ecd-92c1-46fd-8960-efc2df67f2d5\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-5zfkc"
Jul 27 21:52:49.016366 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:52:49.016280    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gn6lp\" (UniqueName: \"kubernetes.io/projected/c2876ecd-92c1-46fd-8960-efc2df67f2d5-kube-api-access-gn6lp\") pod \"isvc-paddle-runtime-predictor-58b99fff76-5zfkc\" (UID: \"c2876ecd-92c1-46fd-8960-efc2df67f2d5\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-5zfkc"
Jul 27 21:52:49.016366 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:52:49.016322    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c2876ecd-92c1-46fd-8960-efc2df67f2d5-proxy-tls\") pod \"isvc-paddle-runtime-predictor-58b99fff76-5zfkc\" (UID: \"c2876ecd-92c1-46fd-8960-efc2df67f2d5\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-5zfkc"
Jul 27 21:52:49.016494 ip-10-0-131-133 kubenswrapper[2559]: E0727 21:52:49.016398    2559 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-paddle-runtime-predictor-serving-cert: secret "isvc-paddle-runtime-predictor-serving-cert" not found
Jul 27 21:52:49.016494 ip-10-0-131-133 kubenswrapper[2559]: E0727 21:52:49.016449    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c2876ecd-92c1-46fd-8960-efc2df67f2d5-proxy-tls podName:c2876ecd-92c1-46fd-8960-efc2df67f2d5 nodeName:}" failed. No retries permitted until 2026-07-27 21:52:49.516435371 +0000 UTC m=+2898.095619355 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/c2876ecd-92c1-46fd-8960-efc2df67f2d5-proxy-tls") pod "isvc-paddle-runtime-predictor-58b99fff76-5zfkc" (UID: "c2876ecd-92c1-46fd-8960-efc2df67f2d5") : secret "isvc-paddle-runtime-predictor-serving-cert" not found
Jul 27 21:52:49.016494 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:52:49.016474    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c2876ecd-92c1-46fd-8960-efc2df67f2d5-cabundle-cert\") pod \"isvc-paddle-runtime-predictor-58b99fff76-5zfkc\" (UID: \"c2876ecd-92c1-46fd-8960-efc2df67f2d5\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-5zfkc"
Jul 27 21:52:49.016637 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:52:49.016519    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c2876ecd-92c1-46fd-8960-efc2df67f2d5-kserve-provision-location\") pod \"isvc-paddle-runtime-predictor-58b99fff76-5zfkc\" (UID: \"c2876ecd-92c1-46fd-8960-efc2df67f2d5\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-5zfkc"
Jul 27 21:52:49.016637 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:52:49.016540    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c2876ecd-92c1-46fd-8960-efc2df67f2d5-kube-rbac-proxy-sar-config\") pod \"isvc-paddle-runtime-predictor-58b99fff76-5zfkc\" (UID: \"c2876ecd-92c1-46fd-8960-efc2df67f2d5\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-5zfkc"
Jul 27 21:52:49.017177 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:52:49.017154    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c2876ecd-92c1-46fd-8960-efc2df67f2d5-kube-rbac-proxy-sar-config\") pod \"isvc-paddle-runtime-predictor-58b99fff76-5zfkc\" (UID: \"c2876ecd-92c1-46fd-8960-efc2df67f2d5\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-5zfkc"
Jul 27 21:52:49.017256 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:52:49.017154    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c2876ecd-92c1-46fd-8960-efc2df67f2d5-cabundle-cert\") pod \"isvc-paddle-runtime-predictor-58b99fff76-5zfkc\" (UID: \"c2876ecd-92c1-46fd-8960-efc2df67f2d5\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-5zfkc"
Jul 27 21:52:49.024973 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:52:49.024954    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gn6lp\" (UniqueName: \"kubernetes.io/projected/c2876ecd-92c1-46fd-8960-efc2df67f2d5-kube-api-access-gn6lp\") pod \"isvc-paddle-runtime-predictor-58b99fff76-5zfkc\" (UID: \"c2876ecd-92c1-46fd-8960-efc2df67f2d5\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-5zfkc"
Jul 27 21:52:49.522303 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:52:49.522260    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c2876ecd-92c1-46fd-8960-efc2df67f2d5-proxy-tls\") pod \"isvc-paddle-runtime-predictor-58b99fff76-5zfkc\" (UID: \"c2876ecd-92c1-46fd-8960-efc2df67f2d5\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-5zfkc"
Jul 27 21:52:49.524625 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:52:49.524596    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c2876ecd-92c1-46fd-8960-efc2df67f2d5-proxy-tls\") pod \"isvc-paddle-runtime-predictor-58b99fff76-5zfkc\" (UID: \"c2876ecd-92c1-46fd-8960-efc2df67f2d5\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-5zfkc"
Jul 27 21:52:49.732738 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:52:49.732700    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-5zfkc"
Jul 27 21:52:49.867448 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:52:49.867422    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-5zfkc"]
Jul 27 21:52:49.869610 ip-10-0-131-133 kubenswrapper[2559]: W0727 21:52:49.869556    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc2876ecd_92c1_46fd_8960_efc2df67f2d5.slice/crio-b5b1ce9f1d84895945b511f30844cac43e923183651c539d3489ce5e197d46fb WatchSource:0}: Error finding container b5b1ce9f1d84895945b511f30844cac43e923183651c539d3489ce5e197d46fb: Status 404 returned error can't find the container with id b5b1ce9f1d84895945b511f30844cac43e923183651c539d3489ce5e197d46fb
Jul 27 21:52:50.610172 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:52:50.610131    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-5zfkc" event={"ID":"c2876ecd-92c1-46fd-8960-efc2df67f2d5","Type":"ContainerStarted","Data":"78ff3667122e95cd56b521c5c115447b27d629bfce2b48ce31a5852d0a213173"}
Jul 27 21:52:50.610363 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:52:50.610232    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-5zfkc" event={"ID":"c2876ecd-92c1-46fd-8960-efc2df67f2d5","Type":"ContainerStarted","Data":"b5b1ce9f1d84895945b511f30844cac43e923183651c539d3489ce5e197d46fb"}
Jul 27 21:52:55.635100 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:52:55.635067    2559 generic.go:356] "Generic (PLEG): container finished" podID="c2876ecd-92c1-46fd-8960-efc2df67f2d5" containerID="78ff3667122e95cd56b521c5c115447b27d629bfce2b48ce31a5852d0a213173" exitCode=0
Jul 27 21:52:55.635516 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:52:55.635123    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-5zfkc" event={"ID":"c2876ecd-92c1-46fd-8960-efc2df67f2d5","Type":"ContainerDied","Data":"78ff3667122e95cd56b521c5c115447b27d629bfce2b48ce31a5852d0a213173"}
Jul 27 21:53:07.709146 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:53:07.709109    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-5zfkc" event={"ID":"c2876ecd-92c1-46fd-8960-efc2df67f2d5","Type":"ContainerStarted","Data":"fcbf59b9a269b2c99f2967a0a728b1b7f83b55f3665bf5b9eec239a30669d667"}
Jul 27 21:53:07.709614 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:53:07.709175    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-5zfkc" event={"ID":"c2876ecd-92c1-46fd-8960-efc2df67f2d5","Type":"ContainerStarted","Data":"219ffa447e576e8c187fadbf26b3928d1f4e8e7076d084e854576fcd2598ff4d"}
Jul 27 21:53:07.709614 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:53:07.709388    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-5zfkc"
Jul 27 21:53:07.728612 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:53:07.728562    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-5zfkc" podStartSLOduration=3.578455549 podStartE2EDuration="19.728548056s" totalImagesPullingTime="11.059910413s" totalInitContainerRuntime="5.090182094s" isStatefulPod=false podCreationTimestamp="2026-07-27 21:52:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 21:53:07.727701737 +0000 UTC m=+2916.306885746" watchObservedRunningTime="2026-07-27 21:53:07.728548056 +0000 UTC m=+2916.307732062"
Jul 27 21:53:08.714400 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:53:08.714356    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-5zfkc"
Jul 27 21:53:08.715564 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:53:08.715541    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-5zfkc" podUID="c2876ecd-92c1-46fd-8960-efc2df67f2d5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.43:8080: connect: connection refused"
Jul 27 21:53:09.719269 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:53:09.719232    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-5zfkc" podUID="c2876ecd-92c1-46fd-8960-efc2df67f2d5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.43:8080: connect: connection refused"
Jul 27 21:53:14.723701 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:53:14.723672    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-5zfkc"
Jul 27 21:53:14.724157 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:53:14.724132    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-5zfkc" podUID="c2876ecd-92c1-46fd-8960-efc2df67f2d5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.43:8080: connect: connection refused"
Jul 27 21:53:24.724424 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:53:24.724385    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-5zfkc" podUID="c2876ecd-92c1-46fd-8960-efc2df67f2d5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.43:8080: connect: connection refused"
Jul 27 21:53:34.724497 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:53:34.724449    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-5zfkc" podUID="c2876ecd-92c1-46fd-8960-efc2df67f2d5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.43:8080: connect: connection refused"
Jul 27 21:53:44.725052 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:53:44.725009    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-5zfkc" podUID="c2876ecd-92c1-46fd-8960-efc2df67f2d5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.43:8080: connect: connection refused"
Jul 27 21:53:54.724916 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:53:54.724883    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-5zfkc"
Jul 27 21:54:00.426412 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:54:00.426239    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-5zfkc"]
Jul 27 21:54:00.427011 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:54:00.426679    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-5zfkc" podUID="c2876ecd-92c1-46fd-8960-efc2df67f2d5" containerName="kserve-container" containerID="cri-o://219ffa447e576e8c187fadbf26b3928d1f4e8e7076d084e854576fcd2598ff4d" gracePeriod=30
Jul 27 21:54:00.427011 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:54:00.426701    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-5zfkc" podUID="c2876ecd-92c1-46fd-8960-efc2df67f2d5" containerName="kube-rbac-proxy" containerID="cri-o://fcbf59b9a269b2c99f2967a0a728b1b7f83b55f3665bf5b9eec239a30669d667" gracePeriod=30
Jul 27 21:54:00.968350 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:54:00.968315    2559 generic.go:356] "Generic (PLEG): container finished" podID="c2876ecd-92c1-46fd-8960-efc2df67f2d5" containerID="fcbf59b9a269b2c99f2967a0a728b1b7f83b55f3665bf5b9eec239a30669d667" exitCode=2
Jul 27 21:54:00.968529 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:54:00.968368    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-5zfkc" event={"ID":"c2876ecd-92c1-46fd-8960-efc2df67f2d5","Type":"ContainerDied","Data":"fcbf59b9a269b2c99f2967a0a728b1b7f83b55f3665bf5b9eec239a30669d667"}
Jul 27 21:54:03.380810 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:54:03.380784    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-5zfkc"
Jul 27 21:54:03.494806 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:54:03.494713    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c2876ecd-92c1-46fd-8960-efc2df67f2d5-kube-rbac-proxy-sar-config\") pod \"c2876ecd-92c1-46fd-8960-efc2df67f2d5\" (UID: \"c2876ecd-92c1-46fd-8960-efc2df67f2d5\") "
Jul 27 21:54:03.494806 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:54:03.494762    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c2876ecd-92c1-46fd-8960-efc2df67f2d5-proxy-tls\") pod \"c2876ecd-92c1-46fd-8960-efc2df67f2d5\" (UID: \"c2876ecd-92c1-46fd-8960-efc2df67f2d5\") "
Jul 27 21:54:03.494806 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:54:03.494803    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c2876ecd-92c1-46fd-8960-efc2df67f2d5-kserve-provision-location\") pod \"c2876ecd-92c1-46fd-8960-efc2df67f2d5\" (UID: \"c2876ecd-92c1-46fd-8960-efc2df67f2d5\") "
Jul 27 21:54:03.495114 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:54:03.494911    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c2876ecd-92c1-46fd-8960-efc2df67f2d5-cabundle-cert\") pod \"c2876ecd-92c1-46fd-8960-efc2df67f2d5\" (UID: \"c2876ecd-92c1-46fd-8960-efc2df67f2d5\") "
Jul 27 21:54:03.495114 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:54:03.495080    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gn6lp\" (UniqueName: \"kubernetes.io/projected/c2876ecd-92c1-46fd-8960-efc2df67f2d5-kube-api-access-gn6lp\") pod \"c2876ecd-92c1-46fd-8960-efc2df67f2d5\" (UID: \"c2876ecd-92c1-46fd-8960-efc2df67f2d5\") "
Jul 27 21:54:03.495247 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:54:03.495110    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c2876ecd-92c1-46fd-8960-efc2df67f2d5-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "c2876ecd-92c1-46fd-8960-efc2df67f2d5" (UID: "c2876ecd-92c1-46fd-8960-efc2df67f2d5"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 21:54:03.495310 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:54:03.495292    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c2876ecd-92c1-46fd-8960-efc2df67f2d5-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "c2876ecd-92c1-46fd-8960-efc2df67f2d5" (UID: "c2876ecd-92c1-46fd-8960-efc2df67f2d5"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 21:54:03.495418 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:54:03.495399    2559 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c2876ecd-92c1-46fd-8960-efc2df67f2d5-kube-rbac-proxy-sar-config\") on node \"ip-10-0-131-133.ec2.internal\" DevicePath \"\""
Jul 27 21:54:03.495471 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:54:03.495418    2559 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c2876ecd-92c1-46fd-8960-efc2df67f2d5-cabundle-cert\") on node \"ip-10-0-131-133.ec2.internal\" DevicePath \"\""
Jul 27 21:54:03.497034 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:54:03.497011    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c2876ecd-92c1-46fd-8960-efc2df67f2d5-kube-api-access-gn6lp" (OuterVolumeSpecName: "kube-api-access-gn6lp") pod "c2876ecd-92c1-46fd-8960-efc2df67f2d5" (UID: "c2876ecd-92c1-46fd-8960-efc2df67f2d5"). InnerVolumeSpecName "kube-api-access-gn6lp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 21:54:03.497034 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:54:03.497028    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c2876ecd-92c1-46fd-8960-efc2df67f2d5-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "c2876ecd-92c1-46fd-8960-efc2df67f2d5" (UID: "c2876ecd-92c1-46fd-8960-efc2df67f2d5"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 21:54:03.504258 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:54:03.504232    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c2876ecd-92c1-46fd-8960-efc2df67f2d5-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "c2876ecd-92c1-46fd-8960-efc2df67f2d5" (UID: "c2876ecd-92c1-46fd-8960-efc2df67f2d5"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 21:54:03.596351 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:54:03.596313    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gn6lp\" (UniqueName: \"kubernetes.io/projected/c2876ecd-92c1-46fd-8960-efc2df67f2d5-kube-api-access-gn6lp\") on node \"ip-10-0-131-133.ec2.internal\" DevicePath \"\""
Jul 27 21:54:03.596351 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:54:03.596347    2559 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c2876ecd-92c1-46fd-8960-efc2df67f2d5-proxy-tls\") on node \"ip-10-0-131-133.ec2.internal\" DevicePath \"\""
Jul 27 21:54:03.596582 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:54:03.596360    2559 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c2876ecd-92c1-46fd-8960-efc2df67f2d5-kserve-provision-location\") on node \"ip-10-0-131-133.ec2.internal\" DevicePath \"\""
Jul 27 21:54:03.986054 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:54:03.986011    2559 generic.go:356] "Generic (PLEG): container finished" podID="c2876ecd-92c1-46fd-8960-efc2df67f2d5" containerID="219ffa447e576e8c187fadbf26b3928d1f4e8e7076d084e854576fcd2598ff4d" exitCode=0
Jul 27 21:54:03.986253 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:54:03.986078    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-5zfkc"
Jul 27 21:54:03.986253 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:54:03.986092    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-5zfkc" event={"ID":"c2876ecd-92c1-46fd-8960-efc2df67f2d5","Type":"ContainerDied","Data":"219ffa447e576e8c187fadbf26b3928d1f4e8e7076d084e854576fcd2598ff4d"}
Jul 27 21:54:03.986253 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:54:03.986158    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-5zfkc" event={"ID":"c2876ecd-92c1-46fd-8960-efc2df67f2d5","Type":"ContainerDied","Data":"b5b1ce9f1d84895945b511f30844cac43e923183651c539d3489ce5e197d46fb"}
Jul 27 21:54:03.986253 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:54:03.986196    2559 scope.go:117] "RemoveContainer" containerID="fcbf59b9a269b2c99f2967a0a728b1b7f83b55f3665bf5b9eec239a30669d667"
Jul 27 21:54:03.995974 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:54:03.995954    2559 scope.go:117] "RemoveContainer" containerID="219ffa447e576e8c187fadbf26b3928d1f4e8e7076d084e854576fcd2598ff4d"
Jul 27 21:54:04.004196 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:54:04.004170    2559 scope.go:117] "RemoveContainer" containerID="78ff3667122e95cd56b521c5c115447b27d629bfce2b48ce31a5852d0a213173"
Jul 27 21:54:04.005205 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:54:04.005188    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-5zfkc"]
Jul 27 21:54:04.009141 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:54:04.009120    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-5zfkc"]
Jul 27 21:54:04.012462 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:54:04.012438    2559 scope.go:117] "RemoveContainer" containerID="fcbf59b9a269b2c99f2967a0a728b1b7f83b55f3665bf5b9eec239a30669d667"
Jul 27 21:54:04.012759 ip-10-0-131-133 kubenswrapper[2559]: E0727 21:54:04.012740    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fcbf59b9a269b2c99f2967a0a728b1b7f83b55f3665bf5b9eec239a30669d667\": container with ID starting with fcbf59b9a269b2c99f2967a0a728b1b7f83b55f3665bf5b9eec239a30669d667 not found: ID does not exist" containerID="fcbf59b9a269b2c99f2967a0a728b1b7f83b55f3665bf5b9eec239a30669d667"
Jul 27 21:54:04.012814 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:54:04.012770    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fcbf59b9a269b2c99f2967a0a728b1b7f83b55f3665bf5b9eec239a30669d667"} err="failed to get container status \"fcbf59b9a269b2c99f2967a0a728b1b7f83b55f3665bf5b9eec239a30669d667\": rpc error: code = NotFound desc = could not find container \"fcbf59b9a269b2c99f2967a0a728b1b7f83b55f3665bf5b9eec239a30669d667\": container with ID starting with fcbf59b9a269b2c99f2967a0a728b1b7f83b55f3665bf5b9eec239a30669d667 not found: ID does not exist"
Jul 27 21:54:04.012814 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:54:04.012788    2559 scope.go:117] "RemoveContainer" containerID="219ffa447e576e8c187fadbf26b3928d1f4e8e7076d084e854576fcd2598ff4d"
Jul 27 21:54:04.013009 ip-10-0-131-133 kubenswrapper[2559]: E0727 21:54:04.012991    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"219ffa447e576e8c187fadbf26b3928d1f4e8e7076d084e854576fcd2598ff4d\": container with ID starting with 219ffa447e576e8c187fadbf26b3928d1f4e8e7076d084e854576fcd2598ff4d not found: ID does not exist" containerID="219ffa447e576e8c187fadbf26b3928d1f4e8e7076d084e854576fcd2598ff4d"
Jul 27 21:54:04.013046 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:54:04.013012    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"219ffa447e576e8c187fadbf26b3928d1f4e8e7076d084e854576fcd2598ff4d"} err="failed to get container status \"219ffa447e576e8c187fadbf26b3928d1f4e8e7076d084e854576fcd2598ff4d\": rpc error: code = NotFound desc = could not find container \"219ffa447e576e8c187fadbf26b3928d1f4e8e7076d084e854576fcd2598ff4d\": container with ID starting with 219ffa447e576e8c187fadbf26b3928d1f4e8e7076d084e854576fcd2598ff4d not found: ID does not exist"
Jul 27 21:54:04.013046 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:54:04.013025    2559 scope.go:117] "RemoveContainer" containerID="78ff3667122e95cd56b521c5c115447b27d629bfce2b48ce31a5852d0a213173"
Jul 27 21:54:04.013254 ip-10-0-131-133 kubenswrapper[2559]: E0727 21:54:04.013237    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"78ff3667122e95cd56b521c5c115447b27d629bfce2b48ce31a5852d0a213173\": container with ID starting with 78ff3667122e95cd56b521c5c115447b27d629bfce2b48ce31a5852d0a213173 not found: ID does not exist" containerID="78ff3667122e95cd56b521c5c115447b27d629bfce2b48ce31a5852d0a213173"
Jul 27 21:54:04.013307 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:54:04.013260    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"78ff3667122e95cd56b521c5c115447b27d629bfce2b48ce31a5852d0a213173"} err="failed to get container status \"78ff3667122e95cd56b521c5c115447b27d629bfce2b48ce31a5852d0a213173\": rpc error: code = NotFound desc = could not find container \"78ff3667122e95cd56b521c5c115447b27d629bfce2b48ce31a5852d0a213173\": container with ID starting with 78ff3667122e95cd56b521c5c115447b27d629bfce2b48ce31a5852d0a213173 not found: ID does not exist"
Jul 27 21:54:05.982078 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:54:05.982037    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c2876ecd-92c1-46fd-8960-efc2df67f2d5" path="/var/lib/kubelet/pods/c2876ecd-92c1-46fd-8960-efc2df67f2d5/volumes"
Jul 27 21:54:32.127151 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:54:32.127125    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-bjvbc_cba53c9a-64c0-4afa-8731-35521e196622/console-operator/1.log"
Jul 27 21:54:32.134851 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:54:32.134831    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-bjvbc_cba53c9a-64c0-4afa-8731-35521e196622/console-operator/1.log"
Jul 27 21:59:32.157352 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:59:32.157323    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-bjvbc_cba53c9a-64c0-4afa-8731-35521e196622/console-operator/1.log"
Jul 27 21:59:32.167721 ip-10-0-131-133 kubenswrapper[2559]: I0727 21:59:32.167694    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-bjvbc_cba53c9a-64c0-4afa-8731-35521e196622/console-operator/1.log"
Jul 27 22:04:32.188317 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:04:32.188286    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-bjvbc_cba53c9a-64c0-4afa-8731-35521e196622/console-operator/1.log"
Jul 27 22:04:32.196246 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:04:32.196225    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-bjvbc_cba53c9a-64c0-4afa-8731-35521e196622/console-operator/1.log"
Jul 27 22:09:32.221981 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:09:32.221953    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-bjvbc_cba53c9a-64c0-4afa-8731-35521e196622/console-operator/1.log"
Jul 27 22:09:32.234109 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:09:32.234087    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-bjvbc_cba53c9a-64c0-4afa-8731-35521e196622/console-operator/1.log"
Jul 27 22:14:32.261552 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:14:32.261514    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-bjvbc_cba53c9a-64c0-4afa-8731-35521e196622/console-operator/1.log"
Jul 27 22:14:32.269490 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:14:32.269465    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-bjvbc_cba53c9a-64c0-4afa-8731-35521e196622/console-operator/1.log"
Jul 27 22:19:32.294383 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:19:32.294303    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-bjvbc_cba53c9a-64c0-4afa-8731-35521e196622/console-operator/1.log"
Jul 27 22:19:32.301882 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:19:32.301861    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-bjvbc_cba53c9a-64c0-4afa-8731-35521e196622/console-operator/1.log"
Jul 27 22:24:32.324949 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:24:32.324920    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-bjvbc_cba53c9a-64c0-4afa-8731-35521e196622/console-operator/1.log"
Jul 27 22:24:32.339280 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:24:32.339251    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-bjvbc_cba53c9a-64c0-4afa-8731-35521e196622/console-operator/1.log"
Jul 27 22:27:22.594604 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:27:22.594568    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-czpmk"]
Jul 27 22:27:22.595087 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:27:22.594922    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c2876ecd-92c1-46fd-8960-efc2df67f2d5" containerName="kube-rbac-proxy"
Jul 27 22:27:22.595087 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:27:22.594933    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="c2876ecd-92c1-46fd-8960-efc2df67f2d5" containerName="kube-rbac-proxy"
Jul 27 22:27:22.595087 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:27:22.594958    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c2876ecd-92c1-46fd-8960-efc2df67f2d5" containerName="storage-initializer"
Jul 27 22:27:22.595087 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:27:22.594964    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="c2876ecd-92c1-46fd-8960-efc2df67f2d5" containerName="storage-initializer"
Jul 27 22:27:22.595087 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:27:22.594975    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c2876ecd-92c1-46fd-8960-efc2df67f2d5" containerName="kserve-container"
Jul 27 22:27:22.595087 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:27:22.594981    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="c2876ecd-92c1-46fd-8960-efc2df67f2d5" containerName="kserve-container"
Jul 27 22:27:22.595087 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:27:22.595049    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="c2876ecd-92c1-46fd-8960-efc2df67f2d5" containerName="kserve-container"
Jul 27 22:27:22.595087 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:27:22.595058    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="c2876ecd-92c1-46fd-8960-efc2df67f2d5" containerName="kube-rbac-proxy"
Jul 27 22:27:22.598322 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:27:22.598300    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-czpmk"
Jul 27 22:27:22.602073 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:27:22.602047    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-runtime-predictor-serving-cert\""
Jul 27 22:27:22.602073 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:27:22.602062    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 27 22:27:22.602275 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:27:22.602085    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 27 22:27:22.602411 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:27:22.602395    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 27 22:27:22.602490 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:27:22.602399    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-runtime-kube-rbac-proxy-sar-config\""
Jul 27 22:27:22.602776 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:27:22.602762    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-v7c66\""
Jul 27 22:27:22.602819 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:27:22.602787    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 27 22:27:22.612393 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:27:22.612043    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-czpmk"]
Jul 27 22:27:22.663500 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:27:22.663461    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/604dd384-dce7-4465-8c5f-5c75133f8156-proxy-tls\") pod \"isvc-xgboost-runtime-predictor-54dd488c85-czpmk\" (UID: \"604dd384-dce7-4465-8c5f-5c75133f8156\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-czpmk"
Jul 27 22:27:22.663686 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:27:22.663522    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m5ft6\" (UniqueName: \"kubernetes.io/projected/604dd384-dce7-4465-8c5f-5c75133f8156-kube-api-access-m5ft6\") pod \"isvc-xgboost-runtime-predictor-54dd488c85-czpmk\" (UID: \"604dd384-dce7-4465-8c5f-5c75133f8156\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-czpmk"
Jul 27 22:27:22.663686 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:27:22.663622    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/604dd384-dce7-4465-8c5f-5c75133f8156-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-runtime-predictor-54dd488c85-czpmk\" (UID: \"604dd384-dce7-4465-8c5f-5c75133f8156\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-czpmk"
Jul 27 22:27:22.663686 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:27:22.663658    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/604dd384-dce7-4465-8c5f-5c75133f8156-cabundle-cert\") pod \"isvc-xgboost-runtime-predictor-54dd488c85-czpmk\" (UID: \"604dd384-dce7-4465-8c5f-5c75133f8156\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-czpmk"
Jul 27 22:27:22.663793 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:27:22.663691    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/604dd384-dce7-4465-8c5f-5c75133f8156-kserve-provision-location\") pod \"isvc-xgboost-runtime-predictor-54dd488c85-czpmk\" (UID: \"604dd384-dce7-4465-8c5f-5c75133f8156\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-czpmk"
Jul 27 22:27:22.764353 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:27:22.764320    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/604dd384-dce7-4465-8c5f-5c75133f8156-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-runtime-predictor-54dd488c85-czpmk\" (UID: \"604dd384-dce7-4465-8c5f-5c75133f8156\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-czpmk"
Jul 27 22:27:22.764353 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:27:22.764360    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/604dd384-dce7-4465-8c5f-5c75133f8156-cabundle-cert\") pod \"isvc-xgboost-runtime-predictor-54dd488c85-czpmk\" (UID: \"604dd384-dce7-4465-8c5f-5c75133f8156\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-czpmk"
Jul 27 22:27:22.764620 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:27:22.764384    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/604dd384-dce7-4465-8c5f-5c75133f8156-kserve-provision-location\") pod \"isvc-xgboost-runtime-predictor-54dd488c85-czpmk\" (UID: \"604dd384-dce7-4465-8c5f-5c75133f8156\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-czpmk"
Jul 27 22:27:22.764620 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:27:22.764425    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/604dd384-dce7-4465-8c5f-5c75133f8156-proxy-tls\") pod \"isvc-xgboost-runtime-predictor-54dd488c85-czpmk\" (UID: \"604dd384-dce7-4465-8c5f-5c75133f8156\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-czpmk"
Jul 27 22:27:22.764620 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:27:22.764464    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m5ft6\" (UniqueName: \"kubernetes.io/projected/604dd384-dce7-4465-8c5f-5c75133f8156-kube-api-access-m5ft6\") pod \"isvc-xgboost-runtime-predictor-54dd488c85-czpmk\" (UID: \"604dd384-dce7-4465-8c5f-5c75133f8156\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-czpmk"
Jul 27 22:27:22.764919 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:27:22.764891    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/604dd384-dce7-4465-8c5f-5c75133f8156-kserve-provision-location\") pod \"isvc-xgboost-runtime-predictor-54dd488c85-czpmk\" (UID: \"604dd384-dce7-4465-8c5f-5c75133f8156\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-czpmk"
Jul 27 22:27:22.765130 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:27:22.765110    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/604dd384-dce7-4465-8c5f-5c75133f8156-cabundle-cert\") pod \"isvc-xgboost-runtime-predictor-54dd488c85-czpmk\" (UID: \"604dd384-dce7-4465-8c5f-5c75133f8156\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-czpmk"
Jul 27 22:27:22.765208 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:27:22.765127    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/604dd384-dce7-4465-8c5f-5c75133f8156-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-runtime-predictor-54dd488c85-czpmk\" (UID: \"604dd384-dce7-4465-8c5f-5c75133f8156\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-czpmk"
Jul 27 22:27:22.766987 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:27:22.766956    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/604dd384-dce7-4465-8c5f-5c75133f8156-proxy-tls\") pod \"isvc-xgboost-runtime-predictor-54dd488c85-czpmk\" (UID: \"604dd384-dce7-4465-8c5f-5c75133f8156\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-czpmk"
Jul 27 22:27:22.772453 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:27:22.772432    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m5ft6\" (UniqueName: \"kubernetes.io/projected/604dd384-dce7-4465-8c5f-5c75133f8156-kube-api-access-m5ft6\") pod \"isvc-xgboost-runtime-predictor-54dd488c85-czpmk\" (UID: \"604dd384-dce7-4465-8c5f-5c75133f8156\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-czpmk"
Jul 27 22:27:22.907949 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:27:22.907918    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-czpmk"
Jul 27 22:27:23.038517 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:27:23.038485    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-czpmk"]
Jul 27 22:27:23.040634 ip-10-0-131-133 kubenswrapper[2559]: W0727 22:27:23.040604    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod604dd384_dce7_4465_8c5f_5c75133f8156.slice/crio-795426804f402403c8d05875f0d77535df8b21b054aef6715c20d431e30e44c5 WatchSource:0}: Error finding container 795426804f402403c8d05875f0d77535df8b21b054aef6715c20d431e30e44c5: Status 404 returned error can't find the container with id 795426804f402403c8d05875f0d77535df8b21b054aef6715c20d431e30e44c5
Jul 27 22:27:23.207078 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:27:23.206985    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-czpmk" event={"ID":"604dd384-dce7-4465-8c5f-5c75133f8156","Type":"ContainerStarted","Data":"718b567a861dbf0b67f96972c9ed88218829ef319bb2c0a125a01a7588e6495e"}
Jul 27 22:27:23.207078 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:27:23.207070    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-czpmk" event={"ID":"604dd384-dce7-4465-8c5f-5c75133f8156","Type":"ContainerStarted","Data":"795426804f402403c8d05875f0d77535df8b21b054aef6715c20d431e30e44c5"}
Jul 27 22:27:27.226316 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:27:27.226280    2559 generic.go:356] "Generic (PLEG): container finished" podID="604dd384-dce7-4465-8c5f-5c75133f8156" containerID="718b567a861dbf0b67f96972c9ed88218829ef319bb2c0a125a01a7588e6495e" exitCode=0
Jul 27 22:27:27.226708 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:27:27.226354    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-czpmk" event={"ID":"604dd384-dce7-4465-8c5f-5c75133f8156","Type":"ContainerDied","Data":"718b567a861dbf0b67f96972c9ed88218829ef319bb2c0a125a01a7588e6495e"}
Jul 27 22:27:27.227592 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:27:27.227576    2559 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 27 22:27:46.331519 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:27:46.331483    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-czpmk" event={"ID":"604dd384-dce7-4465-8c5f-5c75133f8156","Type":"ContainerStarted","Data":"08b885ea84a5a47737e90b2b43e0f762549d11104a9e9aefebfb4426d8aad213"}
Jul 27 22:27:46.331914 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:27:46.331545    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-czpmk" event={"ID":"604dd384-dce7-4465-8c5f-5c75133f8156","Type":"ContainerStarted","Data":"0b49c1e3962c85783a606274d726f061438b28b2548c515b5c6f1aad3bffe44f"}
Jul 27 22:27:46.331914 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:27:46.331737    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-czpmk"
Jul 27 22:27:46.353079 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:27:46.353027    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-czpmk" podStartSLOduration=2.147224646 podStartE2EDuration="24.353011404s" totalImagesPullingTime="18.32525334s" totalInitContainerRuntime="3.880533418s" isStatefulPod=false podCreationTimestamp="2026-07-27 22:27:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 22:27:46.350703793 +0000 UTC m=+4994.929887801" watchObservedRunningTime="2026-07-27 22:27:46.353011404 +0000 UTC m=+4994.932195410"
Jul 27 22:27:47.336229 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:27:47.336179    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-czpmk"
Jul 27 22:27:47.337579 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:27:47.337542    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-czpmk" podUID="604dd384-dce7-4465-8c5f-5c75133f8156" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.44:8080: connect: connection refused"
Jul 27 22:27:48.341041 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:27:48.340995    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-czpmk" podUID="604dd384-dce7-4465-8c5f-5c75133f8156" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.44:8080: connect: connection refused"
Jul 27 22:27:53.346727 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:27:53.346697    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-czpmk"
Jul 27 22:27:53.347277 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:27:53.347198    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-czpmk" podUID="604dd384-dce7-4465-8c5f-5c75133f8156" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.44:8080: connect: connection refused"
Jul 27 22:28:03.347557 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:28:03.347511    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-czpmk" podUID="604dd384-dce7-4465-8c5f-5c75133f8156" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.44:8080: connect: connection refused"
Jul 27 22:28:13.347757 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:28:13.347707    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-czpmk" podUID="604dd384-dce7-4465-8c5f-5c75133f8156" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.44:8080: connect: connection refused"
Jul 27 22:28:23.348129 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:28:23.348030    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-czpmk" podUID="604dd384-dce7-4465-8c5f-5c75133f8156" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.44:8080: connect: connection refused"
Jul 27 22:28:33.347330 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:28:33.347287    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-czpmk" podUID="604dd384-dce7-4465-8c5f-5c75133f8156" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.44:8080: connect: connection refused"
Jul 27 22:28:43.347849 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:28:43.347804    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-czpmk" podUID="604dd384-dce7-4465-8c5f-5c75133f8156" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.44:8080: connect: connection refused"
Jul 27 22:28:53.348419 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:28:53.348387    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-czpmk"
Jul 27 22:29:02.656122 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:29:02.656081    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-czpmk"]
Jul 27 22:29:02.656546 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:29:02.656374    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-czpmk" podUID="604dd384-dce7-4465-8c5f-5c75133f8156" containerName="kserve-container" containerID="cri-o://0b49c1e3962c85783a606274d726f061438b28b2548c515b5c6f1aad3bffe44f" gracePeriod=30
Jul 27 22:29:02.656546 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:29:02.656418    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-czpmk" podUID="604dd384-dce7-4465-8c5f-5c75133f8156" containerName="kube-rbac-proxy" containerID="cri-o://08b885ea84a5a47737e90b2b43e0f762549d11104a9e9aefebfb4426d8aad213" gracePeriod=30
Jul 27 22:29:03.341312 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:29:03.341261    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-czpmk" podUID="604dd384-dce7-4465-8c5f-5c75133f8156" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.44:8643/healthz\": dial tcp 10.132.0.44:8643: connect: connection refused"
Jul 27 22:29:03.347720 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:29:03.347681    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-czpmk" podUID="604dd384-dce7-4465-8c5f-5c75133f8156" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.44:8080: connect: connection refused"
Jul 27 22:29:03.698079 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:29:03.698044    2559 generic.go:356] "Generic (PLEG): container finished" podID="604dd384-dce7-4465-8c5f-5c75133f8156" containerID="08b885ea84a5a47737e90b2b43e0f762549d11104a9e9aefebfb4426d8aad213" exitCode=2
Jul 27 22:29:03.698496 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:29:03.698120    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-czpmk" event={"ID":"604dd384-dce7-4465-8c5f-5c75133f8156","Type":"ContainerDied","Data":"08b885ea84a5a47737e90b2b43e0f762549d11104a9e9aefebfb4426d8aad213"}
Jul 27 22:29:06.394833 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:29:06.394804    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-czpmk"
Jul 27 22:29:06.515466 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:29:06.515379    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/604dd384-dce7-4465-8c5f-5c75133f8156-kube-rbac-proxy-sar-config\") pod \"604dd384-dce7-4465-8c5f-5c75133f8156\" (UID: \"604dd384-dce7-4465-8c5f-5c75133f8156\") "
Jul 27 22:29:06.515466 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:29:06.515432    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/604dd384-dce7-4465-8c5f-5c75133f8156-proxy-tls\") pod \"604dd384-dce7-4465-8c5f-5c75133f8156\" (UID: \"604dd384-dce7-4465-8c5f-5c75133f8156\") "
Jul 27 22:29:06.515466 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:29:06.515460    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/604dd384-dce7-4465-8c5f-5c75133f8156-cabundle-cert\") pod \"604dd384-dce7-4465-8c5f-5c75133f8156\" (UID: \"604dd384-dce7-4465-8c5f-5c75133f8156\") "
Jul 27 22:29:06.515763 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:29:06.515487    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-m5ft6\" (UniqueName: \"kubernetes.io/projected/604dd384-dce7-4465-8c5f-5c75133f8156-kube-api-access-m5ft6\") pod \"604dd384-dce7-4465-8c5f-5c75133f8156\" (UID: \"604dd384-dce7-4465-8c5f-5c75133f8156\") "
Jul 27 22:29:06.515763 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:29:06.515548    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/604dd384-dce7-4465-8c5f-5c75133f8156-kserve-provision-location\") pod \"604dd384-dce7-4465-8c5f-5c75133f8156\" (UID: \"604dd384-dce7-4465-8c5f-5c75133f8156\") "
Jul 27 22:29:06.515763 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:29:06.515718    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/604dd384-dce7-4465-8c5f-5c75133f8156-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "604dd384-dce7-4465-8c5f-5c75133f8156" (UID: "604dd384-dce7-4465-8c5f-5c75133f8156"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 22:29:06.515930 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:29:06.515841    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/604dd384-dce7-4465-8c5f-5c75133f8156-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "604dd384-dce7-4465-8c5f-5c75133f8156" (UID: "604dd384-dce7-4465-8c5f-5c75133f8156"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 22:29:06.515976 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:29:06.515930    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/604dd384-dce7-4465-8c5f-5c75133f8156-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "604dd384-dce7-4465-8c5f-5c75133f8156" (UID: "604dd384-dce7-4465-8c5f-5c75133f8156"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 22:29:06.517642 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:29:06.517620    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/604dd384-dce7-4465-8c5f-5c75133f8156-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "604dd384-dce7-4465-8c5f-5c75133f8156" (UID: "604dd384-dce7-4465-8c5f-5c75133f8156"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 22:29:06.517766 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:29:06.517690    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/604dd384-dce7-4465-8c5f-5c75133f8156-kube-api-access-m5ft6" (OuterVolumeSpecName: "kube-api-access-m5ft6") pod "604dd384-dce7-4465-8c5f-5c75133f8156" (UID: "604dd384-dce7-4465-8c5f-5c75133f8156"). InnerVolumeSpecName "kube-api-access-m5ft6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 22:29:06.616166 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:29:06.616127    2559 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/604dd384-dce7-4465-8c5f-5c75133f8156-kserve-provision-location\") on node \"ip-10-0-131-133.ec2.internal\" DevicePath \"\""
Jul 27 22:29:06.616166 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:29:06.616158    2559 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/604dd384-dce7-4465-8c5f-5c75133f8156-kube-rbac-proxy-sar-config\") on node \"ip-10-0-131-133.ec2.internal\" DevicePath \"\""
Jul 27 22:29:06.616166 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:29:06.616168    2559 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/604dd384-dce7-4465-8c5f-5c75133f8156-proxy-tls\") on node \"ip-10-0-131-133.ec2.internal\" DevicePath \"\""
Jul 27 22:29:06.616166 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:29:06.616178    2559 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/604dd384-dce7-4465-8c5f-5c75133f8156-cabundle-cert\") on node \"ip-10-0-131-133.ec2.internal\" DevicePath \"\""
Jul 27 22:29:06.616468 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:29:06.616186    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m5ft6\" (UniqueName: \"kubernetes.io/projected/604dd384-dce7-4465-8c5f-5c75133f8156-kube-api-access-m5ft6\") on node \"ip-10-0-131-133.ec2.internal\" DevicePath \"\""
Jul 27 22:29:06.713936 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:29:06.713896    2559 generic.go:356] "Generic (PLEG): container finished" podID="604dd384-dce7-4465-8c5f-5c75133f8156" containerID="0b49c1e3962c85783a606274d726f061438b28b2548c515b5c6f1aad3bffe44f" exitCode=0
Jul 27 22:29:06.714100 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:29:06.713967    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-czpmk"
Jul 27 22:29:06.714100 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:29:06.713966    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-czpmk" event={"ID":"604dd384-dce7-4465-8c5f-5c75133f8156","Type":"ContainerDied","Data":"0b49c1e3962c85783a606274d726f061438b28b2548c515b5c6f1aad3bffe44f"}
Jul 27 22:29:06.714100 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:29:06.714029    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-czpmk" event={"ID":"604dd384-dce7-4465-8c5f-5c75133f8156","Type":"ContainerDied","Data":"795426804f402403c8d05875f0d77535df8b21b054aef6715c20d431e30e44c5"}
Jul 27 22:29:06.714100 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:29:06.714055    2559 scope.go:117] "RemoveContainer" containerID="08b885ea84a5a47737e90b2b43e0f762549d11104a9e9aefebfb4426d8aad213"
Jul 27 22:29:06.723853 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:29:06.723833    2559 scope.go:117] "RemoveContainer" containerID="0b49c1e3962c85783a606274d726f061438b28b2548c515b5c6f1aad3bffe44f"
Jul 27 22:29:06.731494 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:29:06.731478    2559 scope.go:117] "RemoveContainer" containerID="718b567a861dbf0b67f96972c9ed88218829ef319bb2c0a125a01a7588e6495e"
Jul 27 22:29:06.737987 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:29:06.737963    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-czpmk"]
Jul 27 22:29:06.738947 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:29:06.738932    2559 scope.go:117] "RemoveContainer" containerID="08b885ea84a5a47737e90b2b43e0f762549d11104a9e9aefebfb4426d8aad213"
Jul 27 22:29:06.739223 ip-10-0-131-133 kubenswrapper[2559]: E0727 22:29:06.739194    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"08b885ea84a5a47737e90b2b43e0f762549d11104a9e9aefebfb4426d8aad213\": container with ID starting with 08b885ea84a5a47737e90b2b43e0f762549d11104a9e9aefebfb4426d8aad213 not found: ID does not exist" containerID="08b885ea84a5a47737e90b2b43e0f762549d11104a9e9aefebfb4426d8aad213"
Jul 27 22:29:06.739289 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:29:06.739260    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"08b885ea84a5a47737e90b2b43e0f762549d11104a9e9aefebfb4426d8aad213"} err="failed to get container status \"08b885ea84a5a47737e90b2b43e0f762549d11104a9e9aefebfb4426d8aad213\": rpc error: code = NotFound desc = could not find container \"08b885ea84a5a47737e90b2b43e0f762549d11104a9e9aefebfb4426d8aad213\": container with ID starting with 08b885ea84a5a47737e90b2b43e0f762549d11104a9e9aefebfb4426d8aad213 not found: ID does not exist"
Jul 27 22:29:06.739343 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:29:06.739294    2559 scope.go:117] "RemoveContainer" containerID="0b49c1e3962c85783a606274d726f061438b28b2548c515b5c6f1aad3bffe44f"
Jul 27 22:29:06.739549 ip-10-0-131-133 kubenswrapper[2559]: E0727 22:29:06.739531    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0b49c1e3962c85783a606274d726f061438b28b2548c515b5c6f1aad3bffe44f\": container with ID starting with 0b49c1e3962c85783a606274d726f061438b28b2548c515b5c6f1aad3bffe44f not found: ID does not exist" containerID="0b49c1e3962c85783a606274d726f061438b28b2548c515b5c6f1aad3bffe44f"
Jul 27 22:29:06.739655 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:29:06.739556    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0b49c1e3962c85783a606274d726f061438b28b2548c515b5c6f1aad3bffe44f"} err="failed to get container status \"0b49c1e3962c85783a606274d726f061438b28b2548c515b5c6f1aad3bffe44f\": rpc error: code = NotFound desc = could not find container \"0b49c1e3962c85783a606274d726f061438b28b2548c515b5c6f1aad3bffe44f\": container with ID starting with 0b49c1e3962c85783a606274d726f061438b28b2548c515b5c6f1aad3bffe44f not found: ID does not exist"
Jul 27 22:29:06.739655 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:29:06.739573    2559 scope.go:117] "RemoveContainer" containerID="718b567a861dbf0b67f96972c9ed88218829ef319bb2c0a125a01a7588e6495e"
Jul 27 22:29:06.739857 ip-10-0-131-133 kubenswrapper[2559]: E0727 22:29:06.739837    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"718b567a861dbf0b67f96972c9ed88218829ef319bb2c0a125a01a7588e6495e\": container with ID starting with 718b567a861dbf0b67f96972c9ed88218829ef319bb2c0a125a01a7588e6495e not found: ID does not exist" containerID="718b567a861dbf0b67f96972c9ed88218829ef319bb2c0a125a01a7588e6495e"
Jul 27 22:29:06.739904 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:29:06.739861    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"718b567a861dbf0b67f96972c9ed88218829ef319bb2c0a125a01a7588e6495e"} err="failed to get container status \"718b567a861dbf0b67f96972c9ed88218829ef319bb2c0a125a01a7588e6495e\": rpc error: code = NotFound desc = could not find container \"718b567a861dbf0b67f96972c9ed88218829ef319bb2c0a125a01a7588e6495e\": container with ID starting with 718b567a861dbf0b67f96972c9ed88218829ef319bb2c0a125a01a7588e6495e not found: ID does not exist"
Jul 27 22:29:06.742540 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:29:06.742517    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-czpmk"]
Jul 27 22:29:07.982232 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:29:07.982176    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="604dd384-dce7-4465-8c5f-5c75133f8156" path="/var/lib/kubelet/pods/604dd384-dce7-4465-8c5f-5c75133f8156/volumes"
Jul 27 22:29:32.361459 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:29:32.361429    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-bjvbc_cba53c9a-64c0-4afa-8731-35521e196622/console-operator/1.log"
Jul 27 22:29:32.370525 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:29:32.370494    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-bjvbc_cba53c9a-64c0-4afa-8731-35521e196622/console-operator/1.log"
Jul 27 22:30:13.028792 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:30:13.028755    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-zwxj7"]
Jul 27 22:30:13.029263 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:30:13.029121    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="604dd384-dce7-4465-8c5f-5c75133f8156" containerName="kserve-container"
Jul 27 22:30:13.029263 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:30:13.029131    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="604dd384-dce7-4465-8c5f-5c75133f8156" containerName="kserve-container"
Jul 27 22:30:13.029263 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:30:13.029140    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="604dd384-dce7-4465-8c5f-5c75133f8156" containerName="kube-rbac-proxy"
Jul 27 22:30:13.029263 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:30:13.029145    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="604dd384-dce7-4465-8c5f-5c75133f8156" containerName="kube-rbac-proxy"
Jul 27 22:30:13.029263 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:30:13.029172    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="604dd384-dce7-4465-8c5f-5c75133f8156" containerName="storage-initializer"
Jul 27 22:30:13.029263 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:30:13.029177    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="604dd384-dce7-4465-8c5f-5c75133f8156" containerName="storage-initializer"
Jul 27 22:30:13.029263 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:30:13.029248    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="604dd384-dce7-4465-8c5f-5c75133f8156" containerName="kube-rbac-proxy"
Jul 27 22:30:13.029263 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:30:13.029257    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="604dd384-dce7-4465-8c5f-5c75133f8156" containerName="kserve-container"
Jul 27 22:30:13.031586 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:30:13.031568    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-zwxj7"
Jul 27 22:30:13.033978 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:30:13.033953    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 27 22:30:13.034126 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:30:13.033954    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 27 22:30:13.034126 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:30:13.033958    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 27 22:30:13.034686 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:30:13.034667    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-v7c66\""
Jul 27 22:30:13.034737 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:30:13.034709    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-v2-kube-rbac-proxy-sar-config\""
Jul 27 22:30:13.034737 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:30:13.034715    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-v2-predictor-serving-cert\""
Jul 27 22:30:13.034844 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:30:13.034740    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 27 22:30:13.042314 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:30:13.042289    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-zwxj7"]
Jul 27 22:30:13.122205 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:30:13.122165    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/14f27bab-7371-4f73-8d18-78ef2234bded-proxy-tls\") pod \"isvc-xgboost-v2-predictor-fcc668bc-zwxj7\" (UID: \"14f27bab-7371-4f73-8d18-78ef2234bded\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-zwxj7"
Jul 27 22:30:13.122205 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:30:13.122205    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hz97j\" (UniqueName: \"kubernetes.io/projected/14f27bab-7371-4f73-8d18-78ef2234bded-kube-api-access-hz97j\") pod \"isvc-xgboost-v2-predictor-fcc668bc-zwxj7\" (UID: \"14f27bab-7371-4f73-8d18-78ef2234bded\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-zwxj7"
Jul 27 22:30:13.122492 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:30:13.122318    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/14f27bab-7371-4f73-8d18-78ef2234bded-cabundle-cert\") pod \"isvc-xgboost-v2-predictor-fcc668bc-zwxj7\" (UID: \"14f27bab-7371-4f73-8d18-78ef2234bded\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-zwxj7"
Jul 27 22:30:13.122492 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:30:13.122366    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/14f27bab-7371-4f73-8d18-78ef2234bded-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-v2-predictor-fcc668bc-zwxj7\" (UID: \"14f27bab-7371-4f73-8d18-78ef2234bded\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-zwxj7"
Jul 27 22:30:13.122492 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:30:13.122387    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/14f27bab-7371-4f73-8d18-78ef2234bded-kserve-provision-location\") pod \"isvc-xgboost-v2-predictor-fcc668bc-zwxj7\" (UID: \"14f27bab-7371-4f73-8d18-78ef2234bded\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-zwxj7"
Jul 27 22:30:13.223659 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:30:13.223618    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/14f27bab-7371-4f73-8d18-78ef2234bded-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-v2-predictor-fcc668bc-zwxj7\" (UID: \"14f27bab-7371-4f73-8d18-78ef2234bded\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-zwxj7"
Jul 27 22:30:13.223659 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:30:13.223661    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/14f27bab-7371-4f73-8d18-78ef2234bded-kserve-provision-location\") pod \"isvc-xgboost-v2-predictor-fcc668bc-zwxj7\" (UID: \"14f27bab-7371-4f73-8d18-78ef2234bded\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-zwxj7"
Jul 27 22:30:13.223927 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:30:13.223765    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/14f27bab-7371-4f73-8d18-78ef2234bded-proxy-tls\") pod \"isvc-xgboost-v2-predictor-fcc668bc-zwxj7\" (UID: \"14f27bab-7371-4f73-8d18-78ef2234bded\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-zwxj7"
Jul 27 22:30:13.223927 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:30:13.223790    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hz97j\" (UniqueName: \"kubernetes.io/projected/14f27bab-7371-4f73-8d18-78ef2234bded-kube-api-access-hz97j\") pod \"isvc-xgboost-v2-predictor-fcc668bc-zwxj7\" (UID: \"14f27bab-7371-4f73-8d18-78ef2234bded\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-zwxj7"
Jul 27 22:30:13.223927 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:30:13.223829    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/14f27bab-7371-4f73-8d18-78ef2234bded-cabundle-cert\") pod \"isvc-xgboost-v2-predictor-fcc668bc-zwxj7\" (UID: \"14f27bab-7371-4f73-8d18-78ef2234bded\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-zwxj7"
Jul 27 22:30:13.223927 ip-10-0-131-133 kubenswrapper[2559]: E0727 22:30:13.223908    2559 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-xgboost-v2-predictor-serving-cert: secret "isvc-xgboost-v2-predictor-serving-cert" not found
Jul 27 22:30:13.224140 ip-10-0-131-133 kubenswrapper[2559]: E0727 22:30:13.223981    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/14f27bab-7371-4f73-8d18-78ef2234bded-proxy-tls podName:14f27bab-7371-4f73-8d18-78ef2234bded nodeName:}" failed. No retries permitted until 2026-07-27 22:30:13.723964203 +0000 UTC m=+5142.303148191 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/14f27bab-7371-4f73-8d18-78ef2234bded-proxy-tls") pod "isvc-xgboost-v2-predictor-fcc668bc-zwxj7" (UID: "14f27bab-7371-4f73-8d18-78ef2234bded") : secret "isvc-xgboost-v2-predictor-serving-cert" not found
Jul 27 22:30:13.224250 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:30:13.224137    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/14f27bab-7371-4f73-8d18-78ef2234bded-kserve-provision-location\") pod \"isvc-xgboost-v2-predictor-fcc668bc-zwxj7\" (UID: \"14f27bab-7371-4f73-8d18-78ef2234bded\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-zwxj7"
Jul 27 22:30:13.224421 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:30:13.224401    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/14f27bab-7371-4f73-8d18-78ef2234bded-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-v2-predictor-fcc668bc-zwxj7\" (UID: \"14f27bab-7371-4f73-8d18-78ef2234bded\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-zwxj7"
Jul 27 22:30:13.224533 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:30:13.224515    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/14f27bab-7371-4f73-8d18-78ef2234bded-cabundle-cert\") pod \"isvc-xgboost-v2-predictor-fcc668bc-zwxj7\" (UID: \"14f27bab-7371-4f73-8d18-78ef2234bded\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-zwxj7"
Jul 27 22:30:13.233171 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:30:13.233143    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hz97j\" (UniqueName: \"kubernetes.io/projected/14f27bab-7371-4f73-8d18-78ef2234bded-kube-api-access-hz97j\") pod \"isvc-xgboost-v2-predictor-fcc668bc-zwxj7\" (UID: \"14f27bab-7371-4f73-8d18-78ef2234bded\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-zwxj7"
Jul 27 22:30:13.727874 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:30:13.727827    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/14f27bab-7371-4f73-8d18-78ef2234bded-proxy-tls\") pod \"isvc-xgboost-v2-predictor-fcc668bc-zwxj7\" (UID: \"14f27bab-7371-4f73-8d18-78ef2234bded\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-zwxj7"
Jul 27 22:30:13.730423 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:30:13.730395    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/14f27bab-7371-4f73-8d18-78ef2234bded-proxy-tls\") pod \"isvc-xgboost-v2-predictor-fcc668bc-zwxj7\" (UID: \"14f27bab-7371-4f73-8d18-78ef2234bded\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-zwxj7"
Jul 27 22:30:13.943002 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:30:13.942957    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-zwxj7"
Jul 27 22:30:14.084363 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:30:14.084335    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-zwxj7"]
Jul 27 22:30:14.086440 ip-10-0-131-133 kubenswrapper[2559]: W0727 22:30:14.086414    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod14f27bab_7371_4f73_8d18_78ef2234bded.slice/crio-dbe18b43e3c9ac035d2162be79a9ab21777827ae638c1d21d7d0a51b6c85c5c1 WatchSource:0}: Error finding container dbe18b43e3c9ac035d2162be79a9ab21777827ae638c1d21d7d0a51b6c85c5c1: Status 404 returned error can't find the container with id dbe18b43e3c9ac035d2162be79a9ab21777827ae638c1d21d7d0a51b6c85c5c1
Jul 27 22:30:15.031718 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:30:15.031678    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-zwxj7" event={"ID":"14f27bab-7371-4f73-8d18-78ef2234bded","Type":"ContainerStarted","Data":"b32b365fc3824455f97d8981d115ea52694b486011de7bdf1ecbfdece7c75425"}
Jul 27 22:30:15.031900 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:30:15.031738    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-zwxj7" event={"ID":"14f27bab-7371-4f73-8d18-78ef2234bded","Type":"ContainerStarted","Data":"dbe18b43e3c9ac035d2162be79a9ab21777827ae638c1d21d7d0a51b6c85c5c1"}
Jul 27 22:30:18.047618 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:30:18.047527    2559 generic.go:356] "Generic (PLEG): container finished" podID="14f27bab-7371-4f73-8d18-78ef2234bded" containerID="b32b365fc3824455f97d8981d115ea52694b486011de7bdf1ecbfdece7c75425" exitCode=0
Jul 27 22:30:18.047618 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:30:18.047600    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-zwxj7" event={"ID":"14f27bab-7371-4f73-8d18-78ef2234bded","Type":"ContainerDied","Data":"b32b365fc3824455f97d8981d115ea52694b486011de7bdf1ecbfdece7c75425"}
Jul 27 22:30:19.054483 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:30:19.054451    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-zwxj7" event={"ID":"14f27bab-7371-4f73-8d18-78ef2234bded","Type":"ContainerStarted","Data":"bd63e8895d398f540d593773eb14d8b6e85e70974db5c48b9c33493e9f682d60"}
Jul 27 22:30:19.054905 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:30:19.054511    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-zwxj7" event={"ID":"14f27bab-7371-4f73-8d18-78ef2234bded","Type":"ContainerStarted","Data":"857a8ffd69283a36e18f1a3f647b0afd38820a8e3041fdaf68ade17154bd19d0"}
Jul 27 22:30:19.054905 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:30:19.054706    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-zwxj7"
Jul 27 22:30:19.079589 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:30:19.079539    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-zwxj7" podStartSLOduration=2.449188907 podStartE2EDuration="6.079524734s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.630335827s" isStatefulPod=false podCreationTimestamp="2026-07-27 22:30:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 22:30:19.076920477 +0000 UTC m=+5147.656104484" watchObservedRunningTime="2026-07-27 22:30:19.079524734 +0000 UTC m=+5147.658708740"
Jul 27 22:30:20.059744 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:30:20.059714    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-zwxj7"
Jul 27 22:30:20.060873 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:30:20.060849    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-zwxj7" podUID="14f27bab-7371-4f73-8d18-78ef2234bded" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.45:8080: connect: connection refused"
Jul 27 22:30:21.064550 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:30:21.064510    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-zwxj7" podUID="14f27bab-7371-4f73-8d18-78ef2234bded" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.45:8080: connect: connection refused"
Jul 27 22:30:26.068835 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:30:26.068799    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-zwxj7"
Jul 27 22:30:26.069442 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:30:26.069416    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-zwxj7" podUID="14f27bab-7371-4f73-8d18-78ef2234bded" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.45:8080: connect: connection refused"
Jul 27 22:30:36.070100 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:30:36.070057    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-zwxj7" podUID="14f27bab-7371-4f73-8d18-78ef2234bded" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.45:8080: connect: connection refused"
Jul 27 22:30:46.069820 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:30:46.069777    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-zwxj7" podUID="14f27bab-7371-4f73-8d18-78ef2234bded" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.45:8080: connect: connection refused"
Jul 27 22:30:56.070268 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:30:56.070209    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-zwxj7" podUID="14f27bab-7371-4f73-8d18-78ef2234bded" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.45:8080: connect: connection refused"
Jul 27 22:31:06.069986 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:31:06.069948    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-zwxj7" podUID="14f27bab-7371-4f73-8d18-78ef2234bded" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.45:8080: connect: connection refused"
Jul 27 22:31:16.069786 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:31:16.069736    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-zwxj7" podUID="14f27bab-7371-4f73-8d18-78ef2234bded" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.45:8080: connect: connection refused"
Jul 27 22:31:26.070354 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:31:26.070272    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-zwxj7"
Jul 27 22:31:33.127429 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:31:33.127396    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-zwxj7"]
Jul 27 22:31:33.127845 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:31:33.127625    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-zwxj7" podUID="14f27bab-7371-4f73-8d18-78ef2234bded" containerName="kserve-container" containerID="cri-o://857a8ffd69283a36e18f1a3f647b0afd38820a8e3041fdaf68ade17154bd19d0" gracePeriod=30
Jul 27 22:31:33.127845 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:31:33.127701    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-zwxj7" podUID="14f27bab-7371-4f73-8d18-78ef2234bded" containerName="kube-rbac-proxy" containerID="cri-o://bd63e8895d398f540d593773eb14d8b6e85e70974db5c48b9c33493e9f682d60" gracePeriod=30
Jul 27 22:31:33.410485 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:31:33.410388    2559 generic.go:356] "Generic (PLEG): container finished" podID="14f27bab-7371-4f73-8d18-78ef2234bded" containerID="bd63e8895d398f540d593773eb14d8b6e85e70974db5c48b9c33493e9f682d60" exitCode=2
Jul 27 22:31:33.410485 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:31:33.410460    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-zwxj7" event={"ID":"14f27bab-7371-4f73-8d18-78ef2234bded","Type":"ContainerDied","Data":"bd63e8895d398f540d593773eb14d8b6e85e70974db5c48b9c33493e9f682d60"}
Jul 27 22:31:36.064768 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:31:36.064722    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-zwxj7" podUID="14f27bab-7371-4f73-8d18-78ef2234bded" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.45:8643/healthz\": dial tcp 10.132.0.45:8643: connect: connection refused"
Jul 27 22:31:36.069535 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:31:36.069501    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-zwxj7" podUID="14f27bab-7371-4f73-8d18-78ef2234bded" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.45:8080: connect: connection refused"
Jul 27 22:31:36.982123 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:31:36.982097    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-zwxj7"
Jul 27 22:31:37.077436 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:31:37.077337    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/14f27bab-7371-4f73-8d18-78ef2234bded-kube-rbac-proxy-sar-config\") pod \"14f27bab-7371-4f73-8d18-78ef2234bded\" (UID: \"14f27bab-7371-4f73-8d18-78ef2234bded\") "
Jul 27 22:31:37.077436 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:31:37.077374    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/14f27bab-7371-4f73-8d18-78ef2234bded-kserve-provision-location\") pod \"14f27bab-7371-4f73-8d18-78ef2234bded\" (UID: \"14f27bab-7371-4f73-8d18-78ef2234bded\") "
Jul 27 22:31:37.077436 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:31:37.077416    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/14f27bab-7371-4f73-8d18-78ef2234bded-cabundle-cert\") pod \"14f27bab-7371-4f73-8d18-78ef2234bded\" (UID: \"14f27bab-7371-4f73-8d18-78ef2234bded\") "
Jul 27 22:31:37.077994 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:31:37.077451    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/14f27bab-7371-4f73-8d18-78ef2234bded-proxy-tls\") pod \"14f27bab-7371-4f73-8d18-78ef2234bded\" (UID: \"14f27bab-7371-4f73-8d18-78ef2234bded\") "
Jul 27 22:31:37.077994 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:31:37.077470    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hz97j\" (UniqueName: \"kubernetes.io/projected/14f27bab-7371-4f73-8d18-78ef2234bded-kube-api-access-hz97j\") pod \"14f27bab-7371-4f73-8d18-78ef2234bded\" (UID: \"14f27bab-7371-4f73-8d18-78ef2234bded\") "
Jul 27 22:31:37.077994 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:31:37.077802    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/14f27bab-7371-4f73-8d18-78ef2234bded-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "14f27bab-7371-4f73-8d18-78ef2234bded" (UID: "14f27bab-7371-4f73-8d18-78ef2234bded"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 22:31:37.077994 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:31:37.077830    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/14f27bab-7371-4f73-8d18-78ef2234bded-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "14f27bab-7371-4f73-8d18-78ef2234bded" (UID: "14f27bab-7371-4f73-8d18-78ef2234bded"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 22:31:37.077994 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:31:37.077839    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/14f27bab-7371-4f73-8d18-78ef2234bded-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "14f27bab-7371-4f73-8d18-78ef2234bded" (UID: "14f27bab-7371-4f73-8d18-78ef2234bded"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 22:31:37.079643 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:31:37.079614    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/14f27bab-7371-4f73-8d18-78ef2234bded-kube-api-access-hz97j" (OuterVolumeSpecName: "kube-api-access-hz97j") pod "14f27bab-7371-4f73-8d18-78ef2234bded" (UID: "14f27bab-7371-4f73-8d18-78ef2234bded"). InnerVolumeSpecName "kube-api-access-hz97j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 22:31:37.079728 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:31:37.079708    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/14f27bab-7371-4f73-8d18-78ef2234bded-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "14f27bab-7371-4f73-8d18-78ef2234bded" (UID: "14f27bab-7371-4f73-8d18-78ef2234bded"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 22:31:37.179018 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:31:37.178977    2559 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/14f27bab-7371-4f73-8d18-78ef2234bded-kube-rbac-proxy-sar-config\") on node \"ip-10-0-131-133.ec2.internal\" DevicePath \"\""
Jul 27 22:31:37.179018 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:31:37.179009    2559 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/14f27bab-7371-4f73-8d18-78ef2234bded-kserve-provision-location\") on node \"ip-10-0-131-133.ec2.internal\" DevicePath \"\""
Jul 27 22:31:37.179018 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:31:37.179024    2559 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/14f27bab-7371-4f73-8d18-78ef2234bded-cabundle-cert\") on node \"ip-10-0-131-133.ec2.internal\" DevicePath \"\""
Jul 27 22:31:37.179310 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:31:37.179039    2559 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/14f27bab-7371-4f73-8d18-78ef2234bded-proxy-tls\") on node \"ip-10-0-131-133.ec2.internal\" DevicePath \"\""
Jul 27 22:31:37.179310 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:31:37.179052    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hz97j\" (UniqueName: \"kubernetes.io/projected/14f27bab-7371-4f73-8d18-78ef2234bded-kube-api-access-hz97j\") on node \"ip-10-0-131-133.ec2.internal\" DevicePath \"\""
Jul 27 22:31:37.431625 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:31:37.431588    2559 generic.go:356] "Generic (PLEG): container finished" podID="14f27bab-7371-4f73-8d18-78ef2234bded" containerID="857a8ffd69283a36e18f1a3f647b0afd38820a8e3041fdaf68ade17154bd19d0" exitCode=0
Jul 27 22:31:37.431810 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:31:37.431654    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-zwxj7"
Jul 27 22:31:37.431810 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:31:37.431719    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-zwxj7" event={"ID":"14f27bab-7371-4f73-8d18-78ef2234bded","Type":"ContainerDied","Data":"857a8ffd69283a36e18f1a3f647b0afd38820a8e3041fdaf68ade17154bd19d0"}
Jul 27 22:31:37.431810 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:31:37.431780    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-zwxj7" event={"ID":"14f27bab-7371-4f73-8d18-78ef2234bded","Type":"ContainerDied","Data":"dbe18b43e3c9ac035d2162be79a9ab21777827ae638c1d21d7d0a51b6c85c5c1"}
Jul 27 22:31:37.431948 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:31:37.431818    2559 scope.go:117] "RemoveContainer" containerID="bd63e8895d398f540d593773eb14d8b6e85e70974db5c48b9c33493e9f682d60"
Jul 27 22:31:37.444073 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:31:37.444047    2559 scope.go:117] "RemoveContainer" containerID="857a8ffd69283a36e18f1a3f647b0afd38820a8e3041fdaf68ade17154bd19d0"
Jul 27 22:31:37.452794 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:31:37.452771    2559 scope.go:117] "RemoveContainer" containerID="b32b365fc3824455f97d8981d115ea52694b486011de7bdf1ecbfdece7c75425"
Jul 27 22:31:37.458795 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:31:37.458767    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-zwxj7"]
Jul 27 22:31:37.461017 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:31:37.460996    2559 scope.go:117] "RemoveContainer" containerID="bd63e8895d398f540d593773eb14d8b6e85e70974db5c48b9c33493e9f682d60"
Jul 27 22:31:37.461355 ip-10-0-131-133 kubenswrapper[2559]: E0727 22:31:37.461328    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bd63e8895d398f540d593773eb14d8b6e85e70974db5c48b9c33493e9f682d60\": container with ID starting with bd63e8895d398f540d593773eb14d8b6e85e70974db5c48b9c33493e9f682d60 not found: ID does not exist" containerID="bd63e8895d398f540d593773eb14d8b6e85e70974db5c48b9c33493e9f682d60"
Jul 27 22:31:37.461419 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:31:37.461364    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bd63e8895d398f540d593773eb14d8b6e85e70974db5c48b9c33493e9f682d60"} err="failed to get container status \"bd63e8895d398f540d593773eb14d8b6e85e70974db5c48b9c33493e9f682d60\": rpc error: code = NotFound desc = could not find container \"bd63e8895d398f540d593773eb14d8b6e85e70974db5c48b9c33493e9f682d60\": container with ID starting with bd63e8895d398f540d593773eb14d8b6e85e70974db5c48b9c33493e9f682d60 not found: ID does not exist"
Jul 27 22:31:37.461419 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:31:37.461387    2559 scope.go:117] "RemoveContainer" containerID="857a8ffd69283a36e18f1a3f647b0afd38820a8e3041fdaf68ade17154bd19d0"
Jul 27 22:31:37.461668 ip-10-0-131-133 kubenswrapper[2559]: E0727 22:31:37.461648    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"857a8ffd69283a36e18f1a3f647b0afd38820a8e3041fdaf68ade17154bd19d0\": container with ID starting with 857a8ffd69283a36e18f1a3f647b0afd38820a8e3041fdaf68ade17154bd19d0 not found: ID does not exist" containerID="857a8ffd69283a36e18f1a3f647b0afd38820a8e3041fdaf68ade17154bd19d0"
Jul 27 22:31:37.461723 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:31:37.461673    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"857a8ffd69283a36e18f1a3f647b0afd38820a8e3041fdaf68ade17154bd19d0"} err="failed to get container status \"857a8ffd69283a36e18f1a3f647b0afd38820a8e3041fdaf68ade17154bd19d0\": rpc error: code = NotFound desc = could not find container \"857a8ffd69283a36e18f1a3f647b0afd38820a8e3041fdaf68ade17154bd19d0\": container with ID starting with 857a8ffd69283a36e18f1a3f647b0afd38820a8e3041fdaf68ade17154bd19d0 not found: ID does not exist"
Jul 27 22:31:37.461723 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:31:37.461693    2559 scope.go:117] "RemoveContainer" containerID="b32b365fc3824455f97d8981d115ea52694b486011de7bdf1ecbfdece7c75425"
Jul 27 22:31:37.462024 ip-10-0-131-133 kubenswrapper[2559]: E0727 22:31:37.462000    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b32b365fc3824455f97d8981d115ea52694b486011de7bdf1ecbfdece7c75425\": container with ID starting with b32b365fc3824455f97d8981d115ea52694b486011de7bdf1ecbfdece7c75425 not found: ID does not exist" containerID="b32b365fc3824455f97d8981d115ea52694b486011de7bdf1ecbfdece7c75425"
Jul 27 22:31:37.462243 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:31:37.462029    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b32b365fc3824455f97d8981d115ea52694b486011de7bdf1ecbfdece7c75425"} err="failed to get container status \"b32b365fc3824455f97d8981d115ea52694b486011de7bdf1ecbfdece7c75425\": rpc error: code = NotFound desc = could not find container \"b32b365fc3824455f97d8981d115ea52694b486011de7bdf1ecbfdece7c75425\": container with ID starting with b32b365fc3824455f97d8981d115ea52694b486011de7bdf1ecbfdece7c75425 not found: ID does not exist"
Jul 27 22:31:37.464852 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:31:37.464832    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-zwxj7"]
Jul 27 22:31:37.982075 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:31:37.982042    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="14f27bab-7371-4f73-8d18-78ef2234bded" path="/var/lib/kubelet/pods/14f27bab-7371-4f73-8d18-78ef2234bded/volumes"
Jul 27 22:31:38.335760 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:31:38.335678    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-predictor-dd446d6b9-pj5zf"]
Jul 27 22:31:38.336185 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:31:38.336032    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="14f27bab-7371-4f73-8d18-78ef2234bded" containerName="storage-initializer"
Jul 27 22:31:38.336185 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:31:38.336043    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="14f27bab-7371-4f73-8d18-78ef2234bded" containerName="storage-initializer"
Jul 27 22:31:38.336185 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:31:38.336056    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="14f27bab-7371-4f73-8d18-78ef2234bded" containerName="kserve-container"
Jul 27 22:31:38.336185 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:31:38.336061    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="14f27bab-7371-4f73-8d18-78ef2234bded" containerName="kserve-container"
Jul 27 22:31:38.336185 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:31:38.336076    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="14f27bab-7371-4f73-8d18-78ef2234bded" containerName="kube-rbac-proxy"
Jul 27 22:31:38.336185 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:31:38.336086    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="14f27bab-7371-4f73-8d18-78ef2234bded" containerName="kube-rbac-proxy"
Jul 27 22:31:38.336185 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:31:38.336146    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="14f27bab-7371-4f73-8d18-78ef2234bded" containerName="kube-rbac-proxy"
Jul 27 22:31:38.336185 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:31:38.336158    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="14f27bab-7371-4f73-8d18-78ef2234bded" containerName="kserve-container"
Jul 27 22:31:38.339099 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:31:38.339080    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-dd446d6b9-pj5zf"
Jul 27 22:31:38.341368 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:31:38.341344    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"storage-config\""
Jul 27 22:31:38.341464 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:31:38.341427    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-v7c66\""
Jul 27 22:31:38.341540 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:31:38.341497    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 27 22:31:38.341540 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:31:38.341509    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-kube-rbac-proxy-sar-config\""
Jul 27 22:31:38.341953 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:31:38.341935    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-predictor-serving-cert\""
Jul 27 22:31:38.342083 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:31:38.342068    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 27 22:31:38.350006 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:31:38.349980    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-predictor-dd446d6b9-pj5zf"]
Jul 27 22:31:38.390410 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:31:38.390373    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b76b321f-5067-46fc-b05e-1348cfd4f318-proxy-tls\") pod \"isvc-sklearn-s3-predictor-dd446d6b9-pj5zf\" (UID: \"b76b321f-5067-46fc-b05e-1348cfd4f318\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-dd446d6b9-pj5zf"
Jul 27 22:31:38.390605 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:31:38.390432    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b76b321f-5067-46fc-b05e-1348cfd4f318-kserve-provision-location\") pod \"isvc-sklearn-s3-predictor-dd446d6b9-pj5zf\" (UID: \"b76b321f-5067-46fc-b05e-1348cfd4f318\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-dd446d6b9-pj5zf"
Jul 27 22:31:38.390605 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:31:38.390500    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b76b321f-5067-46fc-b05e-1348cfd4f318-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-predictor-dd446d6b9-pj5zf\" (UID: \"b76b321f-5067-46fc-b05e-1348cfd4f318\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-dd446d6b9-pj5zf"
Jul 27 22:31:38.390605 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:31:38.390529    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9t8jl\" (UniqueName: \"kubernetes.io/projected/b76b321f-5067-46fc-b05e-1348cfd4f318-kube-api-access-9t8jl\") pod \"isvc-sklearn-s3-predictor-dd446d6b9-pj5zf\" (UID: \"b76b321f-5067-46fc-b05e-1348cfd4f318\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-dd446d6b9-pj5zf"
Jul 27 22:31:38.491683 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:31:38.491645    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b76b321f-5067-46fc-b05e-1348cfd4f318-kserve-provision-location\") pod \"isvc-sklearn-s3-predictor-dd446d6b9-pj5zf\" (UID: \"b76b321f-5067-46fc-b05e-1348cfd4f318\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-dd446d6b9-pj5zf"
Jul 27 22:31:38.491900 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:31:38.491702    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b76b321f-5067-46fc-b05e-1348cfd4f318-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-predictor-dd446d6b9-pj5zf\" (UID: \"b76b321f-5067-46fc-b05e-1348cfd4f318\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-dd446d6b9-pj5zf"
Jul 27 22:31:38.491900 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:31:38.491735    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9t8jl\" (UniqueName: \"kubernetes.io/projected/b76b321f-5067-46fc-b05e-1348cfd4f318-kube-api-access-9t8jl\") pod \"isvc-sklearn-s3-predictor-dd446d6b9-pj5zf\" (UID: \"b76b321f-5067-46fc-b05e-1348cfd4f318\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-dd446d6b9-pj5zf"
Jul 27 22:31:38.491900 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:31:38.491769    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b76b321f-5067-46fc-b05e-1348cfd4f318-proxy-tls\") pod \"isvc-sklearn-s3-predictor-dd446d6b9-pj5zf\" (UID: \"b76b321f-5067-46fc-b05e-1348cfd4f318\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-dd446d6b9-pj5zf"
Jul 27 22:31:38.492144 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:31:38.492122    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b76b321f-5067-46fc-b05e-1348cfd4f318-kserve-provision-location\") pod \"isvc-sklearn-s3-predictor-dd446d6b9-pj5zf\" (UID: \"b76b321f-5067-46fc-b05e-1348cfd4f318\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-dd446d6b9-pj5zf"
Jul 27 22:31:38.492427 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:31:38.492408    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b76b321f-5067-46fc-b05e-1348cfd4f318-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-predictor-dd446d6b9-pj5zf\" (UID: \"b76b321f-5067-46fc-b05e-1348cfd4f318\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-dd446d6b9-pj5zf"
Jul 27 22:31:38.494136 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:31:38.494117    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b76b321f-5067-46fc-b05e-1348cfd4f318-proxy-tls\") pod \"isvc-sklearn-s3-predictor-dd446d6b9-pj5zf\" (UID: \"b76b321f-5067-46fc-b05e-1348cfd4f318\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-dd446d6b9-pj5zf"
Jul 27 22:31:38.500270 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:31:38.500242    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9t8jl\" (UniqueName: \"kubernetes.io/projected/b76b321f-5067-46fc-b05e-1348cfd4f318-kube-api-access-9t8jl\") pod \"isvc-sklearn-s3-predictor-dd446d6b9-pj5zf\" (UID: \"b76b321f-5067-46fc-b05e-1348cfd4f318\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-dd446d6b9-pj5zf"
Jul 27 22:31:38.649767 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:31:38.649721    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-dd446d6b9-pj5zf"
Jul 27 22:31:38.791021 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:31:38.790996    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-predictor-dd446d6b9-pj5zf"]
Jul 27 22:31:38.793379 ip-10-0-131-133 kubenswrapper[2559]: W0727 22:31:38.793345    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb76b321f_5067_46fc_b05e_1348cfd4f318.slice/crio-a06969480e867a62c4bd54f630ac42e8bf468bf6a7d3c917e815f6d5a1f0978d WatchSource:0}: Error finding container a06969480e867a62c4bd54f630ac42e8bf468bf6a7d3c917e815f6d5a1f0978d: Status 404 returned error can't find the container with id a06969480e867a62c4bd54f630ac42e8bf468bf6a7d3c917e815f6d5a1f0978d
Jul 27 22:31:39.448083 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:31:39.446898    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-dd446d6b9-pj5zf" event={"ID":"b76b321f-5067-46fc-b05e-1348cfd4f318","Type":"ContainerStarted","Data":"558bc351fabc8d4a8aca5ac1e1a21a5445e24030de7c37ac804deb21ac9daeb7"}
Jul 27 22:31:39.448083 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:31:39.447054    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-dd446d6b9-pj5zf" event={"ID":"b76b321f-5067-46fc-b05e-1348cfd4f318","Type":"ContainerStarted","Data":"a06969480e867a62c4bd54f630ac42e8bf468bf6a7d3c917e815f6d5a1f0978d"}
Jul 27 22:31:40.452554 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:31:40.452519    2559 generic.go:356] "Generic (PLEG): container finished" podID="b76b321f-5067-46fc-b05e-1348cfd4f318" containerID="558bc351fabc8d4a8aca5ac1e1a21a5445e24030de7c37ac804deb21ac9daeb7" exitCode=0
Jul 27 22:31:40.452971 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:31:40.452601    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-dd446d6b9-pj5zf" event={"ID":"b76b321f-5067-46fc-b05e-1348cfd4f318","Type":"ContainerDied","Data":"558bc351fabc8d4a8aca5ac1e1a21a5445e24030de7c37ac804deb21ac9daeb7"}
Jul 27 22:31:41.459145 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:31:41.459114    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-dd446d6b9-pj5zf" event={"ID":"b76b321f-5067-46fc-b05e-1348cfd4f318","Type":"ContainerStarted","Data":"8c234b13a773b3a492434941aa2081975f7e77a4435a36d802568057b9168766"}
Jul 27 22:31:41.459636 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:31:41.459177    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-dd446d6b9-pj5zf" event={"ID":"b76b321f-5067-46fc-b05e-1348cfd4f318","Type":"ContainerStarted","Data":"852185f959f4be9e0deee47eb0427a6ebb05f28ed4fb0e9a4566bd623ce228b8"}
Jul 27 22:31:41.459636 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:31:41.459226    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-dd446d6b9-pj5zf"
Jul 27 22:31:41.459636 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:31:41.459259    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-dd446d6b9-pj5zf"
Jul 27 22:31:41.460774 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:31:41.460743    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-dd446d6b9-pj5zf" podUID="b76b321f-5067-46fc-b05e-1348cfd4f318" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.46:8080: connect: connection refused"
Jul 27 22:31:41.482898 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:31:41.482851    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-dd446d6b9-pj5zf" podStartSLOduration=2.529836897 podStartE2EDuration="3.482836462s" totalImagesPullingTime="0s" totalInitContainerRuntime="952.999565ms" isStatefulPod=false podCreationTimestamp="2026-07-27 22:31:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 22:31:41.481966221 +0000 UTC m=+5230.061150228" watchObservedRunningTime="2026-07-27 22:31:41.482836462 +0000 UTC m=+5230.062020468"
Jul 27 22:31:42.464264 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:31:42.464202    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-dd446d6b9-pj5zf" podUID="b76b321f-5067-46fc-b05e-1348cfd4f318" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.46:8080: connect: connection refused"
Jul 27 22:31:47.469531 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:31:47.469503    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-dd446d6b9-pj5zf"
Jul 27 22:31:47.470134 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:31:47.470108    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-dd446d6b9-pj5zf" podUID="b76b321f-5067-46fc-b05e-1348cfd4f318" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.46:8080: connect: connection refused"
Jul 27 22:31:57.470459 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:31:57.470418    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-dd446d6b9-pj5zf" podUID="b76b321f-5067-46fc-b05e-1348cfd4f318" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.46:8080: connect: connection refused"
Jul 27 22:32:07.470235 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:32:07.470189    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-dd446d6b9-pj5zf" podUID="b76b321f-5067-46fc-b05e-1348cfd4f318" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.46:8080: connect: connection refused"
Jul 27 22:32:17.470006 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:32:17.469969    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-dd446d6b9-pj5zf" podUID="b76b321f-5067-46fc-b05e-1348cfd4f318" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.46:8080: connect: connection refused"
Jul 27 22:32:27.470946 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:32:27.470908    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-dd446d6b9-pj5zf" podUID="b76b321f-5067-46fc-b05e-1348cfd4f318" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.46:8080: connect: connection refused"
Jul 27 22:32:37.470981 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:32:37.470941    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-dd446d6b9-pj5zf" podUID="b76b321f-5067-46fc-b05e-1348cfd4f318" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.46:8080: connect: connection refused"
Jul 27 22:32:47.470062 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:32:47.470023    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-dd446d6b9-pj5zf" podUID="b76b321f-5067-46fc-b05e-1348cfd4f318" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.46:8080: connect: connection refused"
Jul 27 22:32:57.471365 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:32:57.471337    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-dd446d6b9-pj5zf"
Jul 27 22:32:58.396971 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:32:58.396938    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-predictor-dd446d6b9-pj5zf"]
Jul 27 22:32:58.397176 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:32:58.397151    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-dd446d6b9-pj5zf" podUID="b76b321f-5067-46fc-b05e-1348cfd4f318" containerName="kserve-container" containerID="cri-o://852185f959f4be9e0deee47eb0427a6ebb05f28ed4fb0e9a4566bd623ce228b8" gracePeriod=30
Jul 27 22:32:58.397269 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:32:58.397193    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-dd446d6b9-pj5zf" podUID="b76b321f-5067-46fc-b05e-1348cfd4f318" containerName="kube-rbac-proxy" containerID="cri-o://8c234b13a773b3a492434941aa2081975f7e77a4435a36d802568057b9168766" gracePeriod=30
Jul 27 22:32:58.550392 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:32:58.550353    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-665d8df985-gff9j"]
Jul 27 22:32:58.554111 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:32:58.554088    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-665d8df985-gff9j"
Jul 27 22:32:58.556420 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:32:58.556394    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-tls-global-pass-kube-rbac-proxy-sar-config\""
Jul 27 22:32:58.556565 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:32:58.556532    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-tls-global-pass-predictor-serving-cert\""
Jul 27 22:32:58.556674 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:32:58.556655    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 27 22:32:58.565644 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:32:58.565618    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-665d8df985-gff9j"]
Jul 27 22:32:58.618203 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:32:58.618157    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7mcv7\" (UniqueName: \"kubernetes.io/projected/b6251c55-a270-4a42-85fe-698989b81c7a-kube-api-access-7mcv7\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-665d8df985-gff9j\" (UID: \"b6251c55-a270-4a42-85fe-698989b81c7a\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-665d8df985-gff9j"
Jul 27 22:32:58.618203 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:32:58.618209    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b6251c55-a270-4a42-85fe-698989b81c7a-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-665d8df985-gff9j\" (UID: \"b6251c55-a270-4a42-85fe-698989b81c7a\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-665d8df985-gff9j"
Jul 27 22:32:58.618505 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:32:58.618342    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b6251c55-a270-4a42-85fe-698989b81c7a-cabundle-cert\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-665d8df985-gff9j\" (UID: \"b6251c55-a270-4a42-85fe-698989b81c7a\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-665d8df985-gff9j"
Jul 27 22:32:58.618505 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:32:58.618393    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b6251c55-a270-4a42-85fe-698989b81c7a-proxy-tls\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-665d8df985-gff9j\" (UID: \"b6251c55-a270-4a42-85fe-698989b81c7a\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-665d8df985-gff9j"
Jul 27 22:32:58.618505 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:32:58.618465    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b6251c55-a270-4a42-85fe-698989b81c7a-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-665d8df985-gff9j\" (UID: \"b6251c55-a270-4a42-85fe-698989b81c7a\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-665d8df985-gff9j"
Jul 27 22:32:58.719744 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:32:58.719647    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b6251c55-a270-4a42-85fe-698989b81c7a-cabundle-cert\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-665d8df985-gff9j\" (UID: \"b6251c55-a270-4a42-85fe-698989b81c7a\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-665d8df985-gff9j"
Jul 27 22:32:58.719744 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:32:58.719694    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b6251c55-a270-4a42-85fe-698989b81c7a-proxy-tls\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-665d8df985-gff9j\" (UID: \"b6251c55-a270-4a42-85fe-698989b81c7a\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-665d8df985-gff9j"
Jul 27 22:32:58.719744 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:32:58.719738    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b6251c55-a270-4a42-85fe-698989b81c7a-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-665d8df985-gff9j\" (UID: \"b6251c55-a270-4a42-85fe-698989b81c7a\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-665d8df985-gff9j"
Jul 27 22:32:58.720044 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:32:58.719759    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7mcv7\" (UniqueName: \"kubernetes.io/projected/b6251c55-a270-4a42-85fe-698989b81c7a-kube-api-access-7mcv7\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-665d8df985-gff9j\" (UID: \"b6251c55-a270-4a42-85fe-698989b81c7a\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-665d8df985-gff9j"
Jul 27 22:32:58.720044 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:32:58.719787    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b6251c55-a270-4a42-85fe-698989b81c7a-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-665d8df985-gff9j\" (UID: \"b6251c55-a270-4a42-85fe-698989b81c7a\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-665d8df985-gff9j"
Jul 27 22:32:58.720044 ip-10-0-131-133 kubenswrapper[2559]: E0727 22:32:58.719916    2559 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-serving-cert: secret "isvc-sklearn-s3-tls-global-pass-predictor-serving-cert" not found
Jul 27 22:32:58.720044 ip-10-0-131-133 kubenswrapper[2559]: E0727 22:32:58.720003    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b6251c55-a270-4a42-85fe-698989b81c7a-proxy-tls podName:b6251c55-a270-4a42-85fe-698989b81c7a nodeName:}" failed. No retries permitted until 2026-07-27 22:32:59.219979468 +0000 UTC m=+5307.799163473 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/b6251c55-a270-4a42-85fe-698989b81c7a-proxy-tls") pod "isvc-sklearn-s3-tls-global-pass-predictor-665d8df985-gff9j" (UID: "b6251c55-a270-4a42-85fe-698989b81c7a") : secret "isvc-sklearn-s3-tls-global-pass-predictor-serving-cert" not found
Jul 27 22:32:58.720296 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:32:58.720203    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b6251c55-a270-4a42-85fe-698989b81c7a-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-665d8df985-gff9j\" (UID: \"b6251c55-a270-4a42-85fe-698989b81c7a\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-665d8df985-gff9j"
Jul 27 22:32:58.720421 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:32:58.720399    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b6251c55-a270-4a42-85fe-698989b81c7a-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-665d8df985-gff9j\" (UID: \"b6251c55-a270-4a42-85fe-698989b81c7a\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-665d8df985-gff9j"
Jul 27 22:32:58.720514 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:32:58.720496    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b6251c55-a270-4a42-85fe-698989b81c7a-cabundle-cert\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-665d8df985-gff9j\" (UID: \"b6251c55-a270-4a42-85fe-698989b81c7a\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-665d8df985-gff9j"
Jul 27 22:32:58.728145 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:32:58.728115    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7mcv7\" (UniqueName: \"kubernetes.io/projected/b6251c55-a270-4a42-85fe-698989b81c7a-kube-api-access-7mcv7\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-665d8df985-gff9j\" (UID: \"b6251c55-a270-4a42-85fe-698989b81c7a\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-665d8df985-gff9j"
Jul 27 22:32:58.823264 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:32:58.823198    2559 generic.go:356] "Generic (PLEG): container finished" podID="b76b321f-5067-46fc-b05e-1348cfd4f318" containerID="8c234b13a773b3a492434941aa2081975f7e77a4435a36d802568057b9168766" exitCode=2
Jul 27 22:32:58.823434 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:32:58.823280    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-dd446d6b9-pj5zf" event={"ID":"b76b321f-5067-46fc-b05e-1348cfd4f318","Type":"ContainerDied","Data":"8c234b13a773b3a492434941aa2081975f7e77a4435a36d802568057b9168766"}
Jul 27 22:32:59.224041 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:32:59.224011    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b6251c55-a270-4a42-85fe-698989b81c7a-proxy-tls\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-665d8df985-gff9j\" (UID: \"b6251c55-a270-4a42-85fe-698989b81c7a\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-665d8df985-gff9j"
Jul 27 22:32:59.226484 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:32:59.226451    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b6251c55-a270-4a42-85fe-698989b81c7a-proxy-tls\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-665d8df985-gff9j\" (UID: \"b6251c55-a270-4a42-85fe-698989b81c7a\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-665d8df985-gff9j"
Jul 27 22:32:59.465346 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:32:59.465294    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-665d8df985-gff9j"
Jul 27 22:32:59.600948 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:32:59.600917    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-665d8df985-gff9j"]
Jul 27 22:32:59.603670 ip-10-0-131-133 kubenswrapper[2559]: W0727 22:32:59.603638    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb6251c55_a270_4a42_85fe_698989b81c7a.slice/crio-4d3faf33e86199fd9a9772a9fb2e98ff58245f320beabe492a09e659d3fc6346 WatchSource:0}: Error finding container 4d3faf33e86199fd9a9772a9fb2e98ff58245f320beabe492a09e659d3fc6346: Status 404 returned error can't find the container with id 4d3faf33e86199fd9a9772a9fb2e98ff58245f320beabe492a09e659d3fc6346
Jul 27 22:32:59.831567 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:32:59.831470    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-665d8df985-gff9j" event={"ID":"b6251c55-a270-4a42-85fe-698989b81c7a","Type":"ContainerStarted","Data":"16be5bbf977cfa32eb2240a2733f85aeb8037113a4d471ef7bc632132215708b"}
Jul 27 22:32:59.831567 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:32:59.831528    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-665d8df985-gff9j" event={"ID":"b6251c55-a270-4a42-85fe-698989b81c7a","Type":"ContainerStarted","Data":"4d3faf33e86199fd9a9772a9fb2e98ff58245f320beabe492a09e659d3fc6346"}
Jul 27 22:33:00.838034 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:33:00.837999    2559 generic.go:356] "Generic (PLEG): container finished" podID="b6251c55-a270-4a42-85fe-698989b81c7a" containerID="16be5bbf977cfa32eb2240a2733f85aeb8037113a4d471ef7bc632132215708b" exitCode=0
Jul 27 22:33:00.838463 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:33:00.838037    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-665d8df985-gff9j" event={"ID":"b6251c55-a270-4a42-85fe-698989b81c7a","Type":"ContainerDied","Data":"16be5bbf977cfa32eb2240a2733f85aeb8037113a4d471ef7bc632132215708b"}
Jul 27 22:33:01.845088 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:33:01.845049    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-665d8df985-gff9j" event={"ID":"b6251c55-a270-4a42-85fe-698989b81c7a","Type":"ContainerStarted","Data":"5494cf1127f4589c6e8768631748eb155a99cbf468cd3ead83ea622e0845fc69"}
Jul 27 22:33:01.845550 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:33:01.845122    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-665d8df985-gff9j" event={"ID":"b6251c55-a270-4a42-85fe-698989b81c7a","Type":"ContainerStarted","Data":"b3ea3655c64827007933c6b62b35e66b58a6afb5958b40b9cf1f6a3f0a658893"}
Jul 27 22:33:01.845550 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:33:01.845399    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-665d8df985-gff9j"
Jul 27 22:33:01.845550 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:33:01.845542    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-665d8df985-gff9j"
Jul 27 22:33:01.847273 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:33:01.847241    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-665d8df985-gff9j" podUID="b6251c55-a270-4a42-85fe-698989b81c7a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.47:8080: connect: connection refused"
Jul 27 22:33:01.863842 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:33:01.863788    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-665d8df985-gff9j" podStartSLOduration=2.901346869 podStartE2EDuration="3.86376911s" totalImagesPullingTime="0s" totalInitContainerRuntime="962.422241ms" isStatefulPod=false podCreationTimestamp="2026-07-27 22:32:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 22:33:01.861798051 +0000 UTC m=+5310.440982068" watchObservedRunningTime="2026-07-27 22:33:01.86376911 +0000 UTC m=+5310.442953117"
Jul 27 22:33:02.465209 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:33:02.465166    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-dd446d6b9-pj5zf" podUID="b76b321f-5067-46fc-b05e-1348cfd4f318" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.46:8643/healthz\": dial tcp 10.132.0.46:8643: connect: connection refused"
Jul 27 22:33:02.851102 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:33:02.851003    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-665d8df985-gff9j" podUID="b6251c55-a270-4a42-85fe-698989b81c7a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.47:8080: connect: connection refused"
Jul 27 22:33:03.560026 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:33:03.560002    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-dd446d6b9-pj5zf"
Jul 27 22:33:03.665559 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:33:03.665520    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b76b321f-5067-46fc-b05e-1348cfd4f318-proxy-tls\") pod \"b76b321f-5067-46fc-b05e-1348cfd4f318\" (UID: \"b76b321f-5067-46fc-b05e-1348cfd4f318\") "
Jul 27 22:33:03.665559 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:33:03.665561    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b76b321f-5067-46fc-b05e-1348cfd4f318-kube-rbac-proxy-sar-config\") pod \"b76b321f-5067-46fc-b05e-1348cfd4f318\" (UID: \"b76b321f-5067-46fc-b05e-1348cfd4f318\") "
Jul 27 22:33:03.665810 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:33:03.665611    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9t8jl\" (UniqueName: \"kubernetes.io/projected/b76b321f-5067-46fc-b05e-1348cfd4f318-kube-api-access-9t8jl\") pod \"b76b321f-5067-46fc-b05e-1348cfd4f318\" (UID: \"b76b321f-5067-46fc-b05e-1348cfd4f318\") "
Jul 27 22:33:03.665810 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:33:03.665647    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b76b321f-5067-46fc-b05e-1348cfd4f318-kserve-provision-location\") pod \"b76b321f-5067-46fc-b05e-1348cfd4f318\" (UID: \"b76b321f-5067-46fc-b05e-1348cfd4f318\") "
Jul 27 22:33:03.666004 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:33:03.665977    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b76b321f-5067-46fc-b05e-1348cfd4f318-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "b76b321f-5067-46fc-b05e-1348cfd4f318" (UID: "b76b321f-5067-46fc-b05e-1348cfd4f318"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 22:33:03.666072 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:33:03.666007    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b76b321f-5067-46fc-b05e-1348cfd4f318-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "b76b321f-5067-46fc-b05e-1348cfd4f318" (UID: "b76b321f-5067-46fc-b05e-1348cfd4f318"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 22:33:03.667762 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:33:03.667732    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b76b321f-5067-46fc-b05e-1348cfd4f318-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "b76b321f-5067-46fc-b05e-1348cfd4f318" (UID: "b76b321f-5067-46fc-b05e-1348cfd4f318"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 22:33:03.667860 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:33:03.667845    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b76b321f-5067-46fc-b05e-1348cfd4f318-kube-api-access-9t8jl" (OuterVolumeSpecName: "kube-api-access-9t8jl") pod "b76b321f-5067-46fc-b05e-1348cfd4f318" (UID: "b76b321f-5067-46fc-b05e-1348cfd4f318"). InnerVolumeSpecName "kube-api-access-9t8jl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 22:33:03.767262 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:33:03.767192    2559 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b76b321f-5067-46fc-b05e-1348cfd4f318-proxy-tls\") on node \"ip-10-0-131-133.ec2.internal\" DevicePath \"\""
Jul 27 22:33:03.767262 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:33:03.767257    2559 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b76b321f-5067-46fc-b05e-1348cfd4f318-kube-rbac-proxy-sar-config\") on node \"ip-10-0-131-133.ec2.internal\" DevicePath \"\""
Jul 27 22:33:03.767262 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:33:03.767268    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9t8jl\" (UniqueName: \"kubernetes.io/projected/b76b321f-5067-46fc-b05e-1348cfd4f318-kube-api-access-9t8jl\") on node \"ip-10-0-131-133.ec2.internal\" DevicePath \"\""
Jul 27 22:33:03.767510 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:33:03.767277    2559 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b76b321f-5067-46fc-b05e-1348cfd4f318-kserve-provision-location\") on node \"ip-10-0-131-133.ec2.internal\" DevicePath \"\""
Jul 27 22:33:03.857759 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:33:03.857722    2559 generic.go:356] "Generic (PLEG): container finished" podID="b76b321f-5067-46fc-b05e-1348cfd4f318" containerID="852185f959f4be9e0deee47eb0427a6ebb05f28ed4fb0e9a4566bd623ce228b8" exitCode=0
Jul 27 22:33:03.858152 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:33:03.857781    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-dd446d6b9-pj5zf" event={"ID":"b76b321f-5067-46fc-b05e-1348cfd4f318","Type":"ContainerDied","Data":"852185f959f4be9e0deee47eb0427a6ebb05f28ed4fb0e9a4566bd623ce228b8"}
Jul 27 22:33:03.858152 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:33:03.857845    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-dd446d6b9-pj5zf" event={"ID":"b76b321f-5067-46fc-b05e-1348cfd4f318","Type":"ContainerDied","Data":"a06969480e867a62c4bd54f630ac42e8bf468bf6a7d3c917e815f6d5a1f0978d"}
Jul 27 22:33:03.858152 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:33:03.857787    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-dd446d6b9-pj5zf"
Jul 27 22:33:03.858152 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:33:03.857887    2559 scope.go:117] "RemoveContainer" containerID="8c234b13a773b3a492434941aa2081975f7e77a4435a36d802568057b9168766"
Jul 27 22:33:03.867025 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:33:03.867006    2559 scope.go:117] "RemoveContainer" containerID="852185f959f4be9e0deee47eb0427a6ebb05f28ed4fb0e9a4566bd623ce228b8"
Jul 27 22:33:03.875317 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:33:03.875295    2559 scope.go:117] "RemoveContainer" containerID="558bc351fabc8d4a8aca5ac1e1a21a5445e24030de7c37ac804deb21ac9daeb7"
Jul 27 22:33:03.879840 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:33:03.879813    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-predictor-dd446d6b9-pj5zf"]
Jul 27 22:33:03.883423 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:33:03.883399    2559 scope.go:117] "RemoveContainer" containerID="8c234b13a773b3a492434941aa2081975f7e77a4435a36d802568057b9168766"
Jul 27 22:33:03.883732 ip-10-0-131-133 kubenswrapper[2559]: E0727 22:33:03.883708    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8c234b13a773b3a492434941aa2081975f7e77a4435a36d802568057b9168766\": container with ID starting with 8c234b13a773b3a492434941aa2081975f7e77a4435a36d802568057b9168766 not found: ID does not exist" containerID="8c234b13a773b3a492434941aa2081975f7e77a4435a36d802568057b9168766"
Jul 27 22:33:03.883805 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:33:03.883742    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8c234b13a773b3a492434941aa2081975f7e77a4435a36d802568057b9168766"} err="failed to get container status \"8c234b13a773b3a492434941aa2081975f7e77a4435a36d802568057b9168766\": rpc error: code = NotFound desc = could not find container \"8c234b13a773b3a492434941aa2081975f7e77a4435a36d802568057b9168766\": container with ID starting with 8c234b13a773b3a492434941aa2081975f7e77a4435a36d802568057b9168766 not found: ID does not exist"
Jul 27 22:33:03.883805 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:33:03.883762    2559 scope.go:117] "RemoveContainer" containerID="852185f959f4be9e0deee47eb0427a6ebb05f28ed4fb0e9a4566bd623ce228b8"
Jul 27 22:33:03.884105 ip-10-0-131-133 kubenswrapper[2559]: E0727 22:33:03.884085    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"852185f959f4be9e0deee47eb0427a6ebb05f28ed4fb0e9a4566bd623ce228b8\": container with ID starting with 852185f959f4be9e0deee47eb0427a6ebb05f28ed4fb0e9a4566bd623ce228b8 not found: ID does not exist" containerID="852185f959f4be9e0deee47eb0427a6ebb05f28ed4fb0e9a4566bd623ce228b8"
Jul 27 22:33:03.884164 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:33:03.884112    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"852185f959f4be9e0deee47eb0427a6ebb05f28ed4fb0e9a4566bd623ce228b8"} err="failed to get container status \"852185f959f4be9e0deee47eb0427a6ebb05f28ed4fb0e9a4566bd623ce228b8\": rpc error: code = NotFound desc = could not find container \"852185f959f4be9e0deee47eb0427a6ebb05f28ed4fb0e9a4566bd623ce228b8\": container with ID starting with 852185f959f4be9e0deee47eb0427a6ebb05f28ed4fb0e9a4566bd623ce228b8 not found: ID does not exist"
Jul 27 22:33:03.884164 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:33:03.884129    2559 scope.go:117] "RemoveContainer" containerID="558bc351fabc8d4a8aca5ac1e1a21a5445e24030de7c37ac804deb21ac9daeb7"
Jul 27 22:33:03.884451 ip-10-0-131-133 kubenswrapper[2559]: E0727 22:33:03.884410    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"558bc351fabc8d4a8aca5ac1e1a21a5445e24030de7c37ac804deb21ac9daeb7\": container with ID starting with 558bc351fabc8d4a8aca5ac1e1a21a5445e24030de7c37ac804deb21ac9daeb7 not found: ID does not exist" containerID="558bc351fabc8d4a8aca5ac1e1a21a5445e24030de7c37ac804deb21ac9daeb7"
Jul 27 22:33:03.884451 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:33:03.884433    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"558bc351fabc8d4a8aca5ac1e1a21a5445e24030de7c37ac804deb21ac9daeb7"} err="failed to get container status \"558bc351fabc8d4a8aca5ac1e1a21a5445e24030de7c37ac804deb21ac9daeb7\": rpc error: code = NotFound desc = could not find container \"558bc351fabc8d4a8aca5ac1e1a21a5445e24030de7c37ac804deb21ac9daeb7\": container with ID starting with 558bc351fabc8d4a8aca5ac1e1a21a5445e24030de7c37ac804deb21ac9daeb7 not found: ID does not exist"
Jul 27 22:33:03.886497 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:33:03.886477    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-predictor-dd446d6b9-pj5zf"]
Jul 27 22:33:03.982545 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:33:03.982462    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b76b321f-5067-46fc-b05e-1348cfd4f318" path="/var/lib/kubelet/pods/b76b321f-5067-46fc-b05e-1348cfd4f318/volumes"
Jul 27 22:33:07.855833 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:33:07.855748    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-665d8df985-gff9j"
Jul 27 22:33:07.856274 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:33:07.856247    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-665d8df985-gff9j" podUID="b6251c55-a270-4a42-85fe-698989b81c7a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.47:8080: connect: connection refused"
Jul 27 22:33:17.857235 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:33:17.857171    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-665d8df985-gff9j" podUID="b6251c55-a270-4a42-85fe-698989b81c7a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.47:8080: connect: connection refused"
Jul 27 22:33:27.856706 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:33:27.856664    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-665d8df985-gff9j" podUID="b6251c55-a270-4a42-85fe-698989b81c7a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.47:8080: connect: connection refused"
Jul 27 22:33:37.857118 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:33:37.857078    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-665d8df985-gff9j" podUID="b6251c55-a270-4a42-85fe-698989b81c7a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.47:8080: connect: connection refused"
Jul 27 22:33:47.856946 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:33:47.856902    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-665d8df985-gff9j" podUID="b6251c55-a270-4a42-85fe-698989b81c7a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.47:8080: connect: connection refused"
Jul 27 22:33:57.856285 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:33:57.856249    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-665d8df985-gff9j" podUID="b6251c55-a270-4a42-85fe-698989b81c7a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.47:8080: connect: connection refused"
Jul 27 22:34:07.856403 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:34:07.856367    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-665d8df985-gff9j" podUID="b6251c55-a270-4a42-85fe-698989b81c7a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.47:8080: connect: connection refused"
Jul 27 22:34:17.857376 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:34:17.857339    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-665d8df985-gff9j"
Jul 27 22:34:18.604045 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:34:18.604010    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-665d8df985-gff9j"]
Jul 27 22:34:18.604298 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:34:18.604268    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-665d8df985-gff9j" podUID="b6251c55-a270-4a42-85fe-698989b81c7a" containerName="kserve-container" containerID="cri-o://b3ea3655c64827007933c6b62b35e66b58a6afb5958b40b9cf1f6a3f0a658893" gracePeriod=30
Jul 27 22:34:18.604444 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:34:18.604305    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-665d8df985-gff9j" podUID="b6251c55-a270-4a42-85fe-698989b81c7a" containerName="kube-rbac-proxy" containerID="cri-o://5494cf1127f4589c6e8768631748eb155a99cbf468cd3ead83ea622e0845fc69" gracePeriod=30
Jul 27 22:34:19.220435 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:34:19.220403    2559 generic.go:356] "Generic (PLEG): container finished" podID="b6251c55-a270-4a42-85fe-698989b81c7a" containerID="5494cf1127f4589c6e8768631748eb155a99cbf468cd3ead83ea622e0845fc69" exitCode=2
Jul 27 22:34:19.220833 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:34:19.220484    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-665d8df985-gff9j" event={"ID":"b6251c55-a270-4a42-85fe-698989b81c7a","Type":"ContainerDied","Data":"5494cf1127f4589c6e8768631748eb155a99cbf468cd3ead83ea622e0845fc69"}
Jul 27 22:34:19.673000 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:34:19.672967    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-7d56b5c8fc-4np82"]
Jul 27 22:34:19.673373 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:34:19.673359    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b76b321f-5067-46fc-b05e-1348cfd4f318" containerName="kserve-container"
Jul 27 22:34:19.673430 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:34:19.673375    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="b76b321f-5067-46fc-b05e-1348cfd4f318" containerName="kserve-container"
Jul 27 22:34:19.673430 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:34:19.673394    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b76b321f-5067-46fc-b05e-1348cfd4f318" containerName="storage-initializer"
Jul 27 22:34:19.673430 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:34:19.673399    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="b76b321f-5067-46fc-b05e-1348cfd4f318" containerName="storage-initializer"
Jul 27 22:34:19.673430 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:34:19.673405    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b76b321f-5067-46fc-b05e-1348cfd4f318" containerName="kube-rbac-proxy"
Jul 27 22:34:19.673430 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:34:19.673410    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="b76b321f-5067-46fc-b05e-1348cfd4f318" containerName="kube-rbac-proxy"
Jul 27 22:34:19.673602 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:34:19.673473    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="b76b321f-5067-46fc-b05e-1348cfd4f318" containerName="kube-rbac-proxy"
Jul 27 22:34:19.673602 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:34:19.673482    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="b76b321f-5067-46fc-b05e-1348cfd4f318" containerName="kserve-container"
Jul 27 22:34:19.676663 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:34:19.676645    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-7d56b5c8fc-4np82"
Jul 27 22:34:19.678840 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:34:19.678807    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-tls-global-fail-kube-rbac-proxy-sar-config\""
Jul 27 22:34:19.678917 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:34:19.678889    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-tls-global-fail-predictor-serving-cert\""
Jul 27 22:34:19.686320 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:34:19.686293    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-7d56b5c8fc-4np82"]
Jul 27 22:34:19.854482 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:34:19.854445    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e5668592-7da5-4fff-803e-066c1a216ffa-proxy-tls\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-7d56b5c8fc-4np82\" (UID: \"e5668592-7da5-4fff-803e-066c1a216ffa\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-7d56b5c8fc-4np82"
Jul 27 22:34:19.854482 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:34:19.854490    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e5668592-7da5-4fff-803e-066c1a216ffa-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-7d56b5c8fc-4np82\" (UID: \"e5668592-7da5-4fff-803e-066c1a216ffa\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-7d56b5c8fc-4np82"
Jul 27 22:34:19.854712 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:34:19.854558    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dx8dz\" (UniqueName: \"kubernetes.io/projected/e5668592-7da5-4fff-803e-066c1a216ffa-kube-api-access-dx8dz\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-7d56b5c8fc-4np82\" (UID: \"e5668592-7da5-4fff-803e-066c1a216ffa\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-7d56b5c8fc-4np82"
Jul 27 22:34:19.854712 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:34:19.854576    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e5668592-7da5-4fff-803e-066c1a216ffa-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-7d56b5c8fc-4np82\" (UID: \"e5668592-7da5-4fff-803e-066c1a216ffa\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-7d56b5c8fc-4np82"
Jul 27 22:34:19.955384 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:34:19.955278    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dx8dz\" (UniqueName: \"kubernetes.io/projected/e5668592-7da5-4fff-803e-066c1a216ffa-kube-api-access-dx8dz\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-7d56b5c8fc-4np82\" (UID: \"e5668592-7da5-4fff-803e-066c1a216ffa\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-7d56b5c8fc-4np82"
Jul 27 22:34:19.955384 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:34:19.955332    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e5668592-7da5-4fff-803e-066c1a216ffa-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-7d56b5c8fc-4np82\" (UID: \"e5668592-7da5-4fff-803e-066c1a216ffa\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-7d56b5c8fc-4np82"
Jul 27 22:34:19.955599 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:34:19.955416    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e5668592-7da5-4fff-803e-066c1a216ffa-proxy-tls\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-7d56b5c8fc-4np82\" (UID: \"e5668592-7da5-4fff-803e-066c1a216ffa\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-7d56b5c8fc-4np82"
Jul 27 22:34:19.955599 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:34:19.955447    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e5668592-7da5-4fff-803e-066c1a216ffa-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-7d56b5c8fc-4np82\" (UID: \"e5668592-7da5-4fff-803e-066c1a216ffa\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-7d56b5c8fc-4np82"
Jul 27 22:34:19.955694 ip-10-0-131-133 kubenswrapper[2559]: E0727 22:34:19.955592    2559 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-serving-cert: secret "isvc-sklearn-s3-tls-global-fail-predictor-serving-cert" not found
Jul 27 22:34:19.955694 ip-10-0-131-133 kubenswrapper[2559]: E0727 22:34:19.955670    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e5668592-7da5-4fff-803e-066c1a216ffa-proxy-tls podName:e5668592-7da5-4fff-803e-066c1a216ffa nodeName:}" failed. No retries permitted until 2026-07-27 22:34:20.455650438 +0000 UTC m=+5389.034834440 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/e5668592-7da5-4fff-803e-066c1a216ffa-proxy-tls") pod "isvc-sklearn-s3-tls-global-fail-predictor-7d56b5c8fc-4np82" (UID: "e5668592-7da5-4fff-803e-066c1a216ffa") : secret "isvc-sklearn-s3-tls-global-fail-predictor-serving-cert" not found
Jul 27 22:34:19.955840 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:34:19.955818    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e5668592-7da5-4fff-803e-066c1a216ffa-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-7d56b5c8fc-4np82\" (UID: \"e5668592-7da5-4fff-803e-066c1a216ffa\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-7d56b5c8fc-4np82"
Jul 27 22:34:19.956089 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:34:19.956072    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e5668592-7da5-4fff-803e-066c1a216ffa-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-7d56b5c8fc-4np82\" (UID: \"e5668592-7da5-4fff-803e-066c1a216ffa\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-7d56b5c8fc-4np82"
Jul 27 22:34:19.965592 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:34:19.965568    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dx8dz\" (UniqueName: \"kubernetes.io/projected/e5668592-7da5-4fff-803e-066c1a216ffa-kube-api-access-dx8dz\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-7d56b5c8fc-4np82\" (UID: \"e5668592-7da5-4fff-803e-066c1a216ffa\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-7d56b5c8fc-4np82"
Jul 27 22:34:20.460724 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:34:20.460683    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e5668592-7da5-4fff-803e-066c1a216ffa-proxy-tls\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-7d56b5c8fc-4np82\" (UID: \"e5668592-7da5-4fff-803e-066c1a216ffa\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-7d56b5c8fc-4np82"
Jul 27 22:34:20.463061 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:34:20.463041    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e5668592-7da5-4fff-803e-066c1a216ffa-proxy-tls\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-7d56b5c8fc-4np82\" (UID: \"e5668592-7da5-4fff-803e-066c1a216ffa\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-7d56b5c8fc-4np82"
Jul 27 22:34:20.587568 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:34:20.587533    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-7d56b5c8fc-4np82"
Jul 27 22:34:20.727378 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:34:20.727349    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-7d56b5c8fc-4np82"]
Jul 27 22:34:20.729248 ip-10-0-131-133 kubenswrapper[2559]: W0727 22:34:20.729194    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode5668592_7da5_4fff_803e_066c1a216ffa.slice/crio-463b86f56a9d4e6962770b71d423b3ca55acdb06052fca38e1416f966ee526fa WatchSource:0}: Error finding container 463b86f56a9d4e6962770b71d423b3ca55acdb06052fca38e1416f966ee526fa: Status 404 returned error can't find the container with id 463b86f56a9d4e6962770b71d423b3ca55acdb06052fca38e1416f966ee526fa
Jul 27 22:34:21.232048 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:34:21.232014    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-7d56b5c8fc-4np82" event={"ID":"e5668592-7da5-4fff-803e-066c1a216ffa","Type":"ContainerStarted","Data":"672901104bffab0d93b1317ff73ff03e036e49460de85bf5b187da215c1f4d48"}
Jul 27 22:34:21.232253 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:34:21.232086    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-7d56b5c8fc-4np82" event={"ID":"e5668592-7da5-4fff-803e-066c1a216ffa","Type":"ContainerStarted","Data":"463b86f56a9d4e6962770b71d423b3ca55acdb06052fca38e1416f966ee526fa"}
Jul 27 22:34:22.851786 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:34:22.851735    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-665d8df985-gff9j" podUID="b6251c55-a270-4a42-85fe-698989b81c7a" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.47:8643/healthz\": dial tcp 10.132.0.47:8643: connect: connection refused"
Jul 27 22:34:23.245333 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:34:23.245298    2559 generic.go:356] "Generic (PLEG): container finished" podID="b6251c55-a270-4a42-85fe-698989b81c7a" containerID="b3ea3655c64827007933c6b62b35e66b58a6afb5958b40b9cf1f6a3f0a658893" exitCode=0
Jul 27 22:34:23.245541 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:34:23.245381    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-665d8df985-gff9j" event={"ID":"b6251c55-a270-4a42-85fe-698989b81c7a","Type":"ContainerDied","Data":"b3ea3655c64827007933c6b62b35e66b58a6afb5958b40b9cf1f6a3f0a658893"}
Jul 27 22:34:23.366040 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:34:23.366015    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-665d8df985-gff9j"
Jul 27 22:34:23.488919 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:34:23.488831    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7mcv7\" (UniqueName: \"kubernetes.io/projected/b6251c55-a270-4a42-85fe-698989b81c7a-kube-api-access-7mcv7\") pod \"b6251c55-a270-4a42-85fe-698989b81c7a\" (UID: \"b6251c55-a270-4a42-85fe-698989b81c7a\") "
Jul 27 22:34:23.488919 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:34:23.488871    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b6251c55-a270-4a42-85fe-698989b81c7a-cabundle-cert\") pod \"b6251c55-a270-4a42-85fe-698989b81c7a\" (UID: \"b6251c55-a270-4a42-85fe-698989b81c7a\") "
Jul 27 22:34:23.489115 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:34:23.488931    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b6251c55-a270-4a42-85fe-698989b81c7a-kserve-provision-location\") pod \"b6251c55-a270-4a42-85fe-698989b81c7a\" (UID: \"b6251c55-a270-4a42-85fe-698989b81c7a\") "
Jul 27 22:34:23.489115 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:34:23.488970    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b6251c55-a270-4a42-85fe-698989b81c7a-proxy-tls\") pod \"b6251c55-a270-4a42-85fe-698989b81c7a\" (UID: \"b6251c55-a270-4a42-85fe-698989b81c7a\") "
Jul 27 22:34:23.489115 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:34:23.489066    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b6251c55-a270-4a42-85fe-698989b81c7a-kube-rbac-proxy-sar-config\") pod \"b6251c55-a270-4a42-85fe-698989b81c7a\" (UID: \"b6251c55-a270-4a42-85fe-698989b81c7a\") "
Jul 27 22:34:23.489335 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:34:23.489288    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b6251c55-a270-4a42-85fe-698989b81c7a-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "b6251c55-a270-4a42-85fe-698989b81c7a" (UID: "b6251c55-a270-4a42-85fe-698989b81c7a"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 22:34:23.489395 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:34:23.489334    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b6251c55-a270-4a42-85fe-698989b81c7a-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "b6251c55-a270-4a42-85fe-698989b81c7a" (UID: "b6251c55-a270-4a42-85fe-698989b81c7a"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 22:34:23.489449 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:34:23.489429    2559 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b6251c55-a270-4a42-85fe-698989b81c7a-cabundle-cert\") on node \"ip-10-0-131-133.ec2.internal\" DevicePath \"\""
Jul 27 22:34:23.489487 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:34:23.489456    2559 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b6251c55-a270-4a42-85fe-698989b81c7a-kserve-provision-location\") on node \"ip-10-0-131-133.ec2.internal\" DevicePath \"\""
Jul 27 22:34:23.489487 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:34:23.489464    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b6251c55-a270-4a42-85fe-698989b81c7a-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "b6251c55-a270-4a42-85fe-698989b81c7a" (UID: "b6251c55-a270-4a42-85fe-698989b81c7a"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 22:34:23.491061 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:34:23.491033    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b6251c55-a270-4a42-85fe-698989b81c7a-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "b6251c55-a270-4a42-85fe-698989b81c7a" (UID: "b6251c55-a270-4a42-85fe-698989b81c7a"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 22:34:23.491181 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:34:23.491087    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b6251c55-a270-4a42-85fe-698989b81c7a-kube-api-access-7mcv7" (OuterVolumeSpecName: "kube-api-access-7mcv7") pod "b6251c55-a270-4a42-85fe-698989b81c7a" (UID: "b6251c55-a270-4a42-85fe-698989b81c7a"). InnerVolumeSpecName "kube-api-access-7mcv7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 22:34:23.590387 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:34:23.590349    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7mcv7\" (UniqueName: \"kubernetes.io/projected/b6251c55-a270-4a42-85fe-698989b81c7a-kube-api-access-7mcv7\") on node \"ip-10-0-131-133.ec2.internal\" DevicePath \"\""
Jul 27 22:34:23.590387 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:34:23.590381    2559 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b6251c55-a270-4a42-85fe-698989b81c7a-proxy-tls\") on node \"ip-10-0-131-133.ec2.internal\" DevicePath \"\""
Jul 27 22:34:23.590387 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:34:23.590391    2559 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b6251c55-a270-4a42-85fe-698989b81c7a-kube-rbac-proxy-sar-config\") on node \"ip-10-0-131-133.ec2.internal\" DevicePath \"\""
Jul 27 22:34:24.252628 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:34:24.252534    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-665d8df985-gff9j"
Jul 27 22:34:24.252628 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:34:24.252542    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-665d8df985-gff9j" event={"ID":"b6251c55-a270-4a42-85fe-698989b81c7a","Type":"ContainerDied","Data":"4d3faf33e86199fd9a9772a9fb2e98ff58245f320beabe492a09e659d3fc6346"}
Jul 27 22:34:24.252628 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:34:24.252626    2559 scope.go:117] "RemoveContainer" containerID="5494cf1127f4589c6e8768631748eb155a99cbf468cd3ead83ea622e0845fc69"
Jul 27 22:34:24.261743 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:34:24.261722    2559 scope.go:117] "RemoveContainer" containerID="b3ea3655c64827007933c6b62b35e66b58a6afb5958b40b9cf1f6a3f0a658893"
Jul 27 22:34:24.269874 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:34:24.269847    2559 scope.go:117] "RemoveContainer" containerID="16be5bbf977cfa32eb2240a2733f85aeb8037113a4d471ef7bc632132215708b"
Jul 27 22:34:24.271075 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:34:24.271053    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-665d8df985-gff9j"]
Jul 27 22:34:24.275380 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:34:24.275355    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-665d8df985-gff9j"]
Jul 27 22:34:25.259517 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:34:25.259484    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-global-fail-predictor-7d56b5c8fc-4np82_e5668592-7da5-4fff-803e-066c1a216ffa/storage-initializer/0.log"
Jul 27 22:34:25.260028 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:34:25.259523    2559 generic.go:356] "Generic (PLEG): container finished" podID="e5668592-7da5-4fff-803e-066c1a216ffa" containerID="672901104bffab0d93b1317ff73ff03e036e49460de85bf5b187da215c1f4d48" exitCode=1
Jul 27 22:34:25.260028 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:34:25.259603    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-7d56b5c8fc-4np82" event={"ID":"e5668592-7da5-4fff-803e-066c1a216ffa","Type":"ContainerDied","Data":"672901104bffab0d93b1317ff73ff03e036e49460de85bf5b187da215c1f4d48"}
Jul 27 22:34:25.982182 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:34:25.982150    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b6251c55-a270-4a42-85fe-698989b81c7a" path="/var/lib/kubelet/pods/b6251c55-a270-4a42-85fe-698989b81c7a/volumes"
Jul 27 22:34:26.267605 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:34:26.267533    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-global-fail-predictor-7d56b5c8fc-4np82_e5668592-7da5-4fff-803e-066c1a216ffa/storage-initializer/0.log"
Jul 27 22:34:26.267605 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:34:26.267577    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-7d56b5c8fc-4np82" event={"ID":"e5668592-7da5-4fff-803e-066c1a216ffa","Type":"ContainerStarted","Data":"481435f827925f7594ac4b8fbe23ac2725c25f91cbbcc79a52610a6cd8e83c91"}
Jul 27 22:34:29.665578 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:34:29.665478    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-7d56b5c8fc-4np82"]
Jul 27 22:34:29.665952 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:34:29.665722    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-7d56b5c8fc-4np82" podUID="e5668592-7da5-4fff-803e-066c1a216ffa" containerName="storage-initializer" containerID="cri-o://481435f827925f7594ac4b8fbe23ac2725c25f91cbbcc79a52610a6cd8e83c91" gracePeriod=30
Jul 27 22:34:29.808684 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:34:29.808659    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-global-fail-predictor-7d56b5c8fc-4np82_e5668592-7da5-4fff-803e-066c1a216ffa/storage-initializer/1.log"
Jul 27 22:34:29.809037 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:34:29.809024    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-global-fail-predictor-7d56b5c8fc-4np82_e5668592-7da5-4fff-803e-066c1a216ffa/storage-initializer/0.log"
Jul 27 22:34:29.811518 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:34:29.811502    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-7d56b5c8fc-4np82"
Jul 27 22:34:29.843263 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:34:29.843231    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e5668592-7da5-4fff-803e-066c1a216ffa-kserve-provision-location\") pod \"e5668592-7da5-4fff-803e-066c1a216ffa\" (UID: \"e5668592-7da5-4fff-803e-066c1a216ffa\") "
Jul 27 22:34:29.843427 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:34:29.843338    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e5668592-7da5-4fff-803e-066c1a216ffa-proxy-tls\") pod \"e5668592-7da5-4fff-803e-066c1a216ffa\" (UID: \"e5668592-7da5-4fff-803e-066c1a216ffa\") "
Jul 27 22:34:29.843427 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:34:29.843360    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dx8dz\" (UniqueName: \"kubernetes.io/projected/e5668592-7da5-4fff-803e-066c1a216ffa-kube-api-access-dx8dz\") pod \"e5668592-7da5-4fff-803e-066c1a216ffa\" (UID: \"e5668592-7da5-4fff-803e-066c1a216ffa\") "
Jul 27 22:34:29.843427 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:34:29.843383    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e5668592-7da5-4fff-803e-066c1a216ffa-kube-rbac-proxy-sar-config\") pod \"e5668592-7da5-4fff-803e-066c1a216ffa\" (UID: \"e5668592-7da5-4fff-803e-066c1a216ffa\") "
Jul 27 22:34:29.843525 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:34:29.843481    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e5668592-7da5-4fff-803e-066c1a216ffa-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "e5668592-7da5-4fff-803e-066c1a216ffa" (UID: "e5668592-7da5-4fff-803e-066c1a216ffa"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 22:34:29.843690 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:34:29.843676    2559 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e5668592-7da5-4fff-803e-066c1a216ffa-kserve-provision-location\") on node \"ip-10-0-131-133.ec2.internal\" DevicePath \"\""
Jul 27 22:34:29.843788 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:34:29.843771    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e5668592-7da5-4fff-803e-066c1a216ffa-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "e5668592-7da5-4fff-803e-066c1a216ffa" (UID: "e5668592-7da5-4fff-803e-066c1a216ffa"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 22:34:29.845477 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:34:29.845452    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e5668592-7da5-4fff-803e-066c1a216ffa-kube-api-access-dx8dz" (OuterVolumeSpecName: "kube-api-access-dx8dz") pod "e5668592-7da5-4fff-803e-066c1a216ffa" (UID: "e5668592-7da5-4fff-803e-066c1a216ffa"). InnerVolumeSpecName "kube-api-access-dx8dz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 22:34:29.845691 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:34:29.845670    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e5668592-7da5-4fff-803e-066c1a216ffa-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "e5668592-7da5-4fff-803e-066c1a216ffa" (UID: "e5668592-7da5-4fff-803e-066c1a216ffa"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 22:34:29.944844 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:34:29.944755    2559 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e5668592-7da5-4fff-803e-066c1a216ffa-proxy-tls\") on node \"ip-10-0-131-133.ec2.internal\" DevicePath \"\""
Jul 27 22:34:29.944844 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:34:29.944782    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dx8dz\" (UniqueName: \"kubernetes.io/projected/e5668592-7da5-4fff-803e-066c1a216ffa-kube-api-access-dx8dz\") on node \"ip-10-0-131-133.ec2.internal\" DevicePath \"\""
Jul 27 22:34:29.944844 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:34:29.944792    2559 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e5668592-7da5-4fff-803e-066c1a216ffa-kube-rbac-proxy-sar-config\") on node \"ip-10-0-131-133.ec2.internal\" DevicePath \"\""
Jul 27 22:34:30.288066 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:34:30.287985    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-global-fail-predictor-7d56b5c8fc-4np82_e5668592-7da5-4fff-803e-066c1a216ffa/storage-initializer/1.log"
Jul 27 22:34:30.288397 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:34:30.288381    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-global-fail-predictor-7d56b5c8fc-4np82_e5668592-7da5-4fff-803e-066c1a216ffa/storage-initializer/0.log"
Jul 27 22:34:30.288464 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:34:30.288428    2559 generic.go:356] "Generic (PLEG): container finished" podID="e5668592-7da5-4fff-803e-066c1a216ffa" containerID="481435f827925f7594ac4b8fbe23ac2725c25f91cbbcc79a52610a6cd8e83c91" exitCode=1
Jul 27 22:34:30.288502 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:34:30.288456    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-7d56b5c8fc-4np82" event={"ID":"e5668592-7da5-4fff-803e-066c1a216ffa","Type":"ContainerDied","Data":"481435f827925f7594ac4b8fbe23ac2725c25f91cbbcc79a52610a6cd8e83c91"}
Jul 27 22:34:30.288502 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:34:30.288489    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-7d56b5c8fc-4np82"
Jul 27 22:34:30.288566 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:34:30.288517    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-7d56b5c8fc-4np82" event={"ID":"e5668592-7da5-4fff-803e-066c1a216ffa","Type":"ContainerDied","Data":"463b86f56a9d4e6962770b71d423b3ca55acdb06052fca38e1416f966ee526fa"}
Jul 27 22:34:30.288566 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:34:30.288546    2559 scope.go:117] "RemoveContainer" containerID="481435f827925f7594ac4b8fbe23ac2725c25f91cbbcc79a52610a6cd8e83c91"
Jul 27 22:34:30.299901 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:34:30.299884    2559 scope.go:117] "RemoveContainer" containerID="672901104bffab0d93b1317ff73ff03e036e49460de85bf5b187da215c1f4d48"
Jul 27 22:34:30.307223 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:34:30.307194    2559 scope.go:117] "RemoveContainer" containerID="481435f827925f7594ac4b8fbe23ac2725c25f91cbbcc79a52610a6cd8e83c91"
Jul 27 22:34:30.307481 ip-10-0-131-133 kubenswrapper[2559]: E0727 22:34:30.307459    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"481435f827925f7594ac4b8fbe23ac2725c25f91cbbcc79a52610a6cd8e83c91\": container with ID starting with 481435f827925f7594ac4b8fbe23ac2725c25f91cbbcc79a52610a6cd8e83c91 not found: ID does not exist" containerID="481435f827925f7594ac4b8fbe23ac2725c25f91cbbcc79a52610a6cd8e83c91"
Jul 27 22:34:30.307556 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:34:30.307493    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"481435f827925f7594ac4b8fbe23ac2725c25f91cbbcc79a52610a6cd8e83c91"} err="failed to get container status \"481435f827925f7594ac4b8fbe23ac2725c25f91cbbcc79a52610a6cd8e83c91\": rpc error: code = NotFound desc = could not find container \"481435f827925f7594ac4b8fbe23ac2725c25f91cbbcc79a52610a6cd8e83c91\": container with ID starting with 481435f827925f7594ac4b8fbe23ac2725c25f91cbbcc79a52610a6cd8e83c91 not found: ID does not exist"
Jul 27 22:34:30.307556 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:34:30.307518    2559 scope.go:117] "RemoveContainer" containerID="672901104bffab0d93b1317ff73ff03e036e49460de85bf5b187da215c1f4d48"
Jul 27 22:34:30.307779 ip-10-0-131-133 kubenswrapper[2559]: E0727 22:34:30.307760    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"672901104bffab0d93b1317ff73ff03e036e49460de85bf5b187da215c1f4d48\": container with ID starting with 672901104bffab0d93b1317ff73ff03e036e49460de85bf5b187da215c1f4d48 not found: ID does not exist" containerID="672901104bffab0d93b1317ff73ff03e036e49460de85bf5b187da215c1f4d48"
Jul 27 22:34:30.307819 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:34:30.307785    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"672901104bffab0d93b1317ff73ff03e036e49460de85bf5b187da215c1f4d48"} err="failed to get container status \"672901104bffab0d93b1317ff73ff03e036e49460de85bf5b187da215c1f4d48\": rpc error: code = NotFound desc = could not find container \"672901104bffab0d93b1317ff73ff03e036e49460de85bf5b187da215c1f4d48\": container with ID starting with 672901104bffab0d93b1317ff73ff03e036e49460de85bf5b187da215c1f4d48 not found: ID does not exist"
Jul 27 22:34:30.318942 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:34:30.318915    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-7d56b5c8fc-4np82"]
Jul 27 22:34:30.322554 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:34:30.322532    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-7d56b5c8fc-4np82"]
Jul 27 22:34:31.983128 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:34:31.983094    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e5668592-7da5-4fff-803e-066c1a216ffa" path="/var/lib/kubelet/pods/e5668592-7da5-4fff-803e-066c1a216ffa/volumes"
Jul 27 22:34:32.396413 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:34:32.396378    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-bjvbc_cba53c9a-64c0-4afa-8731-35521e196622/console-operator/1.log"
Jul 27 22:34:32.403074 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:34:32.403052    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-bjvbc_cba53c9a-64c0-4afa-8731-35521e196622/console-operator/1.log"
Jul 27 22:35:51.877900 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:35:51.877865    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-749df86c69-gp4vx"]
Jul 27 22:35:51.878470 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:35:51.878396    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e5668592-7da5-4fff-803e-066c1a216ffa" containerName="storage-initializer"
Jul 27 22:35:51.878470 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:35:51.878412    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="e5668592-7da5-4fff-803e-066c1a216ffa" containerName="storage-initializer"
Jul 27 22:35:51.878470 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:35:51.878440    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b6251c55-a270-4a42-85fe-698989b81c7a" containerName="storage-initializer"
Jul 27 22:35:51.878470 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:35:51.878446    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="b6251c55-a270-4a42-85fe-698989b81c7a" containerName="storage-initializer"
Jul 27 22:35:51.878470 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:35:51.878455    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e5668592-7da5-4fff-803e-066c1a216ffa" containerName="storage-initializer"
Jul 27 22:35:51.878470 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:35:51.878461    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="e5668592-7da5-4fff-803e-066c1a216ffa" containerName="storage-initializer"
Jul 27 22:35:51.878470 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:35:51.878468    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b6251c55-a270-4a42-85fe-698989b81c7a" containerName="kube-rbac-proxy"
Jul 27 22:35:51.878470 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:35:51.878474    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="b6251c55-a270-4a42-85fe-698989b81c7a" containerName="kube-rbac-proxy"
Jul 27 22:35:51.878740 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:35:51.878487    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b6251c55-a270-4a42-85fe-698989b81c7a" containerName="kserve-container"
Jul 27 22:35:51.878740 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:35:51.878492    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="b6251c55-a270-4a42-85fe-698989b81c7a" containerName="kserve-container"
Jul 27 22:35:51.878740 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:35:51.878543    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="e5668592-7da5-4fff-803e-066c1a216ffa" containerName="storage-initializer"
Jul 27 22:35:51.878740 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:35:51.878552    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="e5668592-7da5-4fff-803e-066c1a216ffa" containerName="storage-initializer"
Jul 27 22:35:51.878740 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:35:51.878559    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="b6251c55-a270-4a42-85fe-698989b81c7a" containerName="kube-rbac-proxy"
Jul 27 22:35:51.878740 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:35:51.878565    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="b6251c55-a270-4a42-85fe-698989b81c7a" containerName="kserve-container"
Jul 27 22:35:51.881819 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:35:51.881804    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-749df86c69-gp4vx"
Jul 27 22:35:51.884227 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:35:51.884189    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 27 22:35:51.884227 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:35:51.884206    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 27 22:35:51.884875 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:35:51.884849    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-v7c66\""
Jul 27 22:35:51.884971 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:35:51.884873    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-tls-custom-fail-kube-rbac-proxy-sar-config\""
Jul 27 22:35:51.884971 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:35:51.884888    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"storage-config\""
Jul 27 22:35:51.884971 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:35:51.884850    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-tls-custom-fail-predictor-serving-cert\""
Jul 27 22:35:51.890865 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:35:51.890845    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-749df86c69-gp4vx"]
Jul 27 22:35:51.962247 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:35:51.962176    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hghck\" (UniqueName: \"kubernetes.io/projected/691de555-00ec-4f47-b944-c921261db172-kube-api-access-hghck\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-749df86c69-gp4vx\" (UID: \"691de555-00ec-4f47-b944-c921261db172\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-749df86c69-gp4vx"
Jul 27 22:35:51.962436 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:35:51.962272    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/691de555-00ec-4f47-b944-c921261db172-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-749df86c69-gp4vx\" (UID: \"691de555-00ec-4f47-b944-c921261db172\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-749df86c69-gp4vx"
Jul 27 22:35:51.962436 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:35:51.962295    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/691de555-00ec-4f47-b944-c921261db172-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-749df86c69-gp4vx\" (UID: \"691de555-00ec-4f47-b944-c921261db172\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-749df86c69-gp4vx"
Jul 27 22:35:51.962436 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:35:51.962322    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/691de555-00ec-4f47-b944-c921261db172-proxy-tls\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-749df86c69-gp4vx\" (UID: \"691de555-00ec-4f47-b944-c921261db172\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-749df86c69-gp4vx"
Jul 27 22:35:52.063066 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:35:52.063027    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/691de555-00ec-4f47-b944-c921261db172-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-749df86c69-gp4vx\" (UID: \"691de555-00ec-4f47-b944-c921261db172\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-749df86c69-gp4vx"
Jul 27 22:35:52.063320 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:35:52.063075    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/691de555-00ec-4f47-b944-c921261db172-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-749df86c69-gp4vx\" (UID: \"691de555-00ec-4f47-b944-c921261db172\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-749df86c69-gp4vx"
Jul 27 22:35:52.063320 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:35:52.063107    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/691de555-00ec-4f47-b944-c921261db172-proxy-tls\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-749df86c69-gp4vx\" (UID: \"691de555-00ec-4f47-b944-c921261db172\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-749df86c69-gp4vx"
Jul 27 22:35:52.063320 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:35:52.063186    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hghck\" (UniqueName: \"kubernetes.io/projected/691de555-00ec-4f47-b944-c921261db172-kube-api-access-hghck\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-749df86c69-gp4vx\" (UID: \"691de555-00ec-4f47-b944-c921261db172\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-749df86c69-gp4vx"
Jul 27 22:35:52.063614 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:35:52.063588    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/691de555-00ec-4f47-b944-c921261db172-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-749df86c69-gp4vx\" (UID: \"691de555-00ec-4f47-b944-c921261db172\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-749df86c69-gp4vx"
Jul 27 22:35:52.063831 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:35:52.063808    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/691de555-00ec-4f47-b944-c921261db172-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-749df86c69-gp4vx\" (UID: \"691de555-00ec-4f47-b944-c921261db172\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-749df86c69-gp4vx"
Jul 27 22:35:52.065723 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:35:52.065700    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/691de555-00ec-4f47-b944-c921261db172-proxy-tls\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-749df86c69-gp4vx\" (UID: \"691de555-00ec-4f47-b944-c921261db172\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-749df86c69-gp4vx"
Jul 27 22:35:52.071240 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:35:52.071203    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hghck\" (UniqueName: \"kubernetes.io/projected/691de555-00ec-4f47-b944-c921261db172-kube-api-access-hghck\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-749df86c69-gp4vx\" (UID: \"691de555-00ec-4f47-b944-c921261db172\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-749df86c69-gp4vx"
Jul 27 22:35:52.193224 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:35:52.193081    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-749df86c69-gp4vx"
Jul 27 22:35:52.331917 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:35:52.331888    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-749df86c69-gp4vx"]
Jul 27 22:35:52.333911 ip-10-0-131-133 kubenswrapper[2559]: W0727 22:35:52.333879    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod691de555_00ec_4f47_b944_c921261db172.slice/crio-8cd3431b2c5b454dd5e9870a4f714f6e95e39906c4e53d57c8a03a907c83177a WatchSource:0}: Error finding container 8cd3431b2c5b454dd5e9870a4f714f6e95e39906c4e53d57c8a03a907c83177a: Status 404 returned error can't find the container with id 8cd3431b2c5b454dd5e9870a4f714f6e95e39906c4e53d57c8a03a907c83177a
Jul 27 22:35:52.662451 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:35:52.662409    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-749df86c69-gp4vx" event={"ID":"691de555-00ec-4f47-b944-c921261db172","Type":"ContainerStarted","Data":"e77972e5fb14ab7af78365ce9ff44dc99748c1c8c96900213e8ea4e0bb03c632"}
Jul 27 22:35:52.662632 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:35:52.662478    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-749df86c69-gp4vx" event={"ID":"691de555-00ec-4f47-b944-c921261db172","Type":"ContainerStarted","Data":"8cd3431b2c5b454dd5e9870a4f714f6e95e39906c4e53d57c8a03a907c83177a"}
Jul 27 22:35:58.691644 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:35:58.691613    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-custom-fail-predictor-749df86c69-gp4vx_691de555-00ec-4f47-b944-c921261db172/storage-initializer/0.log"
Jul 27 22:35:58.692081 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:35:58.691654    2559 generic.go:356] "Generic (PLEG): container finished" podID="691de555-00ec-4f47-b944-c921261db172" containerID="e77972e5fb14ab7af78365ce9ff44dc99748c1c8c96900213e8ea4e0bb03c632" exitCode=1
Jul 27 22:35:58.692081 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:35:58.691733    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-749df86c69-gp4vx" event={"ID":"691de555-00ec-4f47-b944-c921261db172","Type":"ContainerDied","Data":"e77972e5fb14ab7af78365ce9ff44dc99748c1c8c96900213e8ea4e0bb03c632"}
Jul 27 22:35:59.698086 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:35:59.698060    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-custom-fail-predictor-749df86c69-gp4vx_691de555-00ec-4f47-b944-c921261db172/storage-initializer/0.log"
Jul 27 22:35:59.698498 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:35:59.698183    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-749df86c69-gp4vx" event={"ID":"691de555-00ec-4f47-b944-c921261db172","Type":"ContainerStarted","Data":"aa916d4b291fd95c261373a6e84066233f518c6548564a1f09c5fe969b082277"}
Jul 27 22:36:01.893934 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:36:01.893900    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-749df86c69-gp4vx"]
Jul 27 22:36:01.894365 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:36:01.894118    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-749df86c69-gp4vx" podUID="691de555-00ec-4f47-b944-c921261db172" containerName="storage-initializer" containerID="cri-o://aa916d4b291fd95c261373a6e84066233f518c6548564a1f09c5fe969b082277" gracePeriod=30
Jul 27 22:36:02.989943 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:36:02.989899    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-6fccdccf8b-4t8h2"]
Jul 27 22:36:02.993772 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:36:02.993746    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-6fccdccf8b-4t8h2"
Jul 27 22:36:02.995944 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:36:02.995918    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-tls-serving-pass-predictor-serving-cert\""
Jul 27 22:36:02.996097 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:36:02.995918    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-tls-serving-pass-kube-rbac-proxy-sar-config\""
Jul 27 22:36:02.996097 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:36:02.995967    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 27 22:36:03.004564 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:36:03.004536    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-6fccdccf8b-4t8h2"]
Jul 27 22:36:03.164193 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:36:03.164148    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f4467b1c-5419-40f1-99ac-63fe63a8198c-cabundle-cert\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-6fccdccf8b-4t8h2\" (UID: \"f4467b1c-5419-40f1-99ac-63fe63a8198c\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-6fccdccf8b-4t8h2"
Jul 27 22:36:03.164423 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:36:03.164202    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f4467b1c-5419-40f1-99ac-63fe63a8198c-proxy-tls\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-6fccdccf8b-4t8h2\" (UID: \"f4467b1c-5419-40f1-99ac-63fe63a8198c\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-6fccdccf8b-4t8h2"
Jul 27 22:36:03.164423 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:36:03.164295    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5sg8q\" (UniqueName: \"kubernetes.io/projected/f4467b1c-5419-40f1-99ac-63fe63a8198c-kube-api-access-5sg8q\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-6fccdccf8b-4t8h2\" (UID: \"f4467b1c-5419-40f1-99ac-63fe63a8198c\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-6fccdccf8b-4t8h2"
Jul 27 22:36:03.164423 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:36:03.164356    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f4467b1c-5419-40f1-99ac-63fe63a8198c-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-6fccdccf8b-4t8h2\" (UID: \"f4467b1c-5419-40f1-99ac-63fe63a8198c\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-6fccdccf8b-4t8h2"
Jul 27 22:36:03.164547 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:36:03.164418    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f4467b1c-5419-40f1-99ac-63fe63a8198c-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-6fccdccf8b-4t8h2\" (UID: \"f4467b1c-5419-40f1-99ac-63fe63a8198c\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-6fccdccf8b-4t8h2"
Jul 27 22:36:03.265281 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:36:03.265159    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f4467b1c-5419-40f1-99ac-63fe63a8198c-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-6fccdccf8b-4t8h2\" (UID: \"f4467b1c-5419-40f1-99ac-63fe63a8198c\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-6fccdccf8b-4t8h2"
Jul 27 22:36:03.265281 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:36:03.265260    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f4467b1c-5419-40f1-99ac-63fe63a8198c-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-6fccdccf8b-4t8h2\" (UID: \"f4467b1c-5419-40f1-99ac-63fe63a8198c\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-6fccdccf8b-4t8h2"
Jul 27 22:36:03.265528 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:36:03.265318    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f4467b1c-5419-40f1-99ac-63fe63a8198c-cabundle-cert\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-6fccdccf8b-4t8h2\" (UID: \"f4467b1c-5419-40f1-99ac-63fe63a8198c\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-6fccdccf8b-4t8h2"
Jul 27 22:36:03.265528 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:36:03.265362    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f4467b1c-5419-40f1-99ac-63fe63a8198c-proxy-tls\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-6fccdccf8b-4t8h2\" (UID: \"f4467b1c-5419-40f1-99ac-63fe63a8198c\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-6fccdccf8b-4t8h2"
Jul 27 22:36:03.265528 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:36:03.265398    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5sg8q\" (UniqueName: \"kubernetes.io/projected/f4467b1c-5419-40f1-99ac-63fe63a8198c-kube-api-access-5sg8q\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-6fccdccf8b-4t8h2\" (UID: \"f4467b1c-5419-40f1-99ac-63fe63a8198c\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-6fccdccf8b-4t8h2"
Jul 27 22:36:03.265838 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:36:03.265812    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f4467b1c-5419-40f1-99ac-63fe63a8198c-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-6fccdccf8b-4t8h2\" (UID: \"f4467b1c-5419-40f1-99ac-63fe63a8198c\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-6fccdccf8b-4t8h2"
Jul 27 22:36:03.266038 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:36:03.266016    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f4467b1c-5419-40f1-99ac-63fe63a8198c-cabundle-cert\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-6fccdccf8b-4t8h2\" (UID: \"f4467b1c-5419-40f1-99ac-63fe63a8198c\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-6fccdccf8b-4t8h2"
Jul 27 22:36:03.266089 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:36:03.266016    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f4467b1c-5419-40f1-99ac-63fe63a8198c-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-6fccdccf8b-4t8h2\" (UID: \"f4467b1c-5419-40f1-99ac-63fe63a8198c\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-6fccdccf8b-4t8h2"
Jul 27 22:36:03.267942 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:36:03.267921    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f4467b1c-5419-40f1-99ac-63fe63a8198c-proxy-tls\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-6fccdccf8b-4t8h2\" (UID: \"f4467b1c-5419-40f1-99ac-63fe63a8198c\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-6fccdccf8b-4t8h2"
Jul 27 22:36:03.273240 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:36:03.273200    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5sg8q\" (UniqueName: \"kubernetes.io/projected/f4467b1c-5419-40f1-99ac-63fe63a8198c-kube-api-access-5sg8q\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-6fccdccf8b-4t8h2\" (UID: \"f4467b1c-5419-40f1-99ac-63fe63a8198c\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-6fccdccf8b-4t8h2"
Jul 27 22:36:03.305303 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:36:03.305256    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-6fccdccf8b-4t8h2"
Jul 27 22:36:03.454703 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:36:03.454654    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-6fccdccf8b-4t8h2"]
Jul 27 22:36:03.457063 ip-10-0-131-133 kubenswrapper[2559]: W0727 22:36:03.457035    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf4467b1c_5419_40f1_99ac_63fe63a8198c.slice/crio-893f3fafcea118583de9ab10bab8633930026c7c6d141ee51e1b92934fd366fa WatchSource:0}: Error finding container 893f3fafcea118583de9ab10bab8633930026c7c6d141ee51e1b92934fd366fa: Status 404 returned error can't find the container with id 893f3fafcea118583de9ab10bab8633930026c7c6d141ee51e1b92934fd366fa
Jul 27 22:36:03.539895 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:36:03.539871    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-custom-fail-predictor-749df86c69-gp4vx_691de555-00ec-4f47-b944-c921261db172/storage-initializer/1.log"
Jul 27 22:36:03.540296 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:36:03.540280    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-custom-fail-predictor-749df86c69-gp4vx_691de555-00ec-4f47-b944-c921261db172/storage-initializer/0.log"
Jul 27 22:36:03.542553 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:36:03.542536    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-749df86c69-gp4vx"
Jul 27 22:36:03.669856 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:36:03.669803    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hghck\" (UniqueName: \"kubernetes.io/projected/691de555-00ec-4f47-b944-c921261db172-kube-api-access-hghck\") pod \"691de555-00ec-4f47-b944-c921261db172\" (UID: \"691de555-00ec-4f47-b944-c921261db172\") "
Jul 27 22:36:03.670065 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:36:03.669873    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/691de555-00ec-4f47-b944-c921261db172-kube-rbac-proxy-sar-config\") pod \"691de555-00ec-4f47-b944-c921261db172\" (UID: \"691de555-00ec-4f47-b944-c921261db172\") "
Jul 27 22:36:03.670065 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:36:03.669911    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/691de555-00ec-4f47-b944-c921261db172-proxy-tls\") pod \"691de555-00ec-4f47-b944-c921261db172\" (UID: \"691de555-00ec-4f47-b944-c921261db172\") "
Jul 27 22:36:03.670065 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:36:03.669942    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/691de555-00ec-4f47-b944-c921261db172-kserve-provision-location\") pod \"691de555-00ec-4f47-b944-c921261db172\" (UID: \"691de555-00ec-4f47-b944-c921261db172\") "
Jul 27 22:36:03.670307 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:36:03.670254    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/691de555-00ec-4f47-b944-c921261db172-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "691de555-00ec-4f47-b944-c921261db172" (UID: "691de555-00ec-4f47-b944-c921261db172"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 22:36:03.670307 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:36:03.670279    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/691de555-00ec-4f47-b944-c921261db172-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "691de555-00ec-4f47-b944-c921261db172" (UID: "691de555-00ec-4f47-b944-c921261db172"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 22:36:03.671979 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:36:03.671956    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/691de555-00ec-4f47-b944-c921261db172-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "691de555-00ec-4f47-b944-c921261db172" (UID: "691de555-00ec-4f47-b944-c921261db172"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 22:36:03.672113 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:36:03.672009    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/691de555-00ec-4f47-b944-c921261db172-kube-api-access-hghck" (OuterVolumeSpecName: "kube-api-access-hghck") pod "691de555-00ec-4f47-b944-c921261db172" (UID: "691de555-00ec-4f47-b944-c921261db172"). InnerVolumeSpecName "kube-api-access-hghck". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 22:36:03.717885 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:36:03.717828    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-6fccdccf8b-4t8h2" event={"ID":"f4467b1c-5419-40f1-99ac-63fe63a8198c","Type":"ContainerStarted","Data":"533fa5f5d1e51ce713ac756e3dccb93332fbde7c25034dc5d7926caffe4634d4"}
Jul 27 22:36:03.718058 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:36:03.717912    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-6fccdccf8b-4t8h2" event={"ID":"f4467b1c-5419-40f1-99ac-63fe63a8198c","Type":"ContainerStarted","Data":"893f3fafcea118583de9ab10bab8633930026c7c6d141ee51e1b92934fd366fa"}
Jul 27 22:36:03.719387 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:36:03.719365    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-custom-fail-predictor-749df86c69-gp4vx_691de555-00ec-4f47-b944-c921261db172/storage-initializer/1.log"
Jul 27 22:36:03.719803 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:36:03.719788    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-custom-fail-predictor-749df86c69-gp4vx_691de555-00ec-4f47-b944-c921261db172/storage-initializer/0.log"
Jul 27 22:36:03.719859 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:36:03.719822    2559 generic.go:356] "Generic (PLEG): container finished" podID="691de555-00ec-4f47-b944-c921261db172" containerID="aa916d4b291fd95c261373a6e84066233f518c6548564a1f09c5fe969b082277" exitCode=1
Jul 27 22:36:03.719905 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:36:03.719868    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-749df86c69-gp4vx" event={"ID":"691de555-00ec-4f47-b944-c921261db172","Type":"ContainerDied","Data":"aa916d4b291fd95c261373a6e84066233f518c6548564a1f09c5fe969b082277"}
Jul 27 22:36:03.719905 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:36:03.719877    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-749df86c69-gp4vx"
Jul 27 22:36:03.719981 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:36:03.719911    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-749df86c69-gp4vx" event={"ID":"691de555-00ec-4f47-b944-c921261db172","Type":"ContainerDied","Data":"8cd3431b2c5b454dd5e9870a4f714f6e95e39906c4e53d57c8a03a907c83177a"}
Jul 27 22:36:03.719981 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:36:03.719936    2559 scope.go:117] "RemoveContainer" containerID="aa916d4b291fd95c261373a6e84066233f518c6548564a1f09c5fe969b082277"
Jul 27 22:36:03.729136 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:36:03.729116    2559 scope.go:117] "RemoveContainer" containerID="e77972e5fb14ab7af78365ce9ff44dc99748c1c8c96900213e8ea4e0bb03c632"
Jul 27 22:36:03.742845 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:36:03.742823    2559 scope.go:117] "RemoveContainer" containerID="aa916d4b291fd95c261373a6e84066233f518c6548564a1f09c5fe969b082277"
Jul 27 22:36:03.743164 ip-10-0-131-133 kubenswrapper[2559]: E0727 22:36:03.743134    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aa916d4b291fd95c261373a6e84066233f518c6548564a1f09c5fe969b082277\": container with ID starting with aa916d4b291fd95c261373a6e84066233f518c6548564a1f09c5fe969b082277 not found: ID does not exist" containerID="aa916d4b291fd95c261373a6e84066233f518c6548564a1f09c5fe969b082277"
Jul 27 22:36:03.743261 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:36:03.743167    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aa916d4b291fd95c261373a6e84066233f518c6548564a1f09c5fe969b082277"} err="failed to get container status \"aa916d4b291fd95c261373a6e84066233f518c6548564a1f09c5fe969b082277\": rpc error: code = NotFound desc = could not find container \"aa916d4b291fd95c261373a6e84066233f518c6548564a1f09c5fe969b082277\": container with ID starting with aa916d4b291fd95c261373a6e84066233f518c6548564a1f09c5fe969b082277 not found: ID does not exist"
Jul 27 22:36:03.743261 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:36:03.743191    2559 scope.go:117] "RemoveContainer" containerID="e77972e5fb14ab7af78365ce9ff44dc99748c1c8c96900213e8ea4e0bb03c632"
Jul 27 22:36:03.743495 ip-10-0-131-133 kubenswrapper[2559]: E0727 22:36:03.743479    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e77972e5fb14ab7af78365ce9ff44dc99748c1c8c96900213e8ea4e0bb03c632\": container with ID starting with e77972e5fb14ab7af78365ce9ff44dc99748c1c8c96900213e8ea4e0bb03c632 not found: ID does not exist" containerID="e77972e5fb14ab7af78365ce9ff44dc99748c1c8c96900213e8ea4e0bb03c632"
Jul 27 22:36:03.743561 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:36:03.743501    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e77972e5fb14ab7af78365ce9ff44dc99748c1c8c96900213e8ea4e0bb03c632"} err="failed to get container status \"e77972e5fb14ab7af78365ce9ff44dc99748c1c8c96900213e8ea4e0bb03c632\": rpc error: code = NotFound desc = could not find container \"e77972e5fb14ab7af78365ce9ff44dc99748c1c8c96900213e8ea4e0bb03c632\": container with ID starting with e77972e5fb14ab7af78365ce9ff44dc99748c1c8c96900213e8ea4e0bb03c632 not found: ID does not exist"
Jul 27 22:36:03.766025 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:36:03.765983    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-749df86c69-gp4vx"]
Jul 27 22:36:03.768429 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:36:03.768404    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-749df86c69-gp4vx"]
Jul 27 22:36:03.770999 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:36:03.770975    2559 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/691de555-00ec-4f47-b944-c921261db172-kube-rbac-proxy-sar-config\") on node \"ip-10-0-131-133.ec2.internal\" DevicePath \"\""
Jul 27 22:36:03.771067 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:36:03.771006    2559 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/691de555-00ec-4f47-b944-c921261db172-proxy-tls\") on node \"ip-10-0-131-133.ec2.internal\" DevicePath \"\""
Jul 27 22:36:03.771067 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:36:03.771022    2559 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/691de555-00ec-4f47-b944-c921261db172-kserve-provision-location\") on node \"ip-10-0-131-133.ec2.internal\" DevicePath \"\""
Jul 27 22:36:03.771067 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:36:03.771037    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hghck\" (UniqueName: \"kubernetes.io/projected/691de555-00ec-4f47-b944-c921261db172-kube-api-access-hghck\") on node \"ip-10-0-131-133.ec2.internal\" DevicePath \"\""
Jul 27 22:36:03.984363 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:36:03.984323    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="691de555-00ec-4f47-b944-c921261db172" path="/var/lib/kubelet/pods/691de555-00ec-4f47-b944-c921261db172/volumes"
Jul 27 22:36:04.726659 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:36:04.726625    2559 generic.go:356] "Generic (PLEG): container finished" podID="f4467b1c-5419-40f1-99ac-63fe63a8198c" containerID="533fa5f5d1e51ce713ac756e3dccb93332fbde7c25034dc5d7926caffe4634d4" exitCode=0
Jul 27 22:36:04.727051 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:36:04.726678    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-6fccdccf8b-4t8h2" event={"ID":"f4467b1c-5419-40f1-99ac-63fe63a8198c","Type":"ContainerDied","Data":"533fa5f5d1e51ce713ac756e3dccb93332fbde7c25034dc5d7926caffe4634d4"}
Jul 27 22:36:05.733571 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:36:05.733533    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-6fccdccf8b-4t8h2" event={"ID":"f4467b1c-5419-40f1-99ac-63fe63a8198c","Type":"ContainerStarted","Data":"0d864dcafbe2b4e68e4dfc278f338629ad0ffbbbc6e8f5b42b7be986333b8ba0"}
Jul 27 22:36:05.733972 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:36:05.733614    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-6fccdccf8b-4t8h2" event={"ID":"f4467b1c-5419-40f1-99ac-63fe63a8198c","Type":"ContainerStarted","Data":"2224fef2002566862a025e3b61ef4fbd830d39e5266837088cd7407c72590940"}
Jul 27 22:36:05.733972 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:36:05.733714    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-6fccdccf8b-4t8h2"
Jul 27 22:36:05.753241 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:36:05.753164    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-6fccdccf8b-4t8h2" podStartSLOduration=2.768196664 podStartE2EDuration="3.75314496s" totalImagesPullingTime="0s" totalInitContainerRuntime="984.948296ms" isStatefulPod=false podCreationTimestamp="2026-07-27 22:36:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 22:36:05.752052002 +0000 UTC m=+5494.331236034" watchObservedRunningTime="2026-07-27 22:36:05.75314496 +0000 UTC m=+5494.332328969"
Jul 27 22:36:06.739290 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:36:06.739255    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-6fccdccf8b-4t8h2"
Jul 27 22:36:06.740568 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:36:06.740531    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-6fccdccf8b-4t8h2" podUID="f4467b1c-5419-40f1-99ac-63fe63a8198c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.50:8080: connect: connection refused"
Jul 27 22:36:07.744516 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:36:07.744471    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-6fccdccf8b-4t8h2" podUID="f4467b1c-5419-40f1-99ac-63fe63a8198c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.50:8080: connect: connection refused"
Jul 27 22:36:12.748909 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:36:12.748878    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-6fccdccf8b-4t8h2"
Jul 27 22:36:12.749504 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:36:12.749472    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-6fccdccf8b-4t8h2" podUID="f4467b1c-5419-40f1-99ac-63fe63a8198c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.50:8080: connect: connection refused"
Jul 27 22:36:22.750360 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:36:22.750312    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-6fccdccf8b-4t8h2" podUID="f4467b1c-5419-40f1-99ac-63fe63a8198c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.50:8080: connect: connection refused"
Jul 27 22:36:32.749667 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:36:32.749620    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-6fccdccf8b-4t8h2" podUID="f4467b1c-5419-40f1-99ac-63fe63a8198c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.50:8080: connect: connection refused"
Jul 27 22:36:42.755000 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:36:42.749575    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-6fccdccf8b-4t8h2" podUID="f4467b1c-5419-40f1-99ac-63fe63a8198c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.50:8080: connect: connection refused"
Jul 27 22:36:52.750084 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:36:52.750041    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-6fccdccf8b-4t8h2" podUID="f4467b1c-5419-40f1-99ac-63fe63a8198c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.50:8080: connect: connection refused"
Jul 27 22:37:02.749444 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:37:02.749404    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-6fccdccf8b-4t8h2" podUID="f4467b1c-5419-40f1-99ac-63fe63a8198c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.50:8080: connect: connection refused"
Jul 27 22:37:12.749516 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:37:12.749476    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-6fccdccf8b-4t8h2" podUID="f4467b1c-5419-40f1-99ac-63fe63a8198c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.50:8080: connect: connection refused"
Jul 27 22:37:22.750389 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:37:22.750358    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-6fccdccf8b-4t8h2"
Jul 27 22:37:23.042846 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:37:23.042759    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-6fccdccf8b-4t8h2"]
Jul 27 22:37:23.107294 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:37:23.107199    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-6fccdccf8b-4t8h2" podUID="f4467b1c-5419-40f1-99ac-63fe63a8198c" containerName="kserve-container" containerID="cri-o://2224fef2002566862a025e3b61ef4fbd830d39e5266837088cd7407c72590940" gracePeriod=30
Jul 27 22:37:23.107294 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:37:23.107240    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-6fccdccf8b-4t8h2" podUID="f4467b1c-5419-40f1-99ac-63fe63a8198c" containerName="kube-rbac-proxy" containerID="cri-o://0d864dcafbe2b4e68e4dfc278f338629ad0ffbbbc6e8f5b42b7be986333b8ba0" gracePeriod=30
Jul 27 22:37:24.114588 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:37:24.114558    2559 generic.go:356] "Generic (PLEG): container finished" podID="f4467b1c-5419-40f1-99ac-63fe63a8198c" containerID="0d864dcafbe2b4e68e4dfc278f338629ad0ffbbbc6e8f5b42b7be986333b8ba0" exitCode=2
Jul 27 22:37:24.114980 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:37:24.114605    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-6fccdccf8b-4t8h2" event={"ID":"f4467b1c-5419-40f1-99ac-63fe63a8198c","Type":"ContainerDied","Data":"0d864dcafbe2b4e68e4dfc278f338629ad0ffbbbc6e8f5b42b7be986333b8ba0"}
Jul 27 22:37:24.184087 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:37:24.184052    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-599448cc44-c8wrl"]
Jul 27 22:37:24.184440 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:37:24.184427    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="691de555-00ec-4f47-b944-c921261db172" containerName="storage-initializer"
Jul 27 22:37:24.184496 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:37:24.184442    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="691de555-00ec-4f47-b944-c921261db172" containerName="storage-initializer"
Jul 27 22:37:24.184535 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:37:24.184513    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="691de555-00ec-4f47-b944-c921261db172" containerName="storage-initializer"
Jul 27 22:37:24.184581 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:37:24.184573    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="691de555-00ec-4f47-b944-c921261db172" containerName="storage-initializer"
Jul 27 22:37:24.184616 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:37:24.184581    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="691de555-00ec-4f47-b944-c921261db172" containerName="storage-initializer"
Jul 27 22:37:24.184655 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:37:24.184644    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="691de555-00ec-4f47-b944-c921261db172" containerName="storage-initializer"
Jul 27 22:37:24.187700 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:37:24.187683    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-599448cc44-c8wrl"
Jul 27 22:37:24.192324 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:37:24.192306    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-tls-serving-fail-kube-rbac-proxy-sar-config\""
Jul 27 22:37:24.192420 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:37:24.192405    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-tls-serving-fail-predictor-serving-cert\""
Jul 27 22:37:24.201795 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:37:24.201768    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-599448cc44-c8wrl"]
Jul 27 22:37:24.274982 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:37:24.274948    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k2qbb\" (UniqueName: \"kubernetes.io/projected/175f8119-47e8-43ab-b3f2-3f0aaa909857-kube-api-access-k2qbb\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-599448cc44-c8wrl\" (UID: \"175f8119-47e8-43ab-b3f2-3f0aaa909857\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-599448cc44-c8wrl"
Jul 27 22:37:24.275141 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:37:24.275030    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/175f8119-47e8-43ab-b3f2-3f0aaa909857-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-599448cc44-c8wrl\" (UID: \"175f8119-47e8-43ab-b3f2-3f0aaa909857\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-599448cc44-c8wrl"
Jul 27 22:37:24.275141 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:37:24.275067    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/175f8119-47e8-43ab-b3f2-3f0aaa909857-proxy-tls\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-599448cc44-c8wrl\" (UID: \"175f8119-47e8-43ab-b3f2-3f0aaa909857\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-599448cc44-c8wrl"
Jul 27 22:37:24.275141 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:37:24.275090    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/175f8119-47e8-43ab-b3f2-3f0aaa909857-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-599448cc44-c8wrl\" (UID: \"175f8119-47e8-43ab-b3f2-3f0aaa909857\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-599448cc44-c8wrl"
Jul 27 22:37:24.376055 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:37:24.375977    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/175f8119-47e8-43ab-b3f2-3f0aaa909857-proxy-tls\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-599448cc44-c8wrl\" (UID: \"175f8119-47e8-43ab-b3f2-3f0aaa909857\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-599448cc44-c8wrl"
Jul 27 22:37:24.376055 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:37:24.376007    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/175f8119-47e8-43ab-b3f2-3f0aaa909857-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-599448cc44-c8wrl\" (UID: \"175f8119-47e8-43ab-b3f2-3f0aaa909857\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-599448cc44-c8wrl"
Jul 27 22:37:24.376262 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:37:24.376141    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k2qbb\" (UniqueName: \"kubernetes.io/projected/175f8119-47e8-43ab-b3f2-3f0aaa909857-kube-api-access-k2qbb\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-599448cc44-c8wrl\" (UID: \"175f8119-47e8-43ab-b3f2-3f0aaa909857\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-599448cc44-c8wrl"
Jul 27 22:37:24.376313 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:37:24.376269    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/175f8119-47e8-43ab-b3f2-3f0aaa909857-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-599448cc44-c8wrl\" (UID: \"175f8119-47e8-43ab-b3f2-3f0aaa909857\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-599448cc44-c8wrl"
Jul 27 22:37:24.376493 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:37:24.376474    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/175f8119-47e8-43ab-b3f2-3f0aaa909857-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-599448cc44-c8wrl\" (UID: \"175f8119-47e8-43ab-b3f2-3f0aaa909857\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-599448cc44-c8wrl"
Jul 27 22:37:24.376869 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:37:24.376851    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/175f8119-47e8-43ab-b3f2-3f0aaa909857-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-599448cc44-c8wrl\" (UID: \"175f8119-47e8-43ab-b3f2-3f0aaa909857\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-599448cc44-c8wrl"
Jul 27 22:37:24.378449 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:37:24.378428    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/175f8119-47e8-43ab-b3f2-3f0aaa909857-proxy-tls\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-599448cc44-c8wrl\" (UID: \"175f8119-47e8-43ab-b3f2-3f0aaa909857\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-599448cc44-c8wrl"
Jul 27 22:37:24.384623 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:37:24.384598    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k2qbb\" (UniqueName: \"kubernetes.io/projected/175f8119-47e8-43ab-b3f2-3f0aaa909857-kube-api-access-k2qbb\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-599448cc44-c8wrl\" (UID: \"175f8119-47e8-43ab-b3f2-3f0aaa909857\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-599448cc44-c8wrl"
Jul 27 22:37:24.498198 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:37:24.498162    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-599448cc44-c8wrl"
Jul 27 22:37:24.630107 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:37:24.630021    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-599448cc44-c8wrl"]
Jul 27 22:37:24.632536 ip-10-0-131-133 kubenswrapper[2559]: W0727 22:37:24.632507    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod175f8119_47e8_43ab_b3f2_3f0aaa909857.slice/crio-bb45bb89134c5a44e41bd5c9ebef3fbd37aabef456125f9fe38629e4cc811270 WatchSource:0}: Error finding container bb45bb89134c5a44e41bd5c9ebef3fbd37aabef456125f9fe38629e4cc811270: Status 404 returned error can't find the container with id bb45bb89134c5a44e41bd5c9ebef3fbd37aabef456125f9fe38629e4cc811270
Jul 27 22:37:25.130710 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:37:25.130672    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-599448cc44-c8wrl" event={"ID":"175f8119-47e8-43ab-b3f2-3f0aaa909857","Type":"ContainerStarted","Data":"17ceac59f33819e65ea253f393a8e82bd164a60e4805115c6063abae1457bc1f"}
Jul 27 22:37:25.131124 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:37:25.130774    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-599448cc44-c8wrl" event={"ID":"175f8119-47e8-43ab-b3f2-3f0aaa909857","Type":"ContainerStarted","Data":"bb45bb89134c5a44e41bd5c9ebef3fbd37aabef456125f9fe38629e4cc811270"}
Jul 27 22:37:27.744608 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:37:27.744569    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-6fccdccf8b-4t8h2" podUID="f4467b1c-5419-40f1-99ac-63fe63a8198c" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.50:8643/healthz\": dial tcp 10.132.0.50:8643: connect: connection refused"
Jul 27 22:37:27.863191 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:37:27.863165    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-6fccdccf8b-4t8h2"
Jul 27 22:37:28.012175 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:37:28.012086    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5sg8q\" (UniqueName: \"kubernetes.io/projected/f4467b1c-5419-40f1-99ac-63fe63a8198c-kube-api-access-5sg8q\") pod \"f4467b1c-5419-40f1-99ac-63fe63a8198c\" (UID: \"f4467b1c-5419-40f1-99ac-63fe63a8198c\") "
Jul 27 22:37:28.012175 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:37:28.012160    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f4467b1c-5419-40f1-99ac-63fe63a8198c-proxy-tls\") pod \"f4467b1c-5419-40f1-99ac-63fe63a8198c\" (UID: \"f4467b1c-5419-40f1-99ac-63fe63a8198c\") "
Jul 27 22:37:28.012443 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:37:28.012192    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f4467b1c-5419-40f1-99ac-63fe63a8198c-kube-rbac-proxy-sar-config\") pod \"f4467b1c-5419-40f1-99ac-63fe63a8198c\" (UID: \"f4467b1c-5419-40f1-99ac-63fe63a8198c\") "
Jul 27 22:37:28.012443 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:37:28.012237    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f4467b1c-5419-40f1-99ac-63fe63a8198c-kserve-provision-location\") pod \"f4467b1c-5419-40f1-99ac-63fe63a8198c\" (UID: \"f4467b1c-5419-40f1-99ac-63fe63a8198c\") "
Jul 27 22:37:28.012443 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:37:28.012266    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f4467b1c-5419-40f1-99ac-63fe63a8198c-cabundle-cert\") pod \"f4467b1c-5419-40f1-99ac-63fe63a8198c\" (UID: \"f4467b1c-5419-40f1-99ac-63fe63a8198c\") "
Jul 27 22:37:28.012601 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:37:28.012564    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f4467b1c-5419-40f1-99ac-63fe63a8198c-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "f4467b1c-5419-40f1-99ac-63fe63a8198c" (UID: "f4467b1c-5419-40f1-99ac-63fe63a8198c"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 22:37:28.012601 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:37:28.012577    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f4467b1c-5419-40f1-99ac-63fe63a8198c-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "f4467b1c-5419-40f1-99ac-63fe63a8198c" (UID: "f4467b1c-5419-40f1-99ac-63fe63a8198c"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 22:37:28.012736 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:37:28.012708    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f4467b1c-5419-40f1-99ac-63fe63a8198c-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "f4467b1c-5419-40f1-99ac-63fe63a8198c" (UID: "f4467b1c-5419-40f1-99ac-63fe63a8198c"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 22:37:28.014389 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:37:28.014362    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f4467b1c-5419-40f1-99ac-63fe63a8198c-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "f4467b1c-5419-40f1-99ac-63fe63a8198c" (UID: "f4467b1c-5419-40f1-99ac-63fe63a8198c"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 22:37:28.014389 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:37:28.014375    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f4467b1c-5419-40f1-99ac-63fe63a8198c-kube-api-access-5sg8q" (OuterVolumeSpecName: "kube-api-access-5sg8q") pod "f4467b1c-5419-40f1-99ac-63fe63a8198c" (UID: "f4467b1c-5419-40f1-99ac-63fe63a8198c"). InnerVolumeSpecName "kube-api-access-5sg8q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 22:37:28.113658 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:37:28.113608    2559 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f4467b1c-5419-40f1-99ac-63fe63a8198c-kube-rbac-proxy-sar-config\") on node \"ip-10-0-131-133.ec2.internal\" DevicePath \"\""
Jul 27 22:37:28.113658 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:37:28.113648    2559 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f4467b1c-5419-40f1-99ac-63fe63a8198c-kserve-provision-location\") on node \"ip-10-0-131-133.ec2.internal\" DevicePath \"\""
Jul 27 22:37:28.113658 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:37:28.113659    2559 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f4467b1c-5419-40f1-99ac-63fe63a8198c-cabundle-cert\") on node \"ip-10-0-131-133.ec2.internal\" DevicePath \"\""
Jul 27 22:37:28.113658 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:37:28.113667    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5sg8q\" (UniqueName: \"kubernetes.io/projected/f4467b1c-5419-40f1-99ac-63fe63a8198c-kube-api-access-5sg8q\") on node \"ip-10-0-131-133.ec2.internal\" DevicePath \"\""
Jul 27 22:37:28.113965 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:37:28.113693    2559 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f4467b1c-5419-40f1-99ac-63fe63a8198c-proxy-tls\") on node \"ip-10-0-131-133.ec2.internal\" DevicePath \"\""
Jul 27 22:37:28.146442 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:37:28.146411    2559 generic.go:356] "Generic (PLEG): container finished" podID="f4467b1c-5419-40f1-99ac-63fe63a8198c" containerID="2224fef2002566862a025e3b61ef4fbd830d39e5266837088cd7407c72590940" exitCode=0
Jul 27 22:37:28.146570 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:37:28.146457    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-6fccdccf8b-4t8h2" event={"ID":"f4467b1c-5419-40f1-99ac-63fe63a8198c","Type":"ContainerDied","Data":"2224fef2002566862a025e3b61ef4fbd830d39e5266837088cd7407c72590940"}
Jul 27 22:37:28.146570 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:37:28.146466    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-6fccdccf8b-4t8h2"
Jul 27 22:37:28.146570 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:37:28.146504    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-6fccdccf8b-4t8h2" event={"ID":"f4467b1c-5419-40f1-99ac-63fe63a8198c","Type":"ContainerDied","Data":"893f3fafcea118583de9ab10bab8633930026c7c6d141ee51e1b92934fd366fa"}
Jul 27 22:37:28.146570 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:37:28.146530    2559 scope.go:117] "RemoveContainer" containerID="0d864dcafbe2b4e68e4dfc278f338629ad0ffbbbc6e8f5b42b7be986333b8ba0"
Jul 27 22:37:28.155763 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:37:28.155743    2559 scope.go:117] "RemoveContainer" containerID="2224fef2002566862a025e3b61ef4fbd830d39e5266837088cd7407c72590940"
Jul 27 22:37:28.163508 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:37:28.163491    2559 scope.go:117] "RemoveContainer" containerID="533fa5f5d1e51ce713ac756e3dccb93332fbde7c25034dc5d7926caffe4634d4"
Jul 27 22:37:28.170503 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:37:28.170479    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-6fccdccf8b-4t8h2"]
Jul 27 22:37:28.171358 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:37:28.171343    2559 scope.go:117] "RemoveContainer" containerID="0d864dcafbe2b4e68e4dfc278f338629ad0ffbbbc6e8f5b42b7be986333b8ba0"
Jul 27 22:37:28.171624 ip-10-0-131-133 kubenswrapper[2559]: E0727 22:37:28.171606    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0d864dcafbe2b4e68e4dfc278f338629ad0ffbbbc6e8f5b42b7be986333b8ba0\": container with ID starting with 0d864dcafbe2b4e68e4dfc278f338629ad0ffbbbc6e8f5b42b7be986333b8ba0 not found: ID does not exist" containerID="0d864dcafbe2b4e68e4dfc278f338629ad0ffbbbc6e8f5b42b7be986333b8ba0"
Jul 27 22:37:28.171670 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:37:28.171632    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0d864dcafbe2b4e68e4dfc278f338629ad0ffbbbc6e8f5b42b7be986333b8ba0"} err="failed to get container status \"0d864dcafbe2b4e68e4dfc278f338629ad0ffbbbc6e8f5b42b7be986333b8ba0\": rpc error: code = NotFound desc = could not find container \"0d864dcafbe2b4e68e4dfc278f338629ad0ffbbbc6e8f5b42b7be986333b8ba0\": container with ID starting with 0d864dcafbe2b4e68e4dfc278f338629ad0ffbbbc6e8f5b42b7be986333b8ba0 not found: ID does not exist"
Jul 27 22:37:28.171670 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:37:28.171649    2559 scope.go:117] "RemoveContainer" containerID="2224fef2002566862a025e3b61ef4fbd830d39e5266837088cd7407c72590940"
Jul 27 22:37:28.171860 ip-10-0-131-133 kubenswrapper[2559]: E0727 22:37:28.171840    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2224fef2002566862a025e3b61ef4fbd830d39e5266837088cd7407c72590940\": container with ID starting with 2224fef2002566862a025e3b61ef4fbd830d39e5266837088cd7407c72590940 not found: ID does not exist" containerID="2224fef2002566862a025e3b61ef4fbd830d39e5266837088cd7407c72590940"
Jul 27 22:37:28.171927 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:37:28.171869    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2224fef2002566862a025e3b61ef4fbd830d39e5266837088cd7407c72590940"} err="failed to get container status \"2224fef2002566862a025e3b61ef4fbd830d39e5266837088cd7407c72590940\": rpc error: code = NotFound desc = could not find container \"2224fef2002566862a025e3b61ef4fbd830d39e5266837088cd7407c72590940\": container with ID starting with 2224fef2002566862a025e3b61ef4fbd830d39e5266837088cd7407c72590940 not found: ID does not exist"
Jul 27 22:37:28.171927 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:37:28.171892    2559 scope.go:117] "RemoveContainer" containerID="533fa5f5d1e51ce713ac756e3dccb93332fbde7c25034dc5d7926caffe4634d4"
Jul 27 22:37:28.172146 ip-10-0-131-133 kubenswrapper[2559]: E0727 22:37:28.172124    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"533fa5f5d1e51ce713ac756e3dccb93332fbde7c25034dc5d7926caffe4634d4\": container with ID starting with 533fa5f5d1e51ce713ac756e3dccb93332fbde7c25034dc5d7926caffe4634d4 not found: ID does not exist" containerID="533fa5f5d1e51ce713ac756e3dccb93332fbde7c25034dc5d7926caffe4634d4"
Jul 27 22:37:28.172185 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:37:28.172151    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"533fa5f5d1e51ce713ac756e3dccb93332fbde7c25034dc5d7926caffe4634d4"} err="failed to get container status \"533fa5f5d1e51ce713ac756e3dccb93332fbde7c25034dc5d7926caffe4634d4\": rpc error: code = NotFound desc = could not find container \"533fa5f5d1e51ce713ac756e3dccb93332fbde7c25034dc5d7926caffe4634d4\": container with ID starting with 533fa5f5d1e51ce713ac756e3dccb93332fbde7c25034dc5d7926caffe4634d4 not found: ID does not exist"
Jul 27 22:37:28.180456 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:37:28.180433    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-6fccdccf8b-4t8h2"]
Jul 27 22:37:29.981811 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:37:29.981783    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f4467b1c-5419-40f1-99ac-63fe63a8198c" path="/var/lib/kubelet/pods/f4467b1c-5419-40f1-99ac-63fe63a8198c/volumes"
Jul 27 22:37:30.158521 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:37:30.158499    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-serving-fail-predictor-599448cc44-c8wrl_175f8119-47e8-43ab-b3f2-3f0aaa909857/storage-initializer/0.log"
Jul 27 22:37:30.158639 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:37:30.158533    2559 generic.go:356] "Generic (PLEG): container finished" podID="175f8119-47e8-43ab-b3f2-3f0aaa909857" containerID="17ceac59f33819e65ea253f393a8e82bd164a60e4805115c6063abae1457bc1f" exitCode=1
Jul 27 22:37:30.158639 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:37:30.158608    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-599448cc44-c8wrl" event={"ID":"175f8119-47e8-43ab-b3f2-3f0aaa909857","Type":"ContainerDied","Data":"17ceac59f33819e65ea253f393a8e82bd164a60e4805115c6063abae1457bc1f"}
Jul 27 22:37:31.164993 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:37:31.164966    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-serving-fail-predictor-599448cc44-c8wrl_175f8119-47e8-43ab-b3f2-3f0aaa909857/storage-initializer/0.log"
Jul 27 22:37:31.165488 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:37:31.165095    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-599448cc44-c8wrl" event={"ID":"175f8119-47e8-43ab-b3f2-3f0aaa909857","Type":"ContainerStarted","Data":"404d883fb5fc4f9bfaa9d5b8502aadacde6f99dba2f71eaca1b94a12f8168ed5"}
Jul 27 22:37:34.102338 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:37:34.102303    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-599448cc44-c8wrl"]
Jul 27 22:37:34.102743 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:37:34.102519    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-599448cc44-c8wrl" podUID="175f8119-47e8-43ab-b3f2-3f0aaa909857" containerName="storage-initializer" containerID="cri-o://404d883fb5fc4f9bfaa9d5b8502aadacde6f99dba2f71eaca1b94a12f8168ed5" gracePeriod=30
Jul 27 22:37:34.957969 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:37:34.957945    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-serving-fail-predictor-599448cc44-c8wrl_175f8119-47e8-43ab-b3f2-3f0aaa909857/storage-initializer/1.log"
Jul 27 22:37:34.958366 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:37:34.958348    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-serving-fail-predictor-599448cc44-c8wrl_175f8119-47e8-43ab-b3f2-3f0aaa909857/storage-initializer/0.log"
Jul 27 22:37:34.960643 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:37:34.960625    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-599448cc44-c8wrl"
Jul 27 22:37:35.080017 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:37:35.079919    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/175f8119-47e8-43ab-b3f2-3f0aaa909857-kserve-provision-location\") pod \"175f8119-47e8-43ab-b3f2-3f0aaa909857\" (UID: \"175f8119-47e8-43ab-b3f2-3f0aaa909857\") "
Jul 27 22:37:35.080198 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:37:35.080034    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/175f8119-47e8-43ab-b3f2-3f0aaa909857-kube-rbac-proxy-sar-config\") pod \"175f8119-47e8-43ab-b3f2-3f0aaa909857\" (UID: \"175f8119-47e8-43ab-b3f2-3f0aaa909857\") "
Jul 27 22:37:35.080198 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:37:35.080072    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/175f8119-47e8-43ab-b3f2-3f0aaa909857-proxy-tls\") pod \"175f8119-47e8-43ab-b3f2-3f0aaa909857\" (UID: \"175f8119-47e8-43ab-b3f2-3f0aaa909857\") "
Jul 27 22:37:35.080198 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:37:35.080095    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-k2qbb\" (UniqueName: \"kubernetes.io/projected/175f8119-47e8-43ab-b3f2-3f0aaa909857-kube-api-access-k2qbb\") pod \"175f8119-47e8-43ab-b3f2-3f0aaa909857\" (UID: \"175f8119-47e8-43ab-b3f2-3f0aaa909857\") "
Jul 27 22:37:35.080398 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:37:35.080273    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/175f8119-47e8-43ab-b3f2-3f0aaa909857-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "175f8119-47e8-43ab-b3f2-3f0aaa909857" (UID: "175f8119-47e8-43ab-b3f2-3f0aaa909857"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 22:37:35.080439 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:37:35.080422    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/175f8119-47e8-43ab-b3f2-3f0aaa909857-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "175f8119-47e8-43ab-b3f2-3f0aaa909857" (UID: "175f8119-47e8-43ab-b3f2-3f0aaa909857"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 22:37:35.082239 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:37:35.082201    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/175f8119-47e8-43ab-b3f2-3f0aaa909857-kube-api-access-k2qbb" (OuterVolumeSpecName: "kube-api-access-k2qbb") pod "175f8119-47e8-43ab-b3f2-3f0aaa909857" (UID: "175f8119-47e8-43ab-b3f2-3f0aaa909857"). InnerVolumeSpecName "kube-api-access-k2qbb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 22:37:35.082312 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:37:35.082262    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/175f8119-47e8-43ab-b3f2-3f0aaa909857-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "175f8119-47e8-43ab-b3f2-3f0aaa909857" (UID: "175f8119-47e8-43ab-b3f2-3f0aaa909857"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 22:37:35.180738 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:37:35.180707    2559 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/175f8119-47e8-43ab-b3f2-3f0aaa909857-kube-rbac-proxy-sar-config\") on node \"ip-10-0-131-133.ec2.internal\" DevicePath \"\""
Jul 27 22:37:35.180738 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:37:35.180735    2559 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/175f8119-47e8-43ab-b3f2-3f0aaa909857-proxy-tls\") on node \"ip-10-0-131-133.ec2.internal\" DevicePath \"\""
Jul 27 22:37:35.180738 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:37:35.180746    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k2qbb\" (UniqueName: \"kubernetes.io/projected/175f8119-47e8-43ab-b3f2-3f0aaa909857-kube-api-access-k2qbb\") on node \"ip-10-0-131-133.ec2.internal\" DevicePath \"\""
Jul 27 22:37:35.181188 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:37:35.180755    2559 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/175f8119-47e8-43ab-b3f2-3f0aaa909857-kserve-provision-location\") on node \"ip-10-0-131-133.ec2.internal\" DevicePath \"\""
Jul 27 22:37:35.186748 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:37:35.186725    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-serving-fail-predictor-599448cc44-c8wrl_175f8119-47e8-43ab-b3f2-3f0aaa909857/storage-initializer/1.log"
Jul 27 22:37:35.187098 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:37:35.187083    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-serving-fail-predictor-599448cc44-c8wrl_175f8119-47e8-43ab-b3f2-3f0aaa909857/storage-initializer/0.log"
Jul 27 22:37:35.187157 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:37:35.187120    2559 generic.go:356] "Generic (PLEG): container finished" podID="175f8119-47e8-43ab-b3f2-3f0aaa909857" containerID="404d883fb5fc4f9bfaa9d5b8502aadacde6f99dba2f71eaca1b94a12f8168ed5" exitCode=1
Jul 27 22:37:35.187231 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:37:35.187154    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-599448cc44-c8wrl" event={"ID":"175f8119-47e8-43ab-b3f2-3f0aaa909857","Type":"ContainerDied","Data":"404d883fb5fc4f9bfaa9d5b8502aadacde6f99dba2f71eaca1b94a12f8168ed5"}
Jul 27 22:37:35.187231 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:37:35.187196    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-599448cc44-c8wrl"
Jul 27 22:37:35.187354 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:37:35.187233    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-599448cc44-c8wrl" event={"ID":"175f8119-47e8-43ab-b3f2-3f0aaa909857","Type":"ContainerDied","Data":"bb45bb89134c5a44e41bd5c9ebef3fbd37aabef456125f9fe38629e4cc811270"}
Jul 27 22:37:35.187354 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:37:35.187261    2559 scope.go:117] "RemoveContainer" containerID="404d883fb5fc4f9bfaa9d5b8502aadacde6f99dba2f71eaca1b94a12f8168ed5"
Jul 27 22:37:35.196598 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:37:35.196578    2559 scope.go:117] "RemoveContainer" containerID="17ceac59f33819e65ea253f393a8e82bd164a60e4805115c6063abae1457bc1f"
Jul 27 22:37:35.204400 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:37:35.204363    2559 scope.go:117] "RemoveContainer" containerID="404d883fb5fc4f9bfaa9d5b8502aadacde6f99dba2f71eaca1b94a12f8168ed5"
Jul 27 22:37:35.204645 ip-10-0-131-133 kubenswrapper[2559]: E0727 22:37:35.204623    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"404d883fb5fc4f9bfaa9d5b8502aadacde6f99dba2f71eaca1b94a12f8168ed5\": container with ID starting with 404d883fb5fc4f9bfaa9d5b8502aadacde6f99dba2f71eaca1b94a12f8168ed5 not found: ID does not exist" containerID="404d883fb5fc4f9bfaa9d5b8502aadacde6f99dba2f71eaca1b94a12f8168ed5"
Jul 27 22:37:35.204688 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:37:35.204654    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"404d883fb5fc4f9bfaa9d5b8502aadacde6f99dba2f71eaca1b94a12f8168ed5"} err="failed to get container status \"404d883fb5fc4f9bfaa9d5b8502aadacde6f99dba2f71eaca1b94a12f8168ed5\": rpc error: code = NotFound desc = could not find container \"404d883fb5fc4f9bfaa9d5b8502aadacde6f99dba2f71eaca1b94a12f8168ed5\": container with ID starting with 404d883fb5fc4f9bfaa9d5b8502aadacde6f99dba2f71eaca1b94a12f8168ed5 not found: ID does not exist"
Jul 27 22:37:35.204688 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:37:35.204672    2559 scope.go:117] "RemoveContainer" containerID="17ceac59f33819e65ea253f393a8e82bd164a60e4805115c6063abae1457bc1f"
Jul 27 22:37:35.204886 ip-10-0-131-133 kubenswrapper[2559]: E0727 22:37:35.204872    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"17ceac59f33819e65ea253f393a8e82bd164a60e4805115c6063abae1457bc1f\": container with ID starting with 17ceac59f33819e65ea253f393a8e82bd164a60e4805115c6063abae1457bc1f not found: ID does not exist" containerID="17ceac59f33819e65ea253f393a8e82bd164a60e4805115c6063abae1457bc1f"
Jul 27 22:37:35.204924 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:37:35.204891    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"17ceac59f33819e65ea253f393a8e82bd164a60e4805115c6063abae1457bc1f"} err="failed to get container status \"17ceac59f33819e65ea253f393a8e82bd164a60e4805115c6063abae1457bc1f\": rpc error: code = NotFound desc = could not find container \"17ceac59f33819e65ea253f393a8e82bd164a60e4805115c6063abae1457bc1f\": container with ID starting with 17ceac59f33819e65ea253f393a8e82bd164a60e4805115c6063abae1457bc1f not found: ID does not exist"
Jul 27 22:37:35.221351 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:37:35.221322    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-599448cc44-c8wrl"]
Jul 27 22:37:35.227398 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:37:35.227372    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-599448cc44-c8wrl"]
Jul 27 22:37:35.982438 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:37:35.982404    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="175f8119-47e8-43ab-b3f2-3f0aaa909857" path="/var/lib/kubelet/pods/175f8119-47e8-43ab-b3f2-3f0aaa909857/volumes"
Jul 27 22:38:13.197975 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:38:13.197940    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-rxl8w_00ef4d6d-0b8e-4ece-ba97-36a9e236ec53/global-pull-secret-syncer/0.log"
Jul 27 22:38:13.402784 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:38:13.402752    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-tfvtq_4ff24566-f9aa-4eab-8795-a2808831cec1/konnectivity-agent/0.log"
Jul 27 22:38:13.425911 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:38:13.425882    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-131-133.ec2.internal_c500a9e213c47a39763b09a5e1d404dc/haproxy/0.log"
Jul 27 22:38:17.653113 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:38:17.653083    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_364d8616-829c-4e2c-8531-cf977166dbf3/alertmanager/0.log"
Jul 27 22:38:17.685117 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:38:17.685092    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_364d8616-829c-4e2c-8531-cf977166dbf3/config-reloader/0.log"
Jul 27 22:38:17.708851 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:38:17.708823    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_364d8616-829c-4e2c-8531-cf977166dbf3/kube-rbac-proxy-web/0.log"
Jul 27 22:38:17.735783 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:38:17.735756    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_364d8616-829c-4e2c-8531-cf977166dbf3/kube-rbac-proxy/0.log"
Jul 27 22:38:17.761858 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:38:17.761831    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_364d8616-829c-4e2c-8531-cf977166dbf3/kube-rbac-proxy-metric/0.log"
Jul 27 22:38:17.790504 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:38:17.790475    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_364d8616-829c-4e2c-8531-cf977166dbf3/prom-label-proxy/0.log"
Jul 27 22:38:17.821323 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:38:17.821299    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_364d8616-829c-4e2c-8531-cf977166dbf3/init-config-reloader/0.log"
Jul 27 22:38:17.882142 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:38:17.882109    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-548b489cb8-dxktc_a05c7c31-10b9-4e3b-84a9-afe9cf929e07/cluster-monitoring-operator/0.log"
Jul 27 22:38:17.911533 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:38:17.911452    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-56d9f944b4-dbv9r_7854d704-78c9-468b-9973-8799da0fe0eb/kube-state-metrics/0.log"
Jul 27 22:38:17.939184 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:38:17.939157    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-56d9f944b4-dbv9r_7854d704-78c9-468b-9973-8799da0fe0eb/kube-rbac-proxy-main/0.log"
Jul 27 22:38:17.968394 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:38:17.968365    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-56d9f944b4-dbv9r_7854d704-78c9-468b-9973-8799da0fe0eb/kube-rbac-proxy-self/0.log"
Jul 27 22:38:18.000785 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:38:18.000748    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-686dbf5f69-zb2mj_45e10f06-0e63-4eac-b3c6-42287beb9a75/metrics-server/0.log"
Jul 27 22:38:18.025594 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:38:18.025565    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-68b7fc9574-gmwds_fbf8db18-7974-4032-b919-6be505ffe0e1/monitoring-plugin/0.log"
Jul 27 22:38:18.057414 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:38:18.057387    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-6ngt5_1b8f587c-fd7c-44dd-903f-0eced954b117/node-exporter/0.log"
Jul 27 22:38:18.081460 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:38:18.081436    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-6ngt5_1b8f587c-fd7c-44dd-903f-0eced954b117/kube-rbac-proxy/0.log"
Jul 27 22:38:18.107746 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:38:18.107719    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-6ngt5_1b8f587c-fd7c-44dd-903f-0eced954b117/init-textfile/0.log"
Jul 27 22:38:18.698920 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:38:18.698894    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-5d57f6db47-l8m59_c5a90613-ef7f-4695-a77e-ad78f0327434/telemeter-client/0.log"
Jul 27 22:38:18.731388 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:38:18.731362    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-5d57f6db47-l8m59_c5a90613-ef7f-4695-a77e-ad78f0327434/reload/0.log"
Jul 27 22:38:18.758840 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:38:18.758814    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-5d57f6db47-l8m59_c5a90613-ef7f-4695-a77e-ad78f0327434/kube-rbac-proxy/0.log"
Jul 27 22:38:18.819104 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:38:18.819077    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-64dfb7c55b-blsbx_3ba37884-dc16-419a-b9ca-ba49c6171c9c/thanos-query/0.log"
Jul 27 22:38:18.858810 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:38:18.858783    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-64dfb7c55b-blsbx_3ba37884-dc16-419a-b9ca-ba49c6171c9c/kube-rbac-proxy-web/0.log"
Jul 27 22:38:18.899818 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:38:18.899790    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-64dfb7c55b-blsbx_3ba37884-dc16-419a-b9ca-ba49c6171c9c/kube-rbac-proxy/0.log"
Jul 27 22:38:18.939771 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:38:18.939744    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-64dfb7c55b-blsbx_3ba37884-dc16-419a-b9ca-ba49c6171c9c/prom-label-proxy/0.log"
Jul 27 22:38:18.984389 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:38:18.984316    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-64dfb7c55b-blsbx_3ba37884-dc16-419a-b9ca-ba49c6171c9c/kube-rbac-proxy-rules/0.log"
Jul 27 22:38:19.019338 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:38:19.019310    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-64dfb7c55b-blsbx_3ba37884-dc16-419a-b9ca-ba49c6171c9c/kube-rbac-proxy-metrics/0.log"
Jul 27 22:38:20.204905 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:38:20.204824    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-console_networking-console-plugin-74c9f884b7-bd6vk_d8270118-51fb-45fb-99ed-6f3aa670345c/networking-console-plugin/0.log"
Jul 27 22:38:20.664897 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:38:20.664868    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-bjvbc_cba53c9a-64c0-4afa-8731-35521e196622/console-operator/1.log"
Jul 27 22:38:20.669408 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:38:20.669383    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-bjvbc_cba53c9a-64c0-4afa-8731-35521e196622/console-operator/2.log"
Jul 27 22:38:21.079410 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:38:21.079333    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-9c975f545-d5zl9_7fa641ac-3826-4461-8804-b57acc5863d9/console/0.log"
Jul 27 22:38:21.119964 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:38:21.119934    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_downloads-7c97dc4555-7x89h_2ded1152-f764-48fc-b45f-744519e49e62/download-server/0.log"
Jul 27 22:38:21.262162 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:38:21.262130    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-t7jph/perf-node-gather-daemonset-zxc8m"]
Jul 27 22:38:21.262555 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:38:21.262489    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f4467b1c-5419-40f1-99ac-63fe63a8198c" containerName="storage-initializer"
Jul 27 22:38:21.262555 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:38:21.262499    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="f4467b1c-5419-40f1-99ac-63fe63a8198c" containerName="storage-initializer"
Jul 27 22:38:21.262555 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:38:21.262512    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f4467b1c-5419-40f1-99ac-63fe63a8198c" containerName="kserve-container"
Jul 27 22:38:21.262555 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:38:21.262518    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="f4467b1c-5419-40f1-99ac-63fe63a8198c" containerName="kserve-container"
Jul 27 22:38:21.262555 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:38:21.262533    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="175f8119-47e8-43ab-b3f2-3f0aaa909857" containerName="storage-initializer"
Jul 27 22:38:21.262555 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:38:21.262538    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="175f8119-47e8-43ab-b3f2-3f0aaa909857" containerName="storage-initializer"
Jul 27 22:38:21.262555 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:38:21.262547    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="175f8119-47e8-43ab-b3f2-3f0aaa909857" containerName="storage-initializer"
Jul 27 22:38:21.262555 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:38:21.262552    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="175f8119-47e8-43ab-b3f2-3f0aaa909857" containerName="storage-initializer"
Jul 27 22:38:21.262555 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:38:21.262560    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f4467b1c-5419-40f1-99ac-63fe63a8198c" containerName="kube-rbac-proxy"
Jul 27 22:38:21.262857 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:38:21.262565    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="f4467b1c-5419-40f1-99ac-63fe63a8198c" containerName="kube-rbac-proxy"
Jul 27 22:38:21.262857 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:38:21.262623    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="175f8119-47e8-43ab-b3f2-3f0aaa909857" containerName="storage-initializer"
Jul 27 22:38:21.262857 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:38:21.262629    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="175f8119-47e8-43ab-b3f2-3f0aaa909857" containerName="storage-initializer"
Jul 27 22:38:21.262857 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:38:21.262635    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="f4467b1c-5419-40f1-99ac-63fe63a8198c" containerName="kube-rbac-proxy"
Jul 27 22:38:21.262857 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:38:21.262644    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="f4467b1c-5419-40f1-99ac-63fe63a8198c" containerName="kserve-container"
Jul 27 22:38:21.265883 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:38:21.265860    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-t7jph/perf-node-gather-daemonset-zxc8m"
Jul 27 22:38:21.268169 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:38:21.268145    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-t7jph\"/\"openshift-service-ca.crt\""
Jul 27 22:38:21.268306 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:38:21.268203    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-t7jph\"/\"kube-root-ca.crt\""
Jul 27 22:38:21.268306 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:38:21.268287    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-t7jph\"/\"default-dockercfg-wwbv9\""
Jul 27 22:38:21.275723 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:38:21.275695    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-t7jph/perf-node-gather-daemonset-zxc8m"]
Jul 27 22:38:21.394007 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:38:21.393975    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/c21bcd2c-1e05-47f9-a6f2-31d6c5809efc-podres\") pod \"perf-node-gather-daemonset-zxc8m\" (UID: \"c21bcd2c-1e05-47f9-a6f2-31d6c5809efc\") " pod="openshift-must-gather-t7jph/perf-node-gather-daemonset-zxc8m"
Jul 27 22:38:21.394195 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:38:21.394019    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/c21bcd2c-1e05-47f9-a6f2-31d6c5809efc-lib-modules\") pod \"perf-node-gather-daemonset-zxc8m\" (UID: \"c21bcd2c-1e05-47f9-a6f2-31d6c5809efc\") " pod="openshift-must-gather-t7jph/perf-node-gather-daemonset-zxc8m"
Jul 27 22:38:21.394195 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:38:21.394039    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/c21bcd2c-1e05-47f9-a6f2-31d6c5809efc-proc\") pod \"perf-node-gather-daemonset-zxc8m\" (UID: \"c21bcd2c-1e05-47f9-a6f2-31d6c5809efc\") " pod="openshift-must-gather-t7jph/perf-node-gather-daemonset-zxc8m"
Jul 27 22:38:21.394195 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:38:21.394134    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c21bcd2c-1e05-47f9-a6f2-31d6c5809efc-sys\") pod \"perf-node-gather-daemonset-zxc8m\" (UID: \"c21bcd2c-1e05-47f9-a6f2-31d6c5809efc\") " pod="openshift-must-gather-t7jph/perf-node-gather-daemonset-zxc8m"
Jul 27 22:38:21.394195 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:38:21.394179    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dc5nd\" (UniqueName: \"kubernetes.io/projected/c21bcd2c-1e05-47f9-a6f2-31d6c5809efc-kube-api-access-dc5nd\") pod \"perf-node-gather-daemonset-zxc8m\" (UID: \"c21bcd2c-1e05-47f9-a6f2-31d6c5809efc\") " pod="openshift-must-gather-t7jph/perf-node-gather-daemonset-zxc8m"
Jul 27 22:38:21.495178 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:38:21.495140    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dc5nd\" (UniqueName: \"kubernetes.io/projected/c21bcd2c-1e05-47f9-a6f2-31d6c5809efc-kube-api-access-dc5nd\") pod \"perf-node-gather-daemonset-zxc8m\" (UID: \"c21bcd2c-1e05-47f9-a6f2-31d6c5809efc\") " pod="openshift-must-gather-t7jph/perf-node-gather-daemonset-zxc8m"
Jul 27 22:38:21.495398 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:38:21.495199    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/c21bcd2c-1e05-47f9-a6f2-31d6c5809efc-podres\") pod \"perf-node-gather-daemonset-zxc8m\" (UID: \"c21bcd2c-1e05-47f9-a6f2-31d6c5809efc\") " pod="openshift-must-gather-t7jph/perf-node-gather-daemonset-zxc8m"
Jul 27 22:38:21.495398 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:38:21.495262    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/c21bcd2c-1e05-47f9-a6f2-31d6c5809efc-lib-modules\") pod \"perf-node-gather-daemonset-zxc8m\" (UID: \"c21bcd2c-1e05-47f9-a6f2-31d6c5809efc\") " pod="openshift-must-gather-t7jph/perf-node-gather-daemonset-zxc8m"
Jul 27 22:38:21.495398 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:38:21.495369    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/c21bcd2c-1e05-47f9-a6f2-31d6c5809efc-podres\") pod \"perf-node-gather-daemonset-zxc8m\" (UID: \"c21bcd2c-1e05-47f9-a6f2-31d6c5809efc\") " pod="openshift-must-gather-t7jph/perf-node-gather-daemonset-zxc8m"
Jul 27 22:38:21.495398 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:38:21.495378    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/c21bcd2c-1e05-47f9-a6f2-31d6c5809efc-lib-modules\") pod \"perf-node-gather-daemonset-zxc8m\" (UID: \"c21bcd2c-1e05-47f9-a6f2-31d6c5809efc\") " pod="openshift-must-gather-t7jph/perf-node-gather-daemonset-zxc8m"
Jul 27 22:38:21.495534 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:38:21.495423    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/c21bcd2c-1e05-47f9-a6f2-31d6c5809efc-proc\") pod \"perf-node-gather-daemonset-zxc8m\" (UID: \"c21bcd2c-1e05-47f9-a6f2-31d6c5809efc\") " pod="openshift-must-gather-t7jph/perf-node-gather-daemonset-zxc8m"
Jul 27 22:38:21.495534 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:38:21.495518    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c21bcd2c-1e05-47f9-a6f2-31d6c5809efc-sys\") pod \"perf-node-gather-daemonset-zxc8m\" (UID: \"c21bcd2c-1e05-47f9-a6f2-31d6c5809efc\") " pod="openshift-must-gather-t7jph/perf-node-gather-daemonset-zxc8m"
Jul 27 22:38:21.495643 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:38:21.495532    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/c21bcd2c-1e05-47f9-a6f2-31d6c5809efc-proc\") pod \"perf-node-gather-daemonset-zxc8m\" (UID: \"c21bcd2c-1e05-47f9-a6f2-31d6c5809efc\") " pod="openshift-must-gather-t7jph/perf-node-gather-daemonset-zxc8m"
Jul 27 22:38:21.495643 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:38:21.495608    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c21bcd2c-1e05-47f9-a6f2-31d6c5809efc-sys\") pod \"perf-node-gather-daemonset-zxc8m\" (UID: \"c21bcd2c-1e05-47f9-a6f2-31d6c5809efc\") " pod="openshift-must-gather-t7jph/perf-node-gather-daemonset-zxc8m"
Jul 27 22:38:21.503202 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:38:21.503170    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dc5nd\" (UniqueName: \"kubernetes.io/projected/c21bcd2c-1e05-47f9-a6f2-31d6c5809efc-kube-api-access-dc5nd\") pod \"perf-node-gather-daemonset-zxc8m\" (UID: \"c21bcd2c-1e05-47f9-a6f2-31d6c5809efc\") " pod="openshift-must-gather-t7jph/perf-node-gather-daemonset-zxc8m"
Jul 27 22:38:21.576105 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:38:21.576057    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-t7jph/perf-node-gather-daemonset-zxc8m"
Jul 27 22:38:21.713744 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:38:21.713710    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-t7jph/perf-node-gather-daemonset-zxc8m"]
Jul 27 22:38:21.716862 ip-10-0-131-133 kubenswrapper[2559]: W0727 22:38:21.716835    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podc21bcd2c_1e05_47f9_a6f2_31d6c5809efc.slice/crio-58be711eb7a763d7952bffbbb6aa3f30b6419f6fde65dba16bb5d57fccc2812c WatchSource:0}: Error finding container 58be711eb7a763d7952bffbbb6aa3f30b6419f6fde65dba16bb5d57fccc2812c: Status 404 returned error can't find the container with id 58be711eb7a763d7952bffbbb6aa3f30b6419f6fde65dba16bb5d57fccc2812c
Jul 27 22:38:22.409005 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:38:22.408957    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-t7jph/perf-node-gather-daemonset-zxc8m" event={"ID":"c21bcd2c-1e05-47f9-a6f2-31d6c5809efc","Type":"ContainerStarted","Data":"608dc4b37744aa6ef8176dcfe7f55052354fac4e86dd673090581e9c8ac47fcf"}
Jul 27 22:38:22.409440 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:38:22.409023    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-t7jph/perf-node-gather-daemonset-zxc8m"
Jul 27 22:38:22.409440 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:38:22.409041    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-t7jph/perf-node-gather-daemonset-zxc8m" event={"ID":"c21bcd2c-1e05-47f9-a6f2-31d6c5809efc","Type":"ContainerStarted","Data":"58be711eb7a763d7952bffbbb6aa3f30b6419f6fde65dba16bb5d57fccc2812c"}
Jul 27 22:38:22.426862 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:38:22.426800    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-t7jph/perf-node-gather-daemonset-zxc8m" podStartSLOduration=1.426783007 podStartE2EDuration="1.426783007s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-27 22:38:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 22:38:22.425003035 +0000 UTC m=+5631.004187043" watchObservedRunningTime="2026-07-27 22:38:22.426783007 +0000 UTC m=+5631.005967072"
Jul 27 22:38:22.669040 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:38:22.668957    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-9kfdx_ac10d2c6-12bd-4f89-8d1e-c931206a2e92/dns/0.log"
Jul 27 22:38:22.693714 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:38:22.693681    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-9kfdx_ac10d2c6-12bd-4f89-8d1e-c931206a2e92/kube-rbac-proxy/0.log"
Jul 27 22:38:22.792125 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:38:22.792096    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-fvbmc_84a7d738-f486-43b0-ac43-3401891da8d1/dns-node-resolver/0.log"
Jul 27 22:38:23.336875 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:38:23.336846    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-qxzfs_3e4c5c3f-db85-4e88-a1d5-48e602b0018a/node-ca/0.log"
Jul 27 22:38:24.457625 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:38:24.457600    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-78ljg_cf59a2e6-f3c6-4549-9158-f047bfb7e764/serve-healthcheck-canary/0.log"
Jul 27 22:38:24.979934 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:38:24.979903    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-dz5rp_922d0edc-f817-477a-bc19-a40bacc930ba/kube-rbac-proxy/0.log"
Jul 27 22:38:25.001486 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:38:25.001464    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-dz5rp_922d0edc-f817-477a-bc19-a40bacc930ba/exporter/0.log"
Jul 27 22:38:25.023938 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:38:25.023915    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-dz5rp_922d0edc-f817-477a-bc19-a40bacc930ba/extractor/0.log"
Jul 27 22:38:27.391170 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:38:27.391137    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_llmisvc-controller-manager-84fc959947-t8pnm_bb51a4e3-357d-4e40-9001-e5b6586860fd/manager/0.log"
Jul 27 22:38:27.948384 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:38:27.948354    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_seaweedfs-64568bcd49-p22l9_349b045e-11fe-4879-ba47-a0f576ba6e25/seaweedfs/0.log"
Jul 27 22:38:27.976912 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:38:27.976884    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_seaweedfs-tls-custom-8b44d767d-qqzzg_2297f5ec-fed9-4ebf-86c7-9470b5c632ae/seaweedfs-tls-custom/0.log"
Jul 27 22:38:28.005457 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:38:28.005432    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_seaweedfs-tls-serving-67b7477c6d-6fgb9_76fc64db-f5ad-464e-b9d7-ec89d17012d5/seaweedfs-tls-serving/0.log"
Jul 27 22:38:28.423980 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:38:28.423944    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-t7jph/perf-node-gather-daemonset-zxc8m"
Jul 27 22:38:32.928153 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:38:32.928125    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-6f4d6fb6bc-9d4zq_9e560b08-69f7-4ee3-8236-22a5c398a476/kube-storage-version-migrator-operator/1.log"
Jul 27 22:38:32.928916 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:38:32.928895    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-6f4d6fb6bc-9d4zq_9e560b08-69f7-4ee3-8236-22a5c398a476/kube-storage-version-migrator-operator/0.log"
Jul 27 22:38:33.804867 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:38:33.804834    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-2pbtk_4275e4fb-a8e5-43a5-8b8c-ed4cc2094674/kube-multus-additional-cni-plugins/0.log"
Jul 27 22:38:33.828508 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:38:33.828480    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-2pbtk_4275e4fb-a8e5-43a5-8b8c-ed4cc2094674/egress-router-binary-copy/0.log"
Jul 27 22:38:33.852746 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:38:33.852719    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-2pbtk_4275e4fb-a8e5-43a5-8b8c-ed4cc2094674/cni-plugins/0.log"
Jul 27 22:38:33.877369 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:38:33.877343    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-2pbtk_4275e4fb-a8e5-43a5-8b8c-ed4cc2094674/bond-cni-plugin/0.log"
Jul 27 22:38:33.902172 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:38:33.902145    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-2pbtk_4275e4fb-a8e5-43a5-8b8c-ed4cc2094674/routeoverride-cni/0.log"
Jul 27 22:38:33.927427 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:38:33.927403    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-2pbtk_4275e4fb-a8e5-43a5-8b8c-ed4cc2094674/whereabouts-cni-bincopy/0.log"
Jul 27 22:38:33.951396 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:38:33.951368    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-2pbtk_4275e4fb-a8e5-43a5-8b8c-ed4cc2094674/whereabouts-cni/0.log"
Jul 27 22:38:34.386802 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:38:34.386767    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-dchr6_6b7af06d-65e7-45ca-80ef-e88bcee2282a/kube-multus/0.log"
Jul 27 22:38:34.469546 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:38:34.469516    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-pr4c2_5851de39-848d-4a52-bc17-fefe1bf5407a/network-metrics-daemon/0.log"
Jul 27 22:38:34.491485 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:38:34.491462    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-pr4c2_5851de39-848d-4a52-bc17-fefe1bf5407a/kube-rbac-proxy/0.log"
Jul 27 22:38:35.928120 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:38:35.928091    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-qs89h_7f66b2c1-260e-4a55-902b-c4a060b723f9/ovn-controller/0.log"
Jul 27 22:38:35.946893 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:38:35.946869    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-qs89h_7f66b2c1-260e-4a55-902b-c4a060b723f9/ovn-acl-logging/0.log"
Jul 27 22:38:35.971616 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:38:35.971590    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-qs89h_7f66b2c1-260e-4a55-902b-c4a060b723f9/kube-rbac-proxy-node/0.log"
Jul 27 22:38:35.997226 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:38:35.997197    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-qs89h_7f66b2c1-260e-4a55-902b-c4a060b723f9/kube-rbac-proxy-ovn-metrics/0.log"
Jul 27 22:38:36.019488 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:38:36.019466    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-qs89h_7f66b2c1-260e-4a55-902b-c4a060b723f9/northd/0.log"
Jul 27 22:38:36.042841 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:38:36.042819    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-qs89h_7f66b2c1-260e-4a55-902b-c4a060b723f9/nbdb/0.log"
Jul 27 22:38:36.085958 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:38:36.085933    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-qs89h_7f66b2c1-260e-4a55-902b-c4a060b723f9/sbdb/0.log"
Jul 27 22:38:36.156194 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:38:36.156162    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-qs89h_7f66b2c1-260e-4a55-902b-c4a060b723f9/ovnkube-controller/0.log"
Jul 27 22:38:37.105775 ip-10-0-131-133 kubenswrapper[2559]: I0727 22:38:37.105748    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-diagnostics_network-check-target-hmjv6_95b2123e-132a-42a8-8edb-33ce19c2df51/network-check-target-container/0.log"