Aug 05 17:10:46.587517 ip-10-0-135-2 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory
Aug 05 17:10:46.587528 ip-10-0-135-2 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory
Aug 05 17:10:46.587534 ip-10-0-135-2 systemd[1]: kubelet.service: Failed with result 'resources'.
Aug 05 17:10:46.587796 ip-10-0-135-2 systemd[1]: Failed to start Kubernetes Kubelet.
Aug 05 17:10:56.764127 ip-10-0-135-2 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found.
Aug 05 17:10:56.764142 ip-10-0-135-2 systemd[1]: kubelet.service: Failed with result 'resources'.
-- Boot bd36aca672d642b0bbeedfd1b1d8be8f --
Aug 05 17:12:52.528626 ip-10-0-135-2 systemd[1]: Starting Kubernetes Kubelet...
Aug 05 17:12:52.875344 ip-10-0-135-2 kubenswrapper[2571]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Aug 05 17:12:52.875344 ip-10-0-135-2 kubenswrapper[2571]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Aug 05 17:12:52.875344 ip-10-0-135-2 kubenswrapper[2571]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Aug 05 17:12:52.875344 ip-10-0-135-2 kubenswrapper[2571]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Aug 05 17:12:52.875344 ip-10-0-135-2 kubenswrapper[2571]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Aug 05 17:12:52.876969 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:12:52.876896    2571 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Aug 05 17:12:52.881237 ip-10-0-135-2 kubenswrapper[2571]: W0805 17:12:52.881223    2571 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Aug 05 17:12:52.881237 ip-10-0-135-2 kubenswrapper[2571]: W0805 17:12:52.881237    2571 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Aug 05 17:12:52.881294 ip-10-0-135-2 kubenswrapper[2571]: W0805 17:12:52.881241    2571 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Aug 05 17:12:52.881294 ip-10-0-135-2 kubenswrapper[2571]: W0805 17:12:52.881244    2571 feature_gate.go:328] unrecognized feature gate: DualReplica
Aug 05 17:12:52.881294 ip-10-0-135-2 kubenswrapper[2571]: W0805 17:12:52.881247    2571 feature_gate.go:328] unrecognized feature gate: PinnedImages
Aug 05 17:12:52.881294 ip-10-0-135-2 kubenswrapper[2571]: W0805 17:12:52.881250    2571 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Aug 05 17:12:52.881294 ip-10-0-135-2 kubenswrapper[2571]: W0805 17:12:52.881252    2571 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Aug 05 17:12:52.881294 ip-10-0-135-2 kubenswrapper[2571]: W0805 17:12:52.881255    2571 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Aug 05 17:12:52.881294 ip-10-0-135-2 kubenswrapper[2571]: W0805 17:12:52.881258    2571 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Aug 05 17:12:52.881294 ip-10-0-135-2 kubenswrapper[2571]: W0805 17:12:52.881263    2571 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Aug 05 17:12:52.881294 ip-10-0-135-2 kubenswrapper[2571]: W0805 17:12:52.881267    2571 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Aug 05 17:12:52.881294 ip-10-0-135-2 kubenswrapper[2571]: W0805 17:12:52.881270    2571 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Aug 05 17:12:52.881294 ip-10-0-135-2 kubenswrapper[2571]: W0805 17:12:52.881273    2571 feature_gate.go:328] unrecognized feature gate: EventTTL
Aug 05 17:12:52.881294 ip-10-0-135-2 kubenswrapper[2571]: W0805 17:12:52.881276    2571 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Aug 05 17:12:52.881294 ip-10-0-135-2 kubenswrapper[2571]: W0805 17:12:52.881279    2571 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Aug 05 17:12:52.881294 ip-10-0-135-2 kubenswrapper[2571]: W0805 17:12:52.881282    2571 feature_gate.go:328] unrecognized feature gate: OSStreams
Aug 05 17:12:52.881294 ip-10-0-135-2 kubenswrapper[2571]: W0805 17:12:52.881285    2571 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Aug 05 17:12:52.881294 ip-10-0-135-2 kubenswrapper[2571]: W0805 17:12:52.881287    2571 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Aug 05 17:12:52.881294 ip-10-0-135-2 kubenswrapper[2571]: W0805 17:12:52.881290    2571 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Aug 05 17:12:52.881294 ip-10-0-135-2 kubenswrapper[2571]: W0805 17:12:52.881293    2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Aug 05 17:12:52.881294 ip-10-0-135-2 kubenswrapper[2571]: W0805 17:12:52.881296    2571 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Aug 05 17:12:52.881294 ip-10-0-135-2 kubenswrapper[2571]: W0805 17:12:52.881299    2571 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Aug 05 17:12:52.882057 ip-10-0-135-2 kubenswrapper[2571]: W0805 17:12:52.881301    2571 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Aug 05 17:12:52.882057 ip-10-0-135-2 kubenswrapper[2571]: W0805 17:12:52.881304    2571 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Aug 05 17:12:52.882057 ip-10-0-135-2 kubenswrapper[2571]: W0805 17:12:52.881308    2571 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Aug 05 17:12:52.882057 ip-10-0-135-2 kubenswrapper[2571]: W0805 17:12:52.881310    2571 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Aug 05 17:12:52.882057 ip-10-0-135-2 kubenswrapper[2571]: W0805 17:12:52.881313    2571 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Aug 05 17:12:52.882057 ip-10-0-135-2 kubenswrapper[2571]: W0805 17:12:52.881315    2571 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Aug 05 17:12:52.882057 ip-10-0-135-2 kubenswrapper[2571]: W0805 17:12:52.881318    2571 feature_gate.go:328] unrecognized feature gate: NewOLM
Aug 05 17:12:52.882057 ip-10-0-135-2 kubenswrapper[2571]: W0805 17:12:52.881320    2571 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Aug 05 17:12:52.882057 ip-10-0-135-2 kubenswrapper[2571]: W0805 17:12:52.881323    2571 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Aug 05 17:12:52.882057 ip-10-0-135-2 kubenswrapper[2571]: W0805 17:12:52.881325    2571 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Aug 05 17:12:52.882057 ip-10-0-135-2 kubenswrapper[2571]: W0805 17:12:52.881328    2571 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Aug 05 17:12:52.882057 ip-10-0-135-2 kubenswrapper[2571]: W0805 17:12:52.881331    2571 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Aug 05 17:12:52.882057 ip-10-0-135-2 kubenswrapper[2571]: W0805 17:12:52.881333    2571 feature_gate.go:328] unrecognized feature gate: SignatureStores
Aug 05 17:12:52.882057 ip-10-0-135-2 kubenswrapper[2571]: W0805 17:12:52.881337    2571 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Aug 05 17:12:52.882057 ip-10-0-135-2 kubenswrapper[2571]: W0805 17:12:52.881339    2571 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Aug 05 17:12:52.882057 ip-10-0-135-2 kubenswrapper[2571]: W0805 17:12:52.881342    2571 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Aug 05 17:12:52.882057 ip-10-0-135-2 kubenswrapper[2571]: W0805 17:12:52.881344    2571 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Aug 05 17:12:52.882057 ip-10-0-135-2 kubenswrapper[2571]: W0805 17:12:52.881347    2571 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Aug 05 17:12:52.882057 ip-10-0-135-2 kubenswrapper[2571]: W0805 17:12:52.881350    2571 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Aug 05 17:12:52.882057 ip-10-0-135-2 kubenswrapper[2571]: W0805 17:12:52.881353    2571 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Aug 05 17:12:52.882744 ip-10-0-135-2 kubenswrapper[2571]: W0805 17:12:52.881355    2571 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Aug 05 17:12:52.882744 ip-10-0-135-2 kubenswrapper[2571]: W0805 17:12:52.881358    2571 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Aug 05 17:12:52.882744 ip-10-0-135-2 kubenswrapper[2571]: W0805 17:12:52.881362    2571 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Aug 05 17:12:52.882744 ip-10-0-135-2 kubenswrapper[2571]: W0805 17:12:52.881365    2571 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Aug 05 17:12:52.882744 ip-10-0-135-2 kubenswrapper[2571]: W0805 17:12:52.881368    2571 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Aug 05 17:12:52.882744 ip-10-0-135-2 kubenswrapper[2571]: W0805 17:12:52.881384    2571 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Aug 05 17:12:52.882744 ip-10-0-135-2 kubenswrapper[2571]: W0805 17:12:52.881388    2571 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Aug 05 17:12:52.882744 ip-10-0-135-2 kubenswrapper[2571]: W0805 17:12:52.881390    2571 feature_gate.go:328] unrecognized feature gate: Example
Aug 05 17:12:52.882744 ip-10-0-135-2 kubenswrapper[2571]: W0805 17:12:52.881393    2571 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Aug 05 17:12:52.882744 ip-10-0-135-2 kubenswrapper[2571]: W0805 17:12:52.881395    2571 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Aug 05 17:12:52.882744 ip-10-0-135-2 kubenswrapper[2571]: W0805 17:12:52.881427    2571 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Aug 05 17:12:52.882744 ip-10-0-135-2 kubenswrapper[2571]: W0805 17:12:52.881464    2571 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Aug 05 17:12:52.882744 ip-10-0-135-2 kubenswrapper[2571]: W0805 17:12:52.881468    2571 feature_gate.go:328] unrecognized feature gate: OVNObservability
Aug 05 17:12:52.882744 ip-10-0-135-2 kubenswrapper[2571]: W0805 17:12:52.881472    2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Aug 05 17:12:52.882744 ip-10-0-135-2 kubenswrapper[2571]: W0805 17:12:52.881475    2571 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Aug 05 17:12:52.882744 ip-10-0-135-2 kubenswrapper[2571]: W0805 17:12:52.881479    2571 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Aug 05 17:12:52.882744 ip-10-0-135-2 kubenswrapper[2571]: W0805 17:12:52.881482    2571 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Aug 05 17:12:52.882744 ip-10-0-135-2 kubenswrapper[2571]: W0805 17:12:52.881485    2571 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Aug 05 17:12:52.882744 ip-10-0-135-2 kubenswrapper[2571]: W0805 17:12:52.881488    2571 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Aug 05 17:12:52.883257 ip-10-0-135-2 kubenswrapper[2571]: W0805 17:12:52.881491    2571 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Aug 05 17:12:52.883257 ip-10-0-135-2 kubenswrapper[2571]: W0805 17:12:52.881494    2571 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Aug 05 17:12:52.883257 ip-10-0-135-2 kubenswrapper[2571]: W0805 17:12:52.881497    2571 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Aug 05 17:12:52.883257 ip-10-0-135-2 kubenswrapper[2571]: W0805 17:12:52.881501    2571 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Aug 05 17:12:52.883257 ip-10-0-135-2 kubenswrapper[2571]: W0805 17:12:52.881507    2571 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Aug 05 17:12:52.883257 ip-10-0-135-2 kubenswrapper[2571]: W0805 17:12:52.881510    2571 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Aug 05 17:12:52.883257 ip-10-0-135-2 kubenswrapper[2571]: W0805 17:12:52.881514    2571 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Aug 05 17:12:52.883257 ip-10-0-135-2 kubenswrapper[2571]: W0805 17:12:52.881517    2571 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Aug 05 17:12:52.883257 ip-10-0-135-2 kubenswrapper[2571]: W0805 17:12:52.881520    2571 feature_gate.go:328] unrecognized feature gate: Example2
Aug 05 17:12:52.883257 ip-10-0-135-2 kubenswrapper[2571]: W0805 17:12:52.881523    2571 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Aug 05 17:12:52.883257 ip-10-0-135-2 kubenswrapper[2571]: W0805 17:12:52.881527    2571 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Aug 05 17:12:52.883257 ip-10-0-135-2 kubenswrapper[2571]: W0805 17:12:52.881532    2571 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Aug 05 17:12:52.883257 ip-10-0-135-2 kubenswrapper[2571]: W0805 17:12:52.881535    2571 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Aug 05 17:12:52.883257 ip-10-0-135-2 kubenswrapper[2571]: W0805 17:12:52.881538    2571 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Aug 05 17:12:52.883257 ip-10-0-135-2 kubenswrapper[2571]: W0805 17:12:52.881541    2571 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Aug 05 17:12:52.883257 ip-10-0-135-2 kubenswrapper[2571]: W0805 17:12:52.881544    2571 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Aug 05 17:12:52.883257 ip-10-0-135-2 kubenswrapper[2571]: W0805 17:12:52.881549    2571 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Aug 05 17:12:52.883257 ip-10-0-135-2 kubenswrapper[2571]: W0805 17:12:52.881552    2571 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Aug 05 17:12:52.883257 ip-10-0-135-2 kubenswrapper[2571]: W0805 17:12:52.881555    2571 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Aug 05 17:12:52.883257 ip-10-0-135-2 kubenswrapper[2571]: W0805 17:12:52.881558    2571 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Aug 05 17:12:52.884078 ip-10-0-135-2 kubenswrapper[2571]: W0805 17:12:52.881560    2571 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Aug 05 17:12:52.884078 ip-10-0-135-2 kubenswrapper[2571]: W0805 17:12:52.881565    2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Aug 05 17:12:52.884078 ip-10-0-135-2 kubenswrapper[2571]: W0805 17:12:52.881569    2571 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Aug 05 17:12:52.884078 ip-10-0-135-2 kubenswrapper[2571]: W0805 17:12:52.883206    2571 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Aug 05 17:12:52.884078 ip-10-0-135-2 kubenswrapper[2571]: W0805 17:12:52.883225    2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Aug 05 17:12:52.884078 ip-10-0-135-2 kubenswrapper[2571]: W0805 17:12:52.883232    2571 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Aug 05 17:12:52.884078 ip-10-0-135-2 kubenswrapper[2571]: W0805 17:12:52.883237    2571 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Aug 05 17:12:52.884078 ip-10-0-135-2 kubenswrapper[2571]: W0805 17:12:52.883242    2571 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Aug 05 17:12:52.884078 ip-10-0-135-2 kubenswrapper[2571]: W0805 17:12:52.883258    2571 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Aug 05 17:12:52.884078 ip-10-0-135-2 kubenswrapper[2571]: W0805 17:12:52.883400    2571 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Aug 05 17:12:52.884078 ip-10-0-135-2 kubenswrapper[2571]: W0805 17:12:52.883411    2571 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Aug 05 17:12:52.884078 ip-10-0-135-2 kubenswrapper[2571]: W0805 17:12:52.883416    2571 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Aug 05 17:12:52.884078 ip-10-0-135-2 kubenswrapper[2571]: W0805 17:12:52.883420    2571 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Aug 05 17:12:52.884078 ip-10-0-135-2 kubenswrapper[2571]: W0805 17:12:52.883424    2571 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Aug 05 17:12:52.884078 ip-10-0-135-2 kubenswrapper[2571]: W0805 17:12:52.883429    2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Aug 05 17:12:52.884078 ip-10-0-135-2 kubenswrapper[2571]: W0805 17:12:52.883433    2571 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Aug 05 17:12:52.884078 ip-10-0-135-2 kubenswrapper[2571]: W0805 17:12:52.883438    2571 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Aug 05 17:12:52.884078 ip-10-0-135-2 kubenswrapper[2571]: W0805 17:12:52.883442    2571 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Aug 05 17:12:52.884864 ip-10-0-135-2 kubenswrapper[2571]: W0805 17:12:52.884642    2571 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Aug 05 17:12:52.884864 ip-10-0-135-2 kubenswrapper[2571]: W0805 17:12:52.884653    2571 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Aug 05 17:12:52.884864 ip-10-0-135-2 kubenswrapper[2571]: W0805 17:12:52.884658    2571 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Aug 05 17:12:52.884864 ip-10-0-135-2 kubenswrapper[2571]: W0805 17:12:52.884662    2571 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Aug 05 17:12:52.884864 ip-10-0-135-2 kubenswrapper[2571]: W0805 17:12:52.884667    2571 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Aug 05 17:12:52.884864 ip-10-0-135-2 kubenswrapper[2571]: W0805 17:12:52.884672    2571 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Aug 05 17:12:52.884864 ip-10-0-135-2 kubenswrapper[2571]: W0805 17:12:52.884677    2571 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Aug 05 17:12:52.884864 ip-10-0-135-2 kubenswrapper[2571]: W0805 17:12:52.884681    2571 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Aug 05 17:12:52.884864 ip-10-0-135-2 kubenswrapper[2571]: W0805 17:12:52.884685    2571 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Aug 05 17:12:52.884864 ip-10-0-135-2 kubenswrapper[2571]: W0805 17:12:52.884689    2571 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Aug 05 17:12:52.884864 ip-10-0-135-2 kubenswrapper[2571]: W0805 17:12:52.884694    2571 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Aug 05 17:12:52.884864 ip-10-0-135-2 kubenswrapper[2571]: W0805 17:12:52.884697    2571 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Aug 05 17:12:52.884864 ip-10-0-135-2 kubenswrapper[2571]: W0805 17:12:52.884701    2571 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Aug 05 17:12:52.884864 ip-10-0-135-2 kubenswrapper[2571]: W0805 17:12:52.884706    2571 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Aug 05 17:12:52.884864 ip-10-0-135-2 kubenswrapper[2571]: W0805 17:12:52.884710    2571 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Aug 05 17:12:52.884864 ip-10-0-135-2 kubenswrapper[2571]: W0805 17:12:52.884714    2571 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Aug 05 17:12:52.884864 ip-10-0-135-2 kubenswrapper[2571]: W0805 17:12:52.884718    2571 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Aug 05 17:12:52.884864 ip-10-0-135-2 kubenswrapper[2571]: W0805 17:12:52.884722    2571 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Aug 05 17:12:52.884864 ip-10-0-135-2 kubenswrapper[2571]: W0805 17:12:52.884730    2571 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Aug 05 17:12:52.885699 ip-10-0-135-2 kubenswrapper[2571]: W0805 17:12:52.884737    2571 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Aug 05 17:12:52.885699 ip-10-0-135-2 kubenswrapper[2571]: W0805 17:12:52.884741    2571 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Aug 05 17:12:52.885699 ip-10-0-135-2 kubenswrapper[2571]: W0805 17:12:52.884746    2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Aug 05 17:12:52.885699 ip-10-0-135-2 kubenswrapper[2571]: W0805 17:12:52.884750    2571 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Aug 05 17:12:52.885699 ip-10-0-135-2 kubenswrapper[2571]: W0805 17:12:52.884754    2571 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Aug 05 17:12:52.885699 ip-10-0-135-2 kubenswrapper[2571]: W0805 17:12:52.884758    2571 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Aug 05 17:12:52.885699 ip-10-0-135-2 kubenswrapper[2571]: W0805 17:12:52.884762    2571 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Aug 05 17:12:52.885699 ip-10-0-135-2 kubenswrapper[2571]: W0805 17:12:52.884766    2571 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Aug 05 17:12:52.885699 ip-10-0-135-2 kubenswrapper[2571]: W0805 17:12:52.884770    2571 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Aug 05 17:12:52.885699 ip-10-0-135-2 kubenswrapper[2571]: W0805 17:12:52.884775    2571 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Aug 05 17:12:52.885699 ip-10-0-135-2 kubenswrapper[2571]: W0805 17:12:52.884780    2571 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Aug 05 17:12:52.885699 ip-10-0-135-2 kubenswrapper[2571]: W0805 17:12:52.884784    2571 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Aug 05 17:12:52.885699 ip-10-0-135-2 kubenswrapper[2571]: W0805 17:12:52.884788    2571 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Aug 05 17:12:52.885699 ip-10-0-135-2 kubenswrapper[2571]: W0805 17:12:52.884794    2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Aug 05 17:12:52.885699 ip-10-0-135-2 kubenswrapper[2571]: W0805 17:12:52.884798    2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Aug 05 17:12:52.885699 ip-10-0-135-2 kubenswrapper[2571]: W0805 17:12:52.884803    2571 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Aug 05 17:12:52.885699 ip-10-0-135-2 kubenswrapper[2571]: W0805 17:12:52.884807    2571 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Aug 05 17:12:52.885699 ip-10-0-135-2 kubenswrapper[2571]: W0805 17:12:52.884811    2571 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Aug 05 17:12:52.885699 ip-10-0-135-2 kubenswrapper[2571]: W0805 17:12:52.884815    2571 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Aug 05 17:12:52.885699 ip-10-0-135-2 kubenswrapper[2571]: W0805 17:12:52.884820    2571 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Aug 05 17:12:52.886591 ip-10-0-135-2 kubenswrapper[2571]: W0805 17:12:52.884824    2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Aug 05 17:12:52.886591 ip-10-0-135-2 kubenswrapper[2571]: W0805 17:12:52.884829    2571 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Aug 05 17:12:52.886591 ip-10-0-135-2 kubenswrapper[2571]: W0805 17:12:52.884833    2571 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Aug 05 17:12:52.886591 ip-10-0-135-2 kubenswrapper[2571]: W0805 17:12:52.884837    2571 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Aug 05 17:12:52.886591 ip-10-0-135-2 kubenswrapper[2571]: W0805 17:12:52.884842    2571 feature_gate.go:328] unrecognized feature gate: SignatureStores
Aug 05 17:12:52.886591 ip-10-0-135-2 kubenswrapper[2571]: W0805 17:12:52.884849    2571 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Aug 05 17:12:52.886591 ip-10-0-135-2 kubenswrapper[2571]: W0805 17:12:52.884855    2571 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Aug 05 17:12:52.886591 ip-10-0-135-2 kubenswrapper[2571]: W0805 17:12:52.884859    2571 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Aug 05 17:12:52.886591 ip-10-0-135-2 kubenswrapper[2571]: W0805 17:12:52.884863    2571 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Aug 05 17:12:52.886591 ip-10-0-135-2 kubenswrapper[2571]: W0805 17:12:52.884867    2571 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Aug 05 17:12:52.886591 ip-10-0-135-2 kubenswrapper[2571]: W0805 17:12:52.884871    2571 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Aug 05 17:12:52.886591 ip-10-0-135-2 kubenswrapper[2571]: W0805 17:12:52.884875    2571 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Aug 05 17:12:52.886591 ip-10-0-135-2 kubenswrapper[2571]: W0805 17:12:52.884879    2571 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Aug 05 17:12:52.886591 ip-10-0-135-2 kubenswrapper[2571]: W0805 17:12:52.884883    2571 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Aug 05 17:12:52.886591 ip-10-0-135-2 kubenswrapper[2571]: W0805 17:12:52.884887    2571 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Aug 05 17:12:52.886591 ip-10-0-135-2 kubenswrapper[2571]: W0805 17:12:52.884891    2571 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Aug 05 17:12:52.886591 ip-10-0-135-2 kubenswrapper[2571]: W0805 17:12:52.884895    2571 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Aug 05 17:12:52.886591 ip-10-0-135-2 kubenswrapper[2571]: W0805 17:12:52.884900    2571 feature_gate.go:328] unrecognized feature gate: PinnedImages
Aug 05 17:12:52.886591 ip-10-0-135-2 kubenswrapper[2571]: W0805 17:12:52.884904    2571 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Aug 05 17:12:52.886591 ip-10-0-135-2 kubenswrapper[2571]: W0805 17:12:52.884908    2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Aug 05 17:12:52.887476 ip-10-0-135-2 kubenswrapper[2571]: W0805 17:12:52.884912    2571 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Aug 05 17:12:52.887476 ip-10-0-135-2 kubenswrapper[2571]: W0805 17:12:52.884916    2571 feature_gate.go:328] unrecognized feature gate: OSStreams
Aug 05 17:12:52.887476 ip-10-0-135-2 kubenswrapper[2571]: W0805 17:12:52.884921    2571 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Aug 05 17:12:52.887476 ip-10-0-135-2 kubenswrapper[2571]: W0805 17:12:52.884925    2571 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Aug 05 17:12:52.887476 ip-10-0-135-2 kubenswrapper[2571]: W0805 17:12:52.884929    2571 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Aug 05 17:12:52.887476 ip-10-0-135-2 kubenswrapper[2571]: W0805 17:12:52.884935    2571 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Aug 05 17:12:52.887476 ip-10-0-135-2 kubenswrapper[2571]: W0805 17:12:52.884939    2571 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Aug 05 17:12:52.887476 ip-10-0-135-2 kubenswrapper[2571]: W0805 17:12:52.884943    2571 feature_gate.go:328] unrecognized feature gate: DualReplica
Aug 05 17:12:52.887476 ip-10-0-135-2 kubenswrapper[2571]: W0805 17:12:52.884947    2571 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Aug 05 17:12:52.887476 ip-10-0-135-2 kubenswrapper[2571]: W0805 17:12:52.884952    2571 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Aug 05 17:12:52.887476 ip-10-0-135-2 kubenswrapper[2571]: W0805 17:12:52.884956    2571 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Aug 05 17:12:52.887476 ip-10-0-135-2 kubenswrapper[2571]: W0805 17:12:52.884960    2571 feature_gate.go:328] unrecognized feature gate: EventTTL
Aug 05 17:12:52.887476 ip-10-0-135-2 kubenswrapper[2571]: W0805 17:12:52.884965    2571 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Aug 05 17:12:52.887476 ip-10-0-135-2 kubenswrapper[2571]: W0805 17:12:52.884969    2571 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Aug 05 17:12:52.887476 ip-10-0-135-2 kubenswrapper[2571]: W0805 17:12:52.884973    2571 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Aug 05 17:12:52.887476 ip-10-0-135-2 kubenswrapper[2571]: W0805 17:12:52.884977    2571 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Aug 05 17:12:52.887476 ip-10-0-135-2 kubenswrapper[2571]: W0805 17:12:52.884981    2571 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Aug 05 17:12:52.887476 ip-10-0-135-2 kubenswrapper[2571]: W0805 17:12:52.884986    2571 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Aug 05 17:12:52.887476 ip-10-0-135-2 kubenswrapper[2571]: W0805 17:12:52.884990    2571 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Aug 05 17:12:52.887476 ip-10-0-135-2 kubenswrapper[2571]: W0805 17:12:52.884994    2571 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Aug 05 17:12:52.888102 ip-10-0-135-2 kubenswrapper[2571]: W0805 17:12:52.884998    2571 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Aug 05 17:12:52.888102 ip-10-0-135-2 kubenswrapper[2571]: W0805 17:12:52.885002    2571 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Aug 05 17:12:52.888102 ip-10-0-135-2 kubenswrapper[2571]: W0805 17:12:52.885007    2571 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Aug 05 17:12:52.888102 ip-10-0-135-2 kubenswrapper[2571]: W0805 17:12:52.885011    2571 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Aug 05 17:12:52.888102 ip-10-0-135-2 kubenswrapper[2571]: W0805 17:12:52.885015    2571 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Aug 05 17:12:52.888102 ip-10-0-135-2 kubenswrapper[2571]: W0805 17:12:52.885019    2571 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Aug 05 17:12:52.888102 ip-10-0-135-2 kubenswrapper[2571]: W0805 17:12:52.885023    2571 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Aug 05 17:12:52.888102 ip-10-0-135-2 kubenswrapper[2571]: W0805 17:12:52.885028    2571 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Aug 05 17:12:52.888102 ip-10-0-135-2 kubenswrapper[2571]: W0805 17:12:52.885032    2571 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Aug 05 17:12:52.888102 ip-10-0-135-2 kubenswrapper[2571]: W0805 17:12:52.885036    2571 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Aug 05 17:12:52.888102 ip-10-0-135-2 kubenswrapper[2571]: W0805 17:12:52.885040    2571 feature_gate.go:328] unrecognized feature gate: Example
Aug 05 17:12:52.888102 ip-10-0-135-2 kubenswrapper[2571]: W0805 17:12:52.885046    2571 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Aug 05 17:12:52.888102 ip-10-0-135-2 kubenswrapper[2571]: W0805 17:12:52.885050    2571 feature_gate.go:328] unrecognized feature gate: OVNObservability
Aug 05 17:12:52.888102 ip-10-0-135-2 kubenswrapper[2571]: W0805 17:12:52.885054    2571 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Aug 05 17:12:52.888102 ip-10-0-135-2 kubenswrapper[2571]: W0805 17:12:52.885059    2571 feature_gate.go:328] unrecognized feature gate: Example2
Aug 05 17:12:52.888102 ip-10-0-135-2 kubenswrapper[2571]: W0805 17:12:52.885063    2571 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Aug 05 17:12:52.888102 ip-10-0-135-2 kubenswrapper[2571]: W0805 17:12:52.885070    2571 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Aug 05 17:12:52.888102 ip-10-0-135-2 kubenswrapper[2571]: W0805 17:12:52.885077    2571 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Aug 05 17:12:52.888102 ip-10-0-135-2 kubenswrapper[2571]: W0805 17:12:52.885082    2571 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Aug 05 17:12:52.888678 ip-10-0-135-2 kubenswrapper[2571]: W0805 17:12:52.885087    2571 feature_gate.go:328] unrecognized feature gate: NewOLM
Aug 05 17:12:52.888678 ip-10-0-135-2 kubenswrapper[2571]: W0805 17:12:52.885686    2571 feature_gate.go:328] unrecognized feature gate: OSStreams
Aug 05 17:12:52.888678 ip-10-0-135-2 kubenswrapper[2571]: W0805 17:12:52.885708    2571 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Aug 05 17:12:52.888678 ip-10-0-135-2 kubenswrapper[2571]: W0805 17:12:52.885715    2571 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Aug 05 17:12:52.888678 ip-10-0-135-2 kubenswrapper[2571]: W0805 17:12:52.885720    2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Aug 05 17:12:52.888678 ip-10-0-135-2 kubenswrapper[2571]: W0805 17:12:52.885724    2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Aug 05 17:12:52.888678 ip-10-0-135-2 kubenswrapper[2571]: W0805 17:12:52.885728    2571 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Aug 05 17:12:52.888678 ip-10-0-135-2 kubenswrapper[2571]: W0805 17:12:52.885733    2571 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Aug 05 17:12:52.888678 ip-10-0-135-2 kubenswrapper[2571]: W0805 17:12:52.885737    2571 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Aug 05 17:12:52.888678 ip-10-0-135-2 kubenswrapper[2571]: W0805 17:12:52.885741    2571 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Aug 05 17:12:52.888678 ip-10-0-135-2 kubenswrapper[2571]: W0805 17:12:52.885746    2571 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Aug 05 17:12:52.888678 ip-10-0-135-2 kubenswrapper[2571]: W0805 17:12:52.885750    2571 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Aug 05 17:12:52.888678 ip-10-0-135-2 kubenswrapper[2571]: W0805 17:12:52.885754    2571 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Aug 05 17:12:52.888678 ip-10-0-135-2 kubenswrapper[2571]: W0805 17:12:52.885758    2571 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Aug 05 17:12:52.888678 ip-10-0-135-2 kubenswrapper[2571]: W0805 17:12:52.885762    2571 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Aug 05 17:12:52.888678 ip-10-0-135-2 kubenswrapper[2571]: W0805 17:12:52.885766    2571 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Aug 05 17:12:52.888678 ip-10-0-135-2 kubenswrapper[2571]: W0805 17:12:52.885772    2571 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Aug 05 17:12:52.888678 ip-10-0-135-2 kubenswrapper[2571]: W0805 17:12:52.885776    2571 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Aug 05 17:12:52.888678 ip-10-0-135-2 kubenswrapper[2571]: W0805 17:12:52.885780    2571 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Aug 05 17:12:52.888678 ip-10-0-135-2 kubenswrapper[2571]: W0805 17:12:52.885784    2571 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Aug 05 17:12:52.889326 ip-10-0-135-2 kubenswrapper[2571]: W0805 17:12:52.885789    2571 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Aug 05 17:12:52.889326 ip-10-0-135-2 kubenswrapper[2571]: W0805 17:12:52.885793    2571 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Aug 05 17:12:52.889326 ip-10-0-135-2 kubenswrapper[2571]: W0805 17:12:52.885797    2571 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Aug 05 17:12:52.889326 ip-10-0-135-2 kubenswrapper[2571]: W0805 17:12:52.885801    2571 feature_gate.go:328] unrecognized feature gate: SignatureStores
Aug 05 17:12:52.889326 ip-10-0-135-2 kubenswrapper[2571]: W0805 17:12:52.885805    2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Aug 05 17:12:52.889326 ip-10-0-135-2 kubenswrapper[2571]: W0805 17:12:52.885810    2571 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Aug 05 17:12:52.889326 ip-10-0-135-2 kubenswrapper[2571]: W0805 17:12:52.885814    2571 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Aug 05 17:12:52.889326 ip-10-0-135-2 kubenswrapper[2571]: W0805 17:12:52.885819    2571 feature_gate.go:328] unrecognized feature gate: DualReplica
Aug 05 17:12:52.889326 ip-10-0-135-2 kubenswrapper[2571]: W0805 17:12:52.885823    2571 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Aug 05 17:12:52.889326 ip-10-0-135-2 kubenswrapper[2571]: W0805 17:12:52.885828    2571 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Aug 05 17:12:52.889326 ip-10-0-135-2 kubenswrapper[2571]: W0805 17:12:52.885833    2571 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Aug 05 17:12:52.889326 ip-10-0-135-2 kubenswrapper[2571]: W0805 17:12:52.885837    2571 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Aug 05 17:12:52.889326 ip-10-0-135-2 kubenswrapper[2571]: W0805 17:12:52.885841    2571 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Aug 05 17:12:52.889326 ip-10-0-135-2 kubenswrapper[2571]: W0805 17:12:52.885845    2571 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Aug 05 17:12:52.889326 ip-10-0-135-2 kubenswrapper[2571]: W0805 17:12:52.885849    2571 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Aug 05 17:12:52.889326 ip-10-0-135-2 kubenswrapper[2571]: W0805 17:12:52.885854    2571 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Aug 05 17:12:52.889326 ip-10-0-135-2 kubenswrapper[2571]: W0805 17:12:52.885858    2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Aug 05 17:12:52.889326 ip-10-0-135-2 kubenswrapper[2571]: W0805 17:12:52.885863    2571 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Aug 05 17:12:52.889326 ip-10-0-135-2 kubenswrapper[2571]: W0805 17:12:52.885867    2571 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Aug 05 17:12:52.889326 ip-10-0-135-2 kubenswrapper[2571]: W0805 17:12:52.885871    2571 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Aug 05 17:12:52.889997 ip-10-0-135-2 kubenswrapper[2571]: W0805 17:12:52.885875    2571 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Aug 05 17:12:52.889997 ip-10-0-135-2 kubenswrapper[2571]: W0805 17:12:52.885879    2571 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Aug 05 17:12:52.889997 ip-10-0-135-2 kubenswrapper[2571]: W0805 17:12:52.885884    2571 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Aug 05 17:12:52.889997 ip-10-0-135-2 kubenswrapper[2571]: W0805 17:12:52.885888    2571 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Aug 05 17:12:52.889997 ip-10-0-135-2 kubenswrapper[2571]: W0805 17:12:52.885892    2571 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Aug 05 17:12:52.889997 ip-10-0-135-2 kubenswrapper[2571]: W0805 17:12:52.885896    2571 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Aug 05 17:12:52.889997 ip-10-0-135-2 kubenswrapper[2571]: W0805 17:12:52.885900    2571 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Aug 05 17:12:52.889997 ip-10-0-135-2 kubenswrapper[2571]: W0805 17:12:52.885904    2571 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Aug 05 17:12:52.889997 ip-10-0-135-2 kubenswrapper[2571]: W0805 17:12:52.885908    2571 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Aug 05 17:12:52.889997 ip-10-0-135-2 kubenswrapper[2571]: W0805 17:12:52.885912    2571 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Aug 05 17:12:52.889997 ip-10-0-135-2 kubenswrapper[2571]: W0805 17:12:52.885917    2571 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Aug 05 17:12:52.889997 ip-10-0-135-2 kubenswrapper[2571]: W0805 17:12:52.885921    2571 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Aug 05 17:12:52.889997 ip-10-0-135-2 kubenswrapper[2571]: W0805 17:12:52.885925    2571 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Aug 05 17:12:52.889997 ip-10-0-135-2 kubenswrapper[2571]: W0805 17:12:52.885930    2571 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Aug 05 17:12:52.889997 ip-10-0-135-2 kubenswrapper[2571]: W0805 17:12:52.885944    2571 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Aug 05 17:12:52.889997 ip-10-0-135-2 kubenswrapper[2571]: W0805 17:12:52.885950    2571 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Aug 05 17:12:52.889997 ip-10-0-135-2 kubenswrapper[2571]: W0805 17:12:52.885955    2571 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Aug 05 17:12:52.889997 ip-10-0-135-2 kubenswrapper[2571]: W0805 17:12:52.885960    2571 feature_gate.go:328] unrecognized feature gate: Example
Aug 05 17:12:52.889997 ip-10-0-135-2 kubenswrapper[2571]: W0805 17:12:52.885964    2571 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Aug 05 17:12:52.890586 ip-10-0-135-2 kubenswrapper[2571]: W0805 17:12:52.885971    2571 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Aug 05 17:12:52.890586 ip-10-0-135-2 kubenswrapper[2571]: W0805 17:12:52.885977    2571 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Aug 05 17:12:52.890586 ip-10-0-135-2 kubenswrapper[2571]: W0805 17:12:52.885982    2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Aug 05 17:12:52.890586 ip-10-0-135-2 kubenswrapper[2571]: W0805 17:12:52.885987    2571 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Aug 05 17:12:52.890586 ip-10-0-135-2 kubenswrapper[2571]: W0805 17:12:52.885991    2571 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Aug 05 17:12:52.890586 ip-10-0-135-2 kubenswrapper[2571]: W0805 17:12:52.885996    2571 feature_gate.go:328] unrecognized feature gate: Example2
Aug 05 17:12:52.890586 ip-10-0-135-2 kubenswrapper[2571]: W0805 17:12:52.886000    2571 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Aug 05 17:12:52.890586 ip-10-0-135-2 kubenswrapper[2571]: W0805 17:12:52.886004    2571 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Aug 05 17:12:52.890586 ip-10-0-135-2 kubenswrapper[2571]: W0805 17:12:52.886010    2571 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Aug 05 17:12:52.890586 ip-10-0-135-2 kubenswrapper[2571]: W0805 17:12:52.886015    2571 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Aug 05 17:12:52.890586 ip-10-0-135-2 kubenswrapper[2571]: W0805 17:12:52.886021    2571 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Aug 05 17:12:52.890586 ip-10-0-135-2 kubenswrapper[2571]: W0805 17:12:52.886025    2571 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Aug 05 17:12:52.890586 ip-10-0-135-2 kubenswrapper[2571]: W0805 17:12:52.886029    2571 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Aug 05 17:12:52.890586 ip-10-0-135-2 kubenswrapper[2571]: W0805 17:12:52.886034    2571 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Aug 05 17:12:52.890586 ip-10-0-135-2 kubenswrapper[2571]: W0805 17:12:52.886038    2571 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Aug 05 17:12:52.890586 ip-10-0-135-2 kubenswrapper[2571]: W0805 17:12:52.886043    2571 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Aug 05 17:12:52.890586 ip-10-0-135-2 kubenswrapper[2571]: W0805 17:12:52.886047    2571 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Aug 05 17:12:52.890586 ip-10-0-135-2 kubenswrapper[2571]: W0805 17:12:52.886051    2571 feature_gate.go:328] unrecognized feature gate: NewOLM
Aug 05 17:12:52.890586 ip-10-0-135-2 kubenswrapper[2571]: W0805 17:12:52.886055    2571 feature_gate.go:328] unrecognized feature gate: PinnedImages
Aug 05 17:12:52.891186 ip-10-0-135-2 kubenswrapper[2571]: W0805 17:12:52.886059    2571 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Aug 05 17:12:52.891186 ip-10-0-135-2 kubenswrapper[2571]: W0805 17:12:52.886063    2571 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Aug 05 17:12:52.891186 ip-10-0-135-2 kubenswrapper[2571]: W0805 17:12:52.886067    2571 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Aug 05 17:12:52.891186 ip-10-0-135-2 kubenswrapper[2571]: W0805 17:12:52.886071    2571 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Aug 05 17:12:52.891186 ip-10-0-135-2 kubenswrapper[2571]: W0805 17:12:52.886076    2571 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Aug 05 17:12:52.891186 ip-10-0-135-2 kubenswrapper[2571]: W0805 17:12:52.886080    2571 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Aug 05 17:12:52.891186 ip-10-0-135-2 kubenswrapper[2571]: W0805 17:12:52.886084    2571 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Aug 05 17:12:52.891186 ip-10-0-135-2 kubenswrapper[2571]: W0805 17:12:52.886088    2571 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Aug 05 17:12:52.891186 ip-10-0-135-2 kubenswrapper[2571]: W0805 17:12:52.886093    2571 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Aug 05 17:12:52.891186 ip-10-0-135-2 kubenswrapper[2571]: W0805 17:12:52.886097    2571 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Aug 05 17:12:52.891186 ip-10-0-135-2 kubenswrapper[2571]: W0805 17:12:52.886101    2571 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Aug 05 17:12:52.891186 ip-10-0-135-2 kubenswrapper[2571]: W0805 17:12:52.886105    2571 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Aug 05 17:12:52.891186 ip-10-0-135-2 kubenswrapper[2571]: W0805 17:12:52.886110    2571 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Aug 05 17:12:52.891186 ip-10-0-135-2 kubenswrapper[2571]: W0805 17:12:52.886115    2571 feature_gate.go:328] unrecognized feature gate: EventTTL
Aug 05 17:12:52.891186 ip-10-0-135-2 kubenswrapper[2571]: W0805 17:12:52.886120    2571 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Aug 05 17:12:52.891186 ip-10-0-135-2 kubenswrapper[2571]: W0805 17:12:52.886125    2571 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Aug 05 17:12:52.891186 ip-10-0-135-2 kubenswrapper[2571]: W0805 17:12:52.886130    2571 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Aug 05 17:12:52.891186 ip-10-0-135-2 kubenswrapper[2571]: W0805 17:12:52.886134    2571 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Aug 05 17:12:52.891186 ip-10-0-135-2 kubenswrapper[2571]: W0805 17:12:52.886138    2571 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Aug 05 17:12:52.891186 ip-10-0-135-2 kubenswrapper[2571]: W0805 17:12:52.886143    2571 feature_gate.go:328] unrecognized feature gate: OVNObservability
Aug 05 17:12:52.891707 ip-10-0-135-2 kubenswrapper[2571]: W0805 17:12:52.886147    2571 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Aug 05 17:12:52.891707 ip-10-0-135-2 kubenswrapper[2571]: W0805 17:12:52.886151    2571 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Aug 05 17:12:52.891707 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:12:52.886246    2571 flags.go:64] FLAG: --address="0.0.0.0"
Aug 05 17:12:52.891707 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:12:52.886256    2571 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Aug 05 17:12:52.891707 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:12:52.886265    2571 flags.go:64] FLAG: --anonymous-auth="true"
Aug 05 17:12:52.891707 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:12:52.886272    2571 flags.go:64] FLAG: --application-metrics-count-limit="100"
Aug 05 17:12:52.891707 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:12:52.886278    2571 flags.go:64] FLAG: --authentication-token-webhook="false"
Aug 05 17:12:52.891707 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:12:52.886284    2571 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Aug 05 17:12:52.891707 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:12:52.886290    2571 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Aug 05 17:12:52.891707 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:12:52.886296    2571 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Aug 05 17:12:52.891707 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:12:52.886302    2571 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Aug 05 17:12:52.891707 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:12:52.886306    2571 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Aug 05 17:12:52.891707 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:12:52.886312    2571 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Aug 05 17:12:52.891707 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:12:52.886316    2571 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Aug 05 17:12:52.891707 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:12:52.886321    2571 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Aug 05 17:12:52.891707 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:12:52.886326    2571 flags.go:64] FLAG: --cgroup-root=""
Aug 05 17:12:52.891707 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:12:52.886331    2571 flags.go:64] FLAG: --cgroups-per-qos="true"
Aug 05 17:12:52.891707 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:12:52.886335    2571 flags.go:64] FLAG: --client-ca-file=""
Aug 05 17:12:52.891707 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:12:52.886340    2571 flags.go:64] FLAG: --cloud-provider="external"
Aug 05 17:12:52.891707 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:12:52.886344    2571 flags.go:64] FLAG: --cluster-dns="[]"
Aug 05 17:12:52.891707 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:12:52.886353    2571 flags.go:64] FLAG: --cluster-domain=""
Aug 05 17:12:52.891707 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:12:52.886357    2571 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Aug 05 17:12:52.891707 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:12:52.886362    2571 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Aug 05 17:12:52.891707 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:12:52.886367    2571 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Aug 05 17:12:52.892324 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:12:52.886388    2571 flags.go:64] FLAG: --container-log-max-files="5"
Aug 05 17:12:52.892324 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:12:52.886395    2571 flags.go:64] FLAG: --container-log-max-size="10Mi"
Aug 05 17:12:52.892324 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:12:52.886403    2571 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Aug 05 17:12:52.892324 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:12:52.886408    2571 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Aug 05 17:12:52.892324 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:12:52.886413    2571 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Aug 05 17:12:52.892324 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:12:52.886417    2571 flags.go:64] FLAG: --contention-profiling="false"
Aug 05 17:12:52.892324 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:12:52.886422    2571 flags.go:64] FLAG: --cpu-cfs-quota="true"
Aug 05 17:12:52.892324 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:12:52.886427    2571 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Aug 05 17:12:52.892324 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:12:52.886432    2571 flags.go:64] FLAG: --cpu-manager-policy="none"
Aug 05 17:12:52.892324 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:12:52.886437    2571 flags.go:64] FLAG: --cpu-manager-policy-options=""
Aug 05 17:12:52.892324 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:12:52.886446    2571 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Aug 05 17:12:52.892324 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:12:52.886451    2571 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Aug 05 17:12:52.892324 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:12:52.886456    2571 flags.go:64] FLAG: --enable-debugging-handlers="true"
Aug 05 17:12:52.892324 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:12:52.886462    2571 flags.go:64] FLAG: --enable-load-reader="false"
Aug 05 17:12:52.892324 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:12:52.886467    2571 flags.go:64] FLAG: --enable-server="true"
Aug 05 17:12:52.892324 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:12:52.886471    2571 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Aug 05 17:12:52.892324 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:12:52.886478    2571 flags.go:64] FLAG: --event-burst="100"
Aug 05 17:12:52.892324 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:12:52.886483    2571 flags.go:64] FLAG: --event-qps="50"
Aug 05 17:12:52.892324 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:12:52.886488    2571 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Aug 05 17:12:52.892324 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:12:52.886492    2571 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Aug 05 17:12:52.892324 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:12:52.886497    2571 flags.go:64] FLAG: --eviction-hard=""
Aug 05 17:12:52.892324 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:12:52.886503    2571 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Aug 05 17:12:52.892324 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:12:52.886507    2571 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Aug 05 17:12:52.892324 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:12:52.886513    2571 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Aug 05 17:12:52.892324 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:12:52.886517    2571 flags.go:64] FLAG: --eviction-soft=""
Aug 05 17:12:52.892969 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:12:52.886522    2571 flags.go:64] FLAG: --eviction-soft-grace-period=""
Aug 05 17:12:52.892969 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:12:52.886527    2571 flags.go:64] FLAG: --exit-on-lock-contention="false"
Aug 05 17:12:52.892969 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:12:52.886531    2571 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Aug 05 17:12:52.892969 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:12:52.886536    2571 flags.go:64] FLAG: --experimental-mounter-path=""
Aug 05 17:12:52.892969 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:12:52.886541    2571 flags.go:64] FLAG: --fail-cgroupv1="false"
Aug 05 17:12:52.892969 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:12:52.886546    2571 flags.go:64] FLAG: --fail-swap-on="true"
Aug 05 17:12:52.892969 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:12:52.886550    2571 flags.go:64] FLAG: --feature-gates=""
Aug 05 17:12:52.892969 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:12:52.886556    2571 flags.go:64] FLAG: --file-check-frequency="20s"
Aug 05 17:12:52.892969 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:12:52.886562    2571 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Aug 05 17:12:52.892969 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:12:52.886569    2571 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Aug 05 17:12:52.892969 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:12:52.886574    2571 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Aug 05 17:12:52.892969 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:12:52.886579    2571 flags.go:64] FLAG: --healthz-port="10248"
Aug 05 17:12:52.892969 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:12:52.886584    2571 flags.go:64] FLAG: --help="false"
Aug 05 17:12:52.892969 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:12:52.886588    2571 flags.go:64] FLAG: --hostname-override="ip-10-0-135-2.ec2.internal"
Aug 05 17:12:52.892969 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:12:52.886593    2571 flags.go:64] FLAG: --housekeeping-interval="10s"
Aug 05 17:12:52.892969 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:12:52.886597    2571 flags.go:64] FLAG: --http-check-frequency="20s"
Aug 05 17:12:52.892969 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:12:52.886602    2571 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Aug 05 17:12:52.892969 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:12:52.886607    2571 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Aug 05 17:12:52.892969 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:12:52.886613    2571 flags.go:64] FLAG: --image-gc-high-threshold="85"
Aug 05 17:12:52.892969 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:12:52.886618    2571 flags.go:64] FLAG: --image-gc-low-threshold="80"
Aug 05 17:12:52.892969 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:12:52.886623    2571 flags.go:64] FLAG: --image-service-endpoint=""
Aug 05 17:12:52.892969 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:12:52.886628    2571 flags.go:64] FLAG: --kernel-memcg-notification="false"
Aug 05 17:12:52.892969 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:12:52.886634    2571 flags.go:64] FLAG: --kube-api-burst="100"
Aug 05 17:12:52.893585 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:12:52.886638    2571 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Aug 05 17:12:52.893585 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:12:52.886644    2571 flags.go:64] FLAG: --kube-api-qps="50"
Aug 05 17:12:52.893585 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:12:52.886650    2571 flags.go:64] FLAG: --kube-reserved=""
Aug 05 17:12:52.893585 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:12:52.886655    2571 flags.go:64] FLAG: --kube-reserved-cgroup=""
Aug 05 17:12:52.893585 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:12:52.886660    2571 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Aug 05 17:12:52.893585 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:12:52.886664    2571 flags.go:64] FLAG: --kubelet-cgroups=""
Aug 05 17:12:52.893585 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:12:52.886669    2571 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Aug 05 17:12:52.893585 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:12:52.886673    2571 flags.go:64] FLAG: --lock-file=""
Aug 05 17:12:52.893585 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:12:52.886678    2571 flags.go:64] FLAG: --log-cadvisor-usage="false"
Aug 05 17:12:52.893585 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:12:52.886683    2571 flags.go:64] FLAG: --log-flush-frequency="5s"
Aug 05 17:12:52.893585 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:12:52.886687    2571 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Aug 05 17:12:52.893585 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:12:52.886696    2571 flags.go:64] FLAG: --log-json-split-stream="false"
Aug 05 17:12:52.893585 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:12:52.886700    2571 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Aug 05 17:12:52.893585 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:12:52.886706    2571 flags.go:64] FLAG: --log-text-split-stream="false"
Aug 05 17:12:52.893585 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:12:52.886711    2571 flags.go:64] FLAG: --logging-format="text"
Aug 05 17:12:52.893585 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:12:52.886715    2571 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Aug 05 17:12:52.893585 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:12:52.886721    2571 flags.go:64] FLAG: --make-iptables-util-chains="true"
Aug 05 17:12:52.893585 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:12:52.886725    2571 flags.go:64] FLAG: --manifest-url=""
Aug 05 17:12:52.893585 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:12:52.886731    2571 flags.go:64] FLAG: --manifest-url-header=""
Aug 05 17:12:52.893585 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:12:52.886738    2571 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Aug 05 17:12:52.893585 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:12:52.886743    2571 flags.go:64] FLAG: --max-open-files="1000000"
Aug 05 17:12:52.893585 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:12:52.886749    2571 flags.go:64] FLAG: --max-pods="110"
Aug 05 17:12:52.893585 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:12:52.886754    2571 flags.go:64] FLAG: --maximum-dead-containers="-1"
Aug 05 17:12:52.893585 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:12:52.886759    2571 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Aug 05 17:12:52.893585 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:12:52.886763    2571 flags.go:64] FLAG: --memory-manager-policy="None"
Aug 05 17:12:52.894268 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:12:52.886768    2571 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Aug 05 17:12:52.894268 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:12:52.886773    2571 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Aug 05 17:12:52.894268 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:12:52.886778    2571 flags.go:64] FLAG: --node-ip="0.0.0.0"
Aug 05 17:12:52.894268 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:12:52.886782    2571 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Aug 05 17:12:52.894268 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:12:52.886793    2571 flags.go:64] FLAG: --node-status-max-images="50"
Aug 05 17:12:52.894268 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:12:52.886798    2571 flags.go:64] FLAG: --node-status-update-frequency="10s"
Aug 05 17:12:52.894268 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:12:52.886804    2571 flags.go:64] FLAG: --oom-score-adj="-999"
Aug 05 17:12:52.894268 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:12:52.886809    2571 flags.go:64] FLAG: --pod-cidr=""
Aug 05 17:12:52.894268 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:12:52.886813    2571 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:267c8c33005f5dc3d4121af550b1c396e30e2928a13624d7362adfbfb7f440f8"
Aug 05 17:12:52.894268 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:12:52.886821    2571 flags.go:64] FLAG: --pod-manifest-path=""
Aug 05 17:12:52.894268 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:12:52.886827    2571 flags.go:64] FLAG: --pod-max-pids="-1"
Aug 05 17:12:52.894268 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:12:52.886832    2571 flags.go:64] FLAG: --pods-per-core="0"
Aug 05 17:12:52.894268 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:12:52.886836    2571 flags.go:64] FLAG: --port="10250"
Aug 05 17:12:52.894268 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:12:52.886841    2571 flags.go:64] FLAG: --protect-kernel-defaults="false"
Aug 05 17:12:52.894268 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:12:52.886846    2571 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0fa896a040d401e48"
Aug 05 17:12:52.894268 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:12:52.886851    2571 flags.go:64] FLAG: --qos-reserved=""
Aug 05 17:12:52.894268 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:12:52.886856    2571 flags.go:64] FLAG: --read-only-port="10255"
Aug 05 17:12:52.894268 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:12:52.886861    2571 flags.go:64] FLAG: --register-node="true"
Aug 05 17:12:52.894268 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:12:52.886865    2571 flags.go:64] FLAG: --register-with-taints=""
Aug 05 17:12:52.894268 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:12:52.886871    2571 flags.go:64] FLAG: --registry-burst="10"
Aug 05 17:12:52.894268 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:12:52.886876    2571 flags.go:64] FLAG: --registry-qps="5"
Aug 05 17:12:52.894268 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:12:52.886880    2571 flags.go:64] FLAG: --reserved-cpus=""
Aug 05 17:12:52.894268 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:12:52.886885    2571 flags.go:64] FLAG: --reserved-memory=""
Aug 05 17:12:52.894268 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:12:52.886891    2571 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Aug 05 17:12:52.894268 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:12:52.886896    2571 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Aug 05 17:12:52.894965 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:12:52.886902    2571 flags.go:64] FLAG: --rotate-certificates="false"
Aug 05 17:12:52.894965 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:12:52.886906    2571 flags.go:64] FLAG: --rotate-server-certificates="false"
Aug 05 17:12:52.894965 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:12:52.886911    2571 flags.go:64] FLAG: --runonce="false"
Aug 05 17:12:52.894965 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:12:52.886916    2571 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Aug 05 17:12:52.894965 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:12:52.886920    2571 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Aug 05 17:12:52.894965 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:12:52.886925    2571 flags.go:64] FLAG: --seccomp-default="false"
Aug 05 17:12:52.894965 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:12:52.886930    2571 flags.go:64] FLAG: --serialize-image-pulls="true"
Aug 05 17:12:52.894965 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:12:52.886935    2571 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Aug 05 17:12:52.894965 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:12:52.886940    2571 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Aug 05 17:12:52.894965 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:12:52.886944    2571 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Aug 05 17:12:52.894965 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:12:52.886949    2571 flags.go:64] FLAG: --storage-driver-password="root"
Aug 05 17:12:52.894965 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:12:52.886954    2571 flags.go:64] FLAG: --storage-driver-secure="false"
Aug 05 17:12:52.894965 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:12:52.886959    2571 flags.go:64] FLAG: --storage-driver-table="stats"
Aug 05 17:12:52.894965 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:12:52.886963    2571 flags.go:64] FLAG: --storage-driver-user="root"
Aug 05 17:12:52.894965 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:12:52.886969    2571 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Aug 05 17:12:52.894965 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:12:52.886974    2571 flags.go:64] FLAG: --sync-frequency="1m0s"
Aug 05 17:12:52.894965 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:12:52.886979    2571 flags.go:64] FLAG: --system-cgroups=""
Aug 05 17:12:52.894965 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:12:52.886986    2571 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Aug 05 17:12:52.894965 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:12:52.886994    2571 flags.go:64] FLAG: --system-reserved-cgroup=""
Aug 05 17:12:52.894965 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:12:52.886999    2571 flags.go:64] FLAG: --tls-cert-file=""
Aug 05 17:12:52.894965 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:12:52.887003    2571 flags.go:64] FLAG: --tls-cipher-suites="[]"
Aug 05 17:12:52.894965 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:12:52.887010    2571 flags.go:64] FLAG: --tls-min-version=""
Aug 05 17:12:52.894965 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:12:52.887015    2571 flags.go:64] FLAG: --tls-private-key-file=""
Aug 05 17:12:52.894965 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:12:52.887019    2571 flags.go:64] FLAG: --topology-manager-policy="none"
Aug 05 17:12:52.894965 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:12:52.887024    2571 flags.go:64] FLAG: --topology-manager-policy-options=""
Aug 05 17:12:52.895601 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:12:52.887029    2571 flags.go:64] FLAG: --topology-manager-scope="container"
Aug 05 17:12:52.895601 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:12:52.887033    2571 flags.go:64] FLAG: --v="2"
Aug 05 17:12:52.895601 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:12:52.887039    2571 flags.go:64] FLAG: --version="false"
Aug 05 17:12:52.895601 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:12:52.887045    2571 flags.go:64] FLAG: --vmodule=""
Aug 05 17:12:52.895601 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:12:52.887051    2571 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Aug 05 17:12:52.895601 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:12:52.887057    2571 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Aug 05 17:12:52.895601 ip-10-0-135-2 kubenswrapper[2571]: W0805 17:12:52.887233    2571 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Aug 05 17:12:52.895601 ip-10-0-135-2 kubenswrapper[2571]: W0805 17:12:52.887242    2571 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Aug 05 17:12:52.895601 ip-10-0-135-2 kubenswrapper[2571]: W0805 17:12:52.887246    2571 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Aug 05 17:12:52.895601 ip-10-0-135-2 kubenswrapper[2571]: W0805 17:12:52.887251    2571 feature_gate.go:328] unrecognized feature gate: PinnedImages
Aug 05 17:12:52.895601 ip-10-0-135-2 kubenswrapper[2571]: W0805 17:12:52.887256    2571 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Aug 05 17:12:52.895601 ip-10-0-135-2 kubenswrapper[2571]: W0805 17:12:52.887260    2571 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Aug 05 17:12:52.895601 ip-10-0-135-2 kubenswrapper[2571]: W0805 17:12:52.887265    2571 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Aug 05 17:12:52.895601 ip-10-0-135-2 kubenswrapper[2571]: W0805 17:12:52.887269    2571 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Aug 05 17:12:52.895601 ip-10-0-135-2 kubenswrapper[2571]: W0805 17:12:52.887273    2571 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Aug 05 17:12:52.895601 ip-10-0-135-2 kubenswrapper[2571]: W0805 17:12:52.887277    2571 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Aug 05 17:12:52.895601 ip-10-0-135-2 kubenswrapper[2571]: W0805 17:12:52.887282    2571 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Aug 05 17:12:52.895601 ip-10-0-135-2 kubenswrapper[2571]: W0805 17:12:52.887286    2571 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Aug 05 17:12:52.895601 ip-10-0-135-2 kubenswrapper[2571]: W0805 17:12:52.887291    2571 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Aug 05 17:12:52.895601 ip-10-0-135-2 kubenswrapper[2571]: W0805 17:12:52.887295    2571 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Aug 05 17:12:52.895601 ip-10-0-135-2 kubenswrapper[2571]: W0805 17:12:52.887299    2571 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Aug 05 17:12:52.896107 ip-10-0-135-2 kubenswrapper[2571]: W0805 17:12:52.887304    2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Aug 05 17:12:52.896107 ip-10-0-135-2 kubenswrapper[2571]: W0805 17:12:52.887309    2571 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Aug 05 17:12:52.896107 ip-10-0-135-2 kubenswrapper[2571]: W0805 17:12:52.887313    2571 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Aug 05 17:12:52.896107 ip-10-0-135-2 kubenswrapper[2571]: W0805 17:12:52.887319    2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Aug 05 17:12:52.896107 ip-10-0-135-2 kubenswrapper[2571]: W0805 17:12:52.887324    2571 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Aug 05 17:12:52.896107 ip-10-0-135-2 kubenswrapper[2571]: W0805 17:12:52.887328    2571 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Aug 05 17:12:52.896107 ip-10-0-135-2 kubenswrapper[2571]: W0805 17:12:52.887332    2571 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Aug 05 17:12:52.896107 ip-10-0-135-2 kubenswrapper[2571]: W0805 17:12:52.887336    2571 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Aug 05 17:12:52.896107 ip-10-0-135-2 kubenswrapper[2571]: W0805 17:12:52.887340    2571 feature_gate.go:328] unrecognized feature gate: SignatureStores
Aug 05 17:12:52.896107 ip-10-0-135-2 kubenswrapper[2571]: W0805 17:12:52.887344    2571 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Aug 05 17:12:52.896107 ip-10-0-135-2 kubenswrapper[2571]: W0805 17:12:52.887349    2571 feature_gate.go:328] unrecognized feature gate: DualReplica
Aug 05 17:12:52.896107 ip-10-0-135-2 kubenswrapper[2571]: W0805 17:12:52.887354    2571 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Aug 05 17:12:52.896107 ip-10-0-135-2 kubenswrapper[2571]: W0805 17:12:52.887358    2571 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Aug 05 17:12:52.896107 ip-10-0-135-2 kubenswrapper[2571]: W0805 17:12:52.887362    2571 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Aug 05 17:12:52.896107 ip-10-0-135-2 kubenswrapper[2571]: W0805 17:12:52.887366    2571 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Aug 05 17:12:52.896107 ip-10-0-135-2 kubenswrapper[2571]: W0805 17:12:52.887387    2571 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Aug 05 17:12:52.896107 ip-10-0-135-2 kubenswrapper[2571]: W0805 17:12:52.887391    2571 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Aug 05 17:12:52.896107 ip-10-0-135-2 kubenswrapper[2571]: W0805 17:12:52.887395    2571 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Aug 05 17:12:52.896107 ip-10-0-135-2 kubenswrapper[2571]: W0805 17:12:52.887402    2571 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Aug 05 17:12:52.896107 ip-10-0-135-2 kubenswrapper[2571]: W0805 17:12:52.887406    2571 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Aug 05 17:12:52.896628 ip-10-0-135-2 kubenswrapper[2571]: W0805 17:12:52.887410    2571 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Aug 05 17:12:52.896628 ip-10-0-135-2 kubenswrapper[2571]: W0805 17:12:52.887414    2571 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Aug 05 17:12:52.896628 ip-10-0-135-2 kubenswrapper[2571]: W0805 17:12:52.887419    2571 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Aug 05 17:12:52.896628 ip-10-0-135-2 kubenswrapper[2571]: W0805 17:12:52.887423    2571 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Aug 05 17:12:52.896628 ip-10-0-135-2 kubenswrapper[2571]: W0805 17:12:52.887427    2571 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Aug 05 17:12:52.896628 ip-10-0-135-2 kubenswrapper[2571]: W0805 17:12:52.887431    2571 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Aug 05 17:12:52.896628 ip-10-0-135-2 kubenswrapper[2571]: W0805 17:12:52.887435    2571 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Aug 05 17:12:52.896628 ip-10-0-135-2 kubenswrapper[2571]: W0805 17:12:52.887439    2571 feature_gate.go:328] unrecognized feature gate: EventTTL
Aug 05 17:12:52.896628 ip-10-0-135-2 kubenswrapper[2571]: W0805 17:12:52.887443    2571 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Aug 05 17:12:52.896628 ip-10-0-135-2 kubenswrapper[2571]: W0805 17:12:52.887448    2571 feature_gate.go:328] unrecognized feature gate: OSStreams
Aug 05 17:12:52.896628 ip-10-0-135-2 kubenswrapper[2571]: W0805 17:12:52.887452    2571 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Aug 05 17:12:52.896628 ip-10-0-135-2 kubenswrapper[2571]: W0805 17:12:52.887457    2571 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Aug 05 17:12:52.896628 ip-10-0-135-2 kubenswrapper[2571]: W0805 17:12:52.887461    2571 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Aug 05 17:12:52.896628 ip-10-0-135-2 kubenswrapper[2571]: W0805 17:12:52.887465    2571 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Aug 05 17:12:52.896628 ip-10-0-135-2 kubenswrapper[2571]: W0805 17:12:52.887470    2571 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Aug 05 17:12:52.896628 ip-10-0-135-2 kubenswrapper[2571]: W0805 17:12:52.887476    2571 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Aug 05 17:12:52.896628 ip-10-0-135-2 kubenswrapper[2571]: W0805 17:12:52.887479    2571 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Aug 05 17:12:52.896628 ip-10-0-135-2 kubenswrapper[2571]: W0805 17:12:52.887483    2571 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Aug 05 17:12:52.896628 ip-10-0-135-2 kubenswrapper[2571]: W0805 17:12:52.887487    2571 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Aug 05 17:12:52.896628 ip-10-0-135-2 kubenswrapper[2571]: W0805 17:12:52.887491    2571 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Aug 05 17:12:52.897125 ip-10-0-135-2 kubenswrapper[2571]: W0805 17:12:52.887495    2571 feature_gate.go:328] unrecognized feature gate: Example2
Aug 05 17:12:52.897125 ip-10-0-135-2 kubenswrapper[2571]: W0805 17:12:52.887499    2571 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Aug 05 17:12:52.897125 ip-10-0-135-2 kubenswrapper[2571]: W0805 17:12:52.887503    2571 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Aug 05 17:12:52.897125 ip-10-0-135-2 kubenswrapper[2571]: W0805 17:12:52.887507    2571 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Aug 05 17:12:52.897125 ip-10-0-135-2 kubenswrapper[2571]: W0805 17:12:52.887511    2571 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Aug 05 17:12:52.897125 ip-10-0-135-2 kubenswrapper[2571]: W0805 17:12:52.887515    2571 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Aug 05 17:12:52.897125 ip-10-0-135-2 kubenswrapper[2571]: W0805 17:12:52.887519    2571 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Aug 05 17:12:52.897125 ip-10-0-135-2 kubenswrapper[2571]: W0805 17:12:52.887526    2571 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Aug 05 17:12:52.897125 ip-10-0-135-2 kubenswrapper[2571]: W0805 17:12:52.887531    2571 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Aug 05 17:12:52.897125 ip-10-0-135-2 kubenswrapper[2571]: W0805 17:12:52.887537    2571 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Aug 05 17:12:52.897125 ip-10-0-135-2 kubenswrapper[2571]: W0805 17:12:52.887543    2571 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Aug 05 17:12:52.897125 ip-10-0-135-2 kubenswrapper[2571]: W0805 17:12:52.887550    2571 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Aug 05 17:12:52.897125 ip-10-0-135-2 kubenswrapper[2571]: W0805 17:12:52.887556    2571 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Aug 05 17:12:52.897125 ip-10-0-135-2 kubenswrapper[2571]: W0805 17:12:52.887560    2571 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Aug 05 17:12:52.897125 ip-10-0-135-2 kubenswrapper[2571]: W0805 17:12:52.887565    2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Aug 05 17:12:52.897125 ip-10-0-135-2 kubenswrapper[2571]: W0805 17:12:52.887570    2571 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Aug 05 17:12:52.897125 ip-10-0-135-2 kubenswrapper[2571]: W0805 17:12:52.887574    2571 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Aug 05 17:12:52.897125 ip-10-0-135-2 kubenswrapper[2571]: W0805 17:12:52.887578    2571 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Aug 05 17:12:52.897125 ip-10-0-135-2 kubenswrapper[2571]: W0805 17:12:52.887582    2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Aug 05 17:12:52.897614 ip-10-0-135-2 kubenswrapper[2571]: W0805 17:12:52.887586    2571 feature_gate.go:328] unrecognized feature gate: OVNObservability
Aug 05 17:12:52.897614 ip-10-0-135-2 kubenswrapper[2571]: W0805 17:12:52.887590    2571 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Aug 05 17:12:52.897614 ip-10-0-135-2 kubenswrapper[2571]: W0805 17:12:52.887595    2571 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Aug 05 17:12:52.897614 ip-10-0-135-2 kubenswrapper[2571]: W0805 17:12:52.887599    2571 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Aug 05 17:12:52.897614 ip-10-0-135-2 kubenswrapper[2571]: W0805 17:12:52.887603    2571 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Aug 05 17:12:52.897614 ip-10-0-135-2 kubenswrapper[2571]: W0805 17:12:52.887608    2571 feature_gate.go:328] unrecognized feature gate: NewOLM
Aug 05 17:12:52.897614 ip-10-0-135-2 kubenswrapper[2571]: W0805 17:12:52.887614    2571 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Aug 05 17:12:52.897614 ip-10-0-135-2 kubenswrapper[2571]: W0805 17:12:52.887620    2571 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Aug 05 17:12:52.897614 ip-10-0-135-2 kubenswrapper[2571]: W0805 17:12:52.887627    2571 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Aug 05 17:12:52.897614 ip-10-0-135-2 kubenswrapper[2571]: W0805 17:12:52.887631    2571 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Aug 05 17:12:52.897614 ip-10-0-135-2 kubenswrapper[2571]: W0805 17:12:52.887636    2571 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Aug 05 17:12:52.897614 ip-10-0-135-2 kubenswrapper[2571]: W0805 17:12:52.887640    2571 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Aug 05 17:12:52.897614 ip-10-0-135-2 kubenswrapper[2571]: W0805 17:12:52.887644    2571 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Aug 05 17:12:52.897614 ip-10-0-135-2 kubenswrapper[2571]: W0805 17:12:52.887648    2571 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Aug 05 17:12:52.897614 ip-10-0-135-2 kubenswrapper[2571]: W0805 17:12:52.887653    2571 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Aug 05 17:12:52.897614 ip-10-0-135-2 kubenswrapper[2571]: W0805 17:12:52.887657    2571 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Aug 05 17:12:52.897614 ip-10-0-135-2 kubenswrapper[2571]: W0805 17:12:52.887661    2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Aug 05 17:12:52.897614 ip-10-0-135-2 kubenswrapper[2571]: W0805 17:12:52.887665    2571 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Aug 05 17:12:52.897614 ip-10-0-135-2 kubenswrapper[2571]: W0805 17:12:52.887669    2571 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Aug 05 17:12:52.898073 ip-10-0-135-2 kubenswrapper[2571]: W0805 17:12:52.887673    2571 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Aug 05 17:12:52.898073 ip-10-0-135-2 kubenswrapper[2571]: W0805 17:12:52.887677    2571 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Aug 05 17:12:52.898073 ip-10-0-135-2 kubenswrapper[2571]: W0805 17:12:52.887682    2571 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Aug 05 17:12:52.898073 ip-10-0-135-2 kubenswrapper[2571]: W0805 17:12:52.887686    2571 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Aug 05 17:12:52.898073 ip-10-0-135-2 kubenswrapper[2571]: W0805 17:12:52.887692    2571 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Aug 05 17:12:52.898073 ip-10-0-135-2 kubenswrapper[2571]: W0805 17:12:52.887696    2571 feature_gate.go:328] unrecognized feature gate: Example
Aug 05 17:12:52.898073 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:12:52.888319    2571 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Aug 05 17:12:52.898073 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:12:52.895362    2571 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Aug 05 17:12:52.898073 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:12:52.895400    2571 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Aug 05 17:12:52.898073 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:12:52.896113    2571 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Aug 05 17:12:52.898073 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:12:52.896121    2571 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Aug 05 17:12:52.898073 ip-10-0-135-2 kubenswrapper[2571]: W0805 17:12:52.896183    2571 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Aug 05 17:12:52.898073 ip-10-0-135-2 kubenswrapper[2571]: W0805 17:12:52.896188    2571 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Aug 05 17:12:52.898073 ip-10-0-135-2 kubenswrapper[2571]: W0805 17:12:52.896192    2571 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Aug 05 17:12:52.898073 ip-10-0-135-2 kubenswrapper[2571]: W0805 17:12:52.896195    2571 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Aug 05 17:12:52.898525 ip-10-0-135-2 kubenswrapper[2571]: W0805 17:12:52.896199    2571 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Aug 05 17:12:52.898525 ip-10-0-135-2 kubenswrapper[2571]: W0805 17:12:52.896202    2571 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Aug 05 17:12:52.898525 ip-10-0-135-2 kubenswrapper[2571]: W0805 17:12:52.896205    2571 feature_gate.go:328] unrecognized feature gate: PinnedImages
Aug 05 17:12:52.898525 ip-10-0-135-2 kubenswrapper[2571]: W0805 17:12:52.896207    2571 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Aug 05 17:12:52.898525 ip-10-0-135-2 kubenswrapper[2571]: W0805 17:12:52.896210    2571 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Aug 05 17:12:52.898525 ip-10-0-135-2 kubenswrapper[2571]: W0805 17:12:52.896213    2571 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Aug 05 17:12:52.898525 ip-10-0-135-2 kubenswrapper[2571]: W0805 17:12:52.896217    2571 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Aug 05 17:12:52.898525 ip-10-0-135-2 kubenswrapper[2571]: W0805 17:12:52.896220    2571 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Aug 05 17:12:52.898525 ip-10-0-135-2 kubenswrapper[2571]: W0805 17:12:52.896222    2571 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Aug 05 17:12:52.898525 ip-10-0-135-2 kubenswrapper[2571]: W0805 17:12:52.896225    2571 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Aug 05 17:12:52.898525 ip-10-0-135-2 kubenswrapper[2571]: W0805 17:12:52.896227    2571 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Aug 05 17:12:52.898525 ip-10-0-135-2 kubenswrapper[2571]: W0805 17:12:52.896230    2571 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Aug 05 17:12:52.898525 ip-10-0-135-2 kubenswrapper[2571]: W0805 17:12:52.896233    2571 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Aug 05 17:12:52.898525 ip-10-0-135-2 kubenswrapper[2571]: W0805 17:12:52.896235    2571 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Aug 05 17:12:52.898525 ip-10-0-135-2 kubenswrapper[2571]: W0805 17:12:52.896238    2571 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Aug 05 17:12:52.898525 ip-10-0-135-2 kubenswrapper[2571]: W0805 17:12:52.896241    2571 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Aug 05 17:12:52.898525 ip-10-0-135-2 kubenswrapper[2571]: W0805 17:12:52.896243    2571 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Aug 05 17:12:52.898525 ip-10-0-135-2 kubenswrapper[2571]: W0805 17:12:52.896246    2571 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Aug 05 17:12:52.898525 ip-10-0-135-2 kubenswrapper[2571]: W0805 17:12:52.896249    2571 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Aug 05 17:12:52.898525 ip-10-0-135-2 kubenswrapper[2571]: W0805 17:12:52.896251    2571 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Aug 05 17:12:52.899014 ip-10-0-135-2 kubenswrapper[2571]: W0805 17:12:52.896255    2571 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Aug 05 17:12:52.899014 ip-10-0-135-2 kubenswrapper[2571]: W0805 17:12:52.896260    2571 feature_gate.go:328] unrecognized feature gate: NewOLM
Aug 05 17:12:52.899014 ip-10-0-135-2 kubenswrapper[2571]: W0805 17:12:52.896263    2571 feature_gate.go:328] unrecognized feature gate: DualReplica
Aug 05 17:12:52.899014 ip-10-0-135-2 kubenswrapper[2571]: W0805 17:12:52.896266    2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Aug 05 17:12:52.899014 ip-10-0-135-2 kubenswrapper[2571]: W0805 17:12:52.896268    2571 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Aug 05 17:12:52.899014 ip-10-0-135-2 kubenswrapper[2571]: W0805 17:12:52.896271    2571 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Aug 05 17:12:52.899014 ip-10-0-135-2 kubenswrapper[2571]: W0805 17:12:52.896275    2571 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Aug 05 17:12:52.899014 ip-10-0-135-2 kubenswrapper[2571]: W0805 17:12:52.896277    2571 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Aug 05 17:12:52.899014 ip-10-0-135-2 kubenswrapper[2571]: W0805 17:12:52.896280    2571 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Aug 05 17:12:52.899014 ip-10-0-135-2 kubenswrapper[2571]: W0805 17:12:52.896282    2571 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Aug 05 17:12:52.899014 ip-10-0-135-2 kubenswrapper[2571]: W0805 17:12:52.896285    2571 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Aug 05 17:12:52.899014 ip-10-0-135-2 kubenswrapper[2571]: W0805 17:12:52.896288    2571 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Aug 05 17:12:52.899014 ip-10-0-135-2 kubenswrapper[2571]: W0805 17:12:52.896291    2571 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Aug 05 17:12:52.899014 ip-10-0-135-2 kubenswrapper[2571]: W0805 17:12:52.896293    2571 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Aug 05 17:12:52.899014 ip-10-0-135-2 kubenswrapper[2571]: W0805 17:12:52.896296    2571 feature_gate.go:328] unrecognized feature gate: SignatureStores
Aug 05 17:12:52.899014 ip-10-0-135-2 kubenswrapper[2571]: W0805 17:12:52.896298    2571 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Aug 05 17:12:52.899014 ip-10-0-135-2 kubenswrapper[2571]: W0805 17:12:52.896301    2571 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Aug 05 17:12:52.899014 ip-10-0-135-2 kubenswrapper[2571]: W0805 17:12:52.896304    2571 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Aug 05 17:12:52.899014 ip-10-0-135-2 kubenswrapper[2571]: W0805 17:12:52.896308    2571 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Aug 05 17:12:52.899014 ip-10-0-135-2 kubenswrapper[2571]: W0805 17:12:52.896313    2571 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Aug 05 17:12:52.899520 ip-10-0-135-2 kubenswrapper[2571]: W0805 17:12:52.896316    2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Aug 05 17:12:52.899520 ip-10-0-135-2 kubenswrapper[2571]: W0805 17:12:52.896319    2571 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Aug 05 17:12:52.899520 ip-10-0-135-2 kubenswrapper[2571]: W0805 17:12:52.896322    2571 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Aug 05 17:12:52.899520 ip-10-0-135-2 kubenswrapper[2571]: W0805 17:12:52.896324    2571 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Aug 05 17:12:52.899520 ip-10-0-135-2 kubenswrapper[2571]: W0805 17:12:52.896327    2571 feature_gate.go:328] unrecognized feature gate: OSStreams
Aug 05 17:12:52.899520 ip-10-0-135-2 kubenswrapper[2571]: W0805 17:12:52.896329    2571 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Aug 05 17:12:52.899520 ip-10-0-135-2 kubenswrapper[2571]: W0805 17:12:52.896331    2571 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Aug 05 17:12:52.899520 ip-10-0-135-2 kubenswrapper[2571]: W0805 17:12:52.896334    2571 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Aug 05 17:12:52.899520 ip-10-0-135-2 kubenswrapper[2571]: W0805 17:12:52.896336    2571 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Aug 05 17:12:52.899520 ip-10-0-135-2 kubenswrapper[2571]: W0805 17:12:52.896339    2571 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Aug 05 17:12:52.899520 ip-10-0-135-2 kubenswrapper[2571]: W0805 17:12:52.896341    2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Aug 05 17:12:52.899520 ip-10-0-135-2 kubenswrapper[2571]: W0805 17:12:52.896344    2571 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Aug 05 17:12:52.899520 ip-10-0-135-2 kubenswrapper[2571]: W0805 17:12:52.896346    2571 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Aug 05 17:12:52.899520 ip-10-0-135-2 kubenswrapper[2571]: W0805 17:12:52.896349    2571 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Aug 05 17:12:52.899520 ip-10-0-135-2 kubenswrapper[2571]: W0805 17:12:52.896351    2571 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Aug 05 17:12:52.899520 ip-10-0-135-2 kubenswrapper[2571]: W0805 17:12:52.896354    2571 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Aug 05 17:12:52.899520 ip-10-0-135-2 kubenswrapper[2571]: W0805 17:12:52.896356    2571 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Aug 05 17:12:52.899520 ip-10-0-135-2 kubenswrapper[2571]: W0805 17:12:52.896360    2571 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Aug 05 17:12:52.899520 ip-10-0-135-2 kubenswrapper[2571]: W0805 17:12:52.896363    2571 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Aug 05 17:12:52.899520 ip-10-0-135-2 kubenswrapper[2571]: W0805 17:12:52.896365    2571 feature_gate.go:328] unrecognized feature gate: EventTTL
Aug 05 17:12:52.900020 ip-10-0-135-2 kubenswrapper[2571]: W0805 17:12:52.896368    2571 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Aug 05 17:12:52.900020 ip-10-0-135-2 kubenswrapper[2571]: W0805 17:12:52.896370    2571 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Aug 05 17:12:52.900020 ip-10-0-135-2 kubenswrapper[2571]: W0805 17:12:52.896387    2571 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Aug 05 17:12:52.900020 ip-10-0-135-2 kubenswrapper[2571]: W0805 17:12:52.896391    2571 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Aug 05 17:12:52.900020 ip-10-0-135-2 kubenswrapper[2571]: W0805 17:12:52.896394    2571 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Aug 05 17:12:52.900020 ip-10-0-135-2 kubenswrapper[2571]: W0805 17:12:52.896396    2571 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Aug 05 17:12:52.900020 ip-10-0-135-2 kubenswrapper[2571]: W0805 17:12:52.896399    2571 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Aug 05 17:12:52.900020 ip-10-0-135-2 kubenswrapper[2571]: W0805 17:12:52.896401    2571 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Aug 05 17:12:52.900020 ip-10-0-135-2 kubenswrapper[2571]: W0805 17:12:52.896404    2571 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Aug 05 17:12:52.900020 ip-10-0-135-2 kubenswrapper[2571]: W0805 17:12:52.896406    2571 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Aug 05 17:12:52.900020 ip-10-0-135-2 kubenswrapper[2571]: W0805 17:12:52.896409    2571 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Aug 05 17:12:52.900020 ip-10-0-135-2 kubenswrapper[2571]: W0805 17:12:52.896411    2571 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Aug 05 17:12:52.900020 ip-10-0-135-2 kubenswrapper[2571]: W0805 17:12:52.896415    2571 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Aug 05 17:12:52.900020 ip-10-0-135-2 kubenswrapper[2571]: W0805 17:12:52.896417    2571 feature_gate.go:328] unrecognized feature gate: Example
Aug 05 17:12:52.900020 ip-10-0-135-2 kubenswrapper[2571]: W0805 17:12:52.896420    2571 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Aug 05 17:12:52.900020 ip-10-0-135-2 kubenswrapper[2571]: W0805 17:12:52.896424    2571 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Aug 05 17:12:52.900020 ip-10-0-135-2 kubenswrapper[2571]: W0805 17:12:52.896427    2571 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Aug 05 17:12:52.900020 ip-10-0-135-2 kubenswrapper[2571]: W0805 17:12:52.896430    2571 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Aug 05 17:12:52.900020 ip-10-0-135-2 kubenswrapper[2571]: W0805 17:12:52.896433    2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Aug 05 17:12:52.900522 ip-10-0-135-2 kubenswrapper[2571]: W0805 17:12:52.896436    2571 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Aug 05 17:12:52.900522 ip-10-0-135-2 kubenswrapper[2571]: W0805 17:12:52.896438    2571 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Aug 05 17:12:52.900522 ip-10-0-135-2 kubenswrapper[2571]: W0805 17:12:52.896441    2571 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Aug 05 17:12:52.900522 ip-10-0-135-2 kubenswrapper[2571]: W0805 17:12:52.896444    2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Aug 05 17:12:52.900522 ip-10-0-135-2 kubenswrapper[2571]: W0805 17:12:52.896446    2571 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Aug 05 17:12:52.900522 ip-10-0-135-2 kubenswrapper[2571]: W0805 17:12:52.896449    2571 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Aug 05 17:12:52.900522 ip-10-0-135-2 kubenswrapper[2571]: W0805 17:12:52.896452    2571 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Aug 05 17:12:52.900522 ip-10-0-135-2 kubenswrapper[2571]: W0805 17:12:52.896455    2571 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Aug 05 17:12:52.900522 ip-10-0-135-2 kubenswrapper[2571]: W0805 17:12:52.896457    2571 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Aug 05 17:12:52.900522 ip-10-0-135-2 kubenswrapper[2571]: W0805 17:12:52.896459    2571 feature_gate.go:328] unrecognized feature gate: Example2
Aug 05 17:12:52.900522 ip-10-0-135-2 kubenswrapper[2571]: W0805 17:12:52.896462    2571 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Aug 05 17:12:52.900522 ip-10-0-135-2 kubenswrapper[2571]: W0805 17:12:52.896465    2571 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Aug 05 17:12:52.900522 ip-10-0-135-2 kubenswrapper[2571]: W0805 17:12:52.896468    2571 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Aug 05 17:12:52.900522 ip-10-0-135-2 kubenswrapper[2571]: W0805 17:12:52.896470    2571 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Aug 05 17:12:52.900522 ip-10-0-135-2 kubenswrapper[2571]: W0805 17:12:52.896473    2571 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Aug 05 17:12:52.900522 ip-10-0-135-2 kubenswrapper[2571]: W0805 17:12:52.896475    2571 feature_gate.go:328] unrecognized feature gate: OVNObservability
Aug 05 17:12:52.900916 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:12:52.896480    2571 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Aug 05 17:12:52.900916 ip-10-0-135-2 kubenswrapper[2571]: W0805 17:12:52.896587    2571 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Aug 05 17:12:52.900916 ip-10-0-135-2 kubenswrapper[2571]: W0805 17:12:52.896591    2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Aug 05 17:12:52.900916 ip-10-0-135-2 kubenswrapper[2571]: W0805 17:12:52.896593    2571 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Aug 05 17:12:52.900916 ip-10-0-135-2 kubenswrapper[2571]: W0805 17:12:52.896596    2571 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Aug 05 17:12:52.900916 ip-10-0-135-2 kubenswrapper[2571]: W0805 17:12:52.896598    2571 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Aug 05 17:12:52.900916 ip-10-0-135-2 kubenswrapper[2571]: W0805 17:12:52.896601    2571 feature_gate.go:328] unrecognized feature gate: EventTTL
Aug 05 17:12:52.900916 ip-10-0-135-2 kubenswrapper[2571]: W0805 17:12:52.896603    2571 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Aug 05 17:12:52.900916 ip-10-0-135-2 kubenswrapper[2571]: W0805 17:12:52.896606    2571 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Aug 05 17:12:52.900916 ip-10-0-135-2 kubenswrapper[2571]: W0805 17:12:52.896608    2571 feature_gate.go:328] unrecognized feature gate: NewOLM
Aug 05 17:12:52.900916 ip-10-0-135-2 kubenswrapper[2571]: W0805 17:12:52.896611    2571 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Aug 05 17:12:52.900916 ip-10-0-135-2 kubenswrapper[2571]: W0805 17:12:52.896614    2571 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Aug 05 17:12:52.900916 ip-10-0-135-2 kubenswrapper[2571]: W0805 17:12:52.896617    2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Aug 05 17:12:52.900916 ip-10-0-135-2 kubenswrapper[2571]: W0805 17:12:52.896619    2571 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Aug 05 17:12:52.900916 ip-10-0-135-2 kubenswrapper[2571]: W0805 17:12:52.896621    2571 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Aug 05 17:12:52.901286 ip-10-0-135-2 kubenswrapper[2571]: W0805 17:12:52.896624    2571 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Aug 05 17:12:52.901286 ip-10-0-135-2 kubenswrapper[2571]: W0805 17:12:52.896626    2571 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Aug 05 17:12:52.901286 ip-10-0-135-2 kubenswrapper[2571]: W0805 17:12:52.896629    2571 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Aug 05 17:12:52.901286 ip-10-0-135-2 kubenswrapper[2571]: W0805 17:12:52.896633    2571 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Aug 05 17:12:52.901286 ip-10-0-135-2 kubenswrapper[2571]: W0805 17:12:52.896637    2571 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Aug 05 17:12:52.901286 ip-10-0-135-2 kubenswrapper[2571]: W0805 17:12:52.896640    2571 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Aug 05 17:12:52.901286 ip-10-0-135-2 kubenswrapper[2571]: W0805 17:12:52.896645    2571 feature_gate.go:328] unrecognized feature gate: SignatureStores
Aug 05 17:12:52.901286 ip-10-0-135-2 kubenswrapper[2571]: W0805 17:12:52.896648    2571 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Aug 05 17:12:52.901286 ip-10-0-135-2 kubenswrapper[2571]: W0805 17:12:52.896651    2571 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Aug 05 17:12:52.901286 ip-10-0-135-2 kubenswrapper[2571]: W0805 17:12:52.896654    2571 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Aug 05 17:12:52.901286 ip-10-0-135-2 kubenswrapper[2571]: W0805 17:12:52.896657    2571 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Aug 05 17:12:52.901286 ip-10-0-135-2 kubenswrapper[2571]: W0805 17:12:52.896660    2571 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Aug 05 17:12:52.901286 ip-10-0-135-2 kubenswrapper[2571]: W0805 17:12:52.896663    2571 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Aug 05 17:12:52.901286 ip-10-0-135-2 kubenswrapper[2571]: W0805 17:12:52.896666    2571 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Aug 05 17:12:52.901286 ip-10-0-135-2 kubenswrapper[2571]: W0805 17:12:52.896668    2571 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Aug 05 17:12:52.901286 ip-10-0-135-2 kubenswrapper[2571]: W0805 17:12:52.896671    2571 feature_gate.go:328] unrecognized feature gate: OVNObservability
Aug 05 17:12:52.901286 ip-10-0-135-2 kubenswrapper[2571]: W0805 17:12:52.896673    2571 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Aug 05 17:12:52.901286 ip-10-0-135-2 kubenswrapper[2571]: W0805 17:12:52.896676    2571 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Aug 05 17:12:52.901286 ip-10-0-135-2 kubenswrapper[2571]: W0805 17:12:52.896678    2571 feature_gate.go:328] unrecognized feature gate: PinnedImages
Aug 05 17:12:52.901781 ip-10-0-135-2 kubenswrapper[2571]: W0805 17:12:52.896681    2571 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Aug 05 17:12:52.901781 ip-10-0-135-2 kubenswrapper[2571]: W0805 17:12:52.896684    2571 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Aug 05 17:12:52.901781 ip-10-0-135-2 kubenswrapper[2571]: W0805 17:12:52.896686    2571 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Aug 05 17:12:52.901781 ip-10-0-135-2 kubenswrapper[2571]: W0805 17:12:52.896689    2571 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Aug 05 17:12:52.901781 ip-10-0-135-2 kubenswrapper[2571]: W0805 17:12:52.896691    2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Aug 05 17:12:52.901781 ip-10-0-135-2 kubenswrapper[2571]: W0805 17:12:52.896694    2571 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Aug 05 17:12:52.901781 ip-10-0-135-2 kubenswrapper[2571]: W0805 17:12:52.896698    2571 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Aug 05 17:12:52.901781 ip-10-0-135-2 kubenswrapper[2571]: W0805 17:12:52.896701    2571 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Aug 05 17:12:52.901781 ip-10-0-135-2 kubenswrapper[2571]: W0805 17:12:52.896704    2571 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Aug 05 17:12:52.901781 ip-10-0-135-2 kubenswrapper[2571]: W0805 17:12:52.896708    2571 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Aug 05 17:12:52.901781 ip-10-0-135-2 kubenswrapper[2571]: W0805 17:12:52.896711    2571 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Aug 05 17:12:52.901781 ip-10-0-135-2 kubenswrapper[2571]: W0805 17:12:52.896714    2571 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Aug 05 17:12:52.901781 ip-10-0-135-2 kubenswrapper[2571]: W0805 17:12:52.896717    2571 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Aug 05 17:12:52.901781 ip-10-0-135-2 kubenswrapper[2571]: W0805 17:12:52.896719    2571 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Aug 05 17:12:52.901781 ip-10-0-135-2 kubenswrapper[2571]: W0805 17:12:52.896722    2571 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Aug 05 17:12:52.901781 ip-10-0-135-2 kubenswrapper[2571]: W0805 17:12:52.896724    2571 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Aug 05 17:12:52.901781 ip-10-0-135-2 kubenswrapper[2571]: W0805 17:12:52.896727    2571 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Aug 05 17:12:52.901781 ip-10-0-135-2 kubenswrapper[2571]: W0805 17:12:52.896729    2571 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Aug 05 17:12:52.901781 ip-10-0-135-2 kubenswrapper[2571]: W0805 17:12:52.896732    2571 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Aug 05 17:12:52.902241 ip-10-0-135-2 kubenswrapper[2571]: W0805 17:12:52.896734    2571 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Aug 05 17:12:52.902241 ip-10-0-135-2 kubenswrapper[2571]: W0805 17:12:52.896737    2571 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Aug 05 17:12:52.902241 ip-10-0-135-2 kubenswrapper[2571]: W0805 17:12:52.896739    2571 feature_gate.go:328] unrecognized feature gate: Example
Aug 05 17:12:52.902241 ip-10-0-135-2 kubenswrapper[2571]: W0805 17:12:52.896742    2571 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Aug 05 17:12:52.902241 ip-10-0-135-2 kubenswrapper[2571]: W0805 17:12:52.896745    2571 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Aug 05 17:12:52.902241 ip-10-0-135-2 kubenswrapper[2571]: W0805 17:12:52.896748    2571 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Aug 05 17:12:52.902241 ip-10-0-135-2 kubenswrapper[2571]: W0805 17:12:52.896750    2571 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Aug 05 17:12:52.902241 ip-10-0-135-2 kubenswrapper[2571]: W0805 17:12:52.896753    2571 feature_gate.go:328] unrecognized feature gate: DualReplica
Aug 05 17:12:52.902241 ip-10-0-135-2 kubenswrapper[2571]: W0805 17:12:52.896756    2571 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Aug 05 17:12:52.902241 ip-10-0-135-2 kubenswrapper[2571]: W0805 17:12:52.896758    2571 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Aug 05 17:12:52.902241 ip-10-0-135-2 kubenswrapper[2571]: W0805 17:12:52.896761    2571 feature_gate.go:328] unrecognized feature gate: Example2
Aug 05 17:12:52.902241 ip-10-0-135-2 kubenswrapper[2571]: W0805 17:12:52.896763    2571 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Aug 05 17:12:52.902241 ip-10-0-135-2 kubenswrapper[2571]: W0805 17:12:52.896766    2571 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Aug 05 17:12:52.902241 ip-10-0-135-2 kubenswrapper[2571]: W0805 17:12:52.896769    2571 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Aug 05 17:12:52.902241 ip-10-0-135-2 kubenswrapper[2571]: W0805 17:12:52.896771    2571 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Aug 05 17:12:52.902241 ip-10-0-135-2 kubenswrapper[2571]: W0805 17:12:52.896774    2571 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Aug 05 17:12:52.902241 ip-10-0-135-2 kubenswrapper[2571]: W0805 17:12:52.896776    2571 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Aug 05 17:12:52.902241 ip-10-0-135-2 kubenswrapper[2571]: W0805 17:12:52.896779    2571 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Aug 05 17:12:52.902241 ip-10-0-135-2 kubenswrapper[2571]: W0805 17:12:52.896782    2571 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Aug 05 17:12:52.902241 ip-10-0-135-2 kubenswrapper[2571]: W0805 17:12:52.896784    2571 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Aug 05 17:12:52.902888 ip-10-0-135-2 kubenswrapper[2571]: W0805 17:12:52.896787    2571 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Aug 05 17:12:52.902888 ip-10-0-135-2 kubenswrapper[2571]: W0805 17:12:52.896789    2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Aug 05 17:12:52.902888 ip-10-0-135-2 kubenswrapper[2571]: W0805 17:12:52.896792    2571 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Aug 05 17:12:52.902888 ip-10-0-135-2 kubenswrapper[2571]: W0805 17:12:52.896795    2571 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Aug 05 17:12:52.902888 ip-10-0-135-2 kubenswrapper[2571]: W0805 17:12:52.896797    2571 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Aug 05 17:12:52.902888 ip-10-0-135-2 kubenswrapper[2571]: W0805 17:12:52.896800    2571 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Aug 05 17:12:52.902888 ip-10-0-135-2 kubenswrapper[2571]: W0805 17:12:52.896802    2571 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Aug 05 17:12:52.902888 ip-10-0-135-2 kubenswrapper[2571]: W0805 17:12:52.896804    2571 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Aug 05 17:12:52.902888 ip-10-0-135-2 kubenswrapper[2571]: W0805 17:12:52.896807    2571 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Aug 05 17:12:52.902888 ip-10-0-135-2 kubenswrapper[2571]: W0805 17:12:52.896810    2571 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Aug 05 17:12:52.902888 ip-10-0-135-2 kubenswrapper[2571]: W0805 17:12:52.896812    2571 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Aug 05 17:12:52.902888 ip-10-0-135-2 kubenswrapper[2571]: W0805 17:12:52.896814    2571 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Aug 05 17:12:52.902888 ip-10-0-135-2 kubenswrapper[2571]: W0805 17:12:52.896817    2571 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Aug 05 17:12:52.902888 ip-10-0-135-2 kubenswrapper[2571]: W0805 17:12:52.896819    2571 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Aug 05 17:12:52.902888 ip-10-0-135-2 kubenswrapper[2571]: W0805 17:12:52.896822    2571 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Aug 05 17:12:52.902888 ip-10-0-135-2 kubenswrapper[2571]: W0805 17:12:52.896826    2571 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Aug 05 17:12:52.902888 ip-10-0-135-2 kubenswrapper[2571]: W0805 17:12:52.896828    2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Aug 05 17:12:52.902888 ip-10-0-135-2 kubenswrapper[2571]: W0805 17:12:52.896831    2571 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Aug 05 17:12:52.902888 ip-10-0-135-2 kubenswrapper[2571]: W0805 17:12:52.896833    2571 feature_gate.go:328] unrecognized feature gate: OSStreams
Aug 05 17:12:52.902888 ip-10-0-135-2 kubenswrapper[2571]: W0805 17:12:52.896836    2571 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Aug 05 17:12:52.903385 ip-10-0-135-2 kubenswrapper[2571]: W0805 17:12:52.896838    2571 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Aug 05 17:12:52.903385 ip-10-0-135-2 kubenswrapper[2571]: W0805 17:12:52.896841    2571 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Aug 05 17:12:52.903385 ip-10-0-135-2 kubenswrapper[2571]: W0805 17:12:52.896843    2571 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Aug 05 17:12:52.903385 ip-10-0-135-2 kubenswrapper[2571]: W0805 17:12:52.896846    2571 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Aug 05 17:12:52.903385 ip-10-0-135-2 kubenswrapper[2571]: W0805 17:12:52.896848    2571 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Aug 05 17:12:52.903385 ip-10-0-135-2 kubenswrapper[2571]: W0805 17:12:52.896851    2571 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Aug 05 17:12:52.903385 ip-10-0-135-2 kubenswrapper[2571]: W0805 17:12:52.896853    2571 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Aug 05 17:12:52.903385 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:12:52.896858    2571 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Aug 05 17:12:52.903385 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:12:52.896948    2571 server.go:962] "Client rotation is on, will bootstrap in background"
Aug 05 17:12:52.903385 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:12:52.899090    2571 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Aug 05 17:12:52.903385 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:12:52.900333    2571 server.go:1019] "Starting client certificate rotation"
Aug 05 17:12:52.903385 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:12:52.900439    2571 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Aug 05 17:12:52.903385 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:12:52.901083    2571 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Aug 05 17:12:52.903724 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:12:52.902629    2571 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Aug 05 17:12:52.907043 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:12:52.907029    2571 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Aug 05 17:12:52.915687 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:12:52.915665    2571 log.go:25] "Validated CRI v1 runtime API"
Aug 05 17:12:52.917902 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:12:52.917889    2571 log.go:25] "Validated CRI v1 image API"
Aug 05 17:12:52.919259 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:12:52.919240    2571 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Aug 05 17:12:52.921472 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:12:52.921454    2571 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 e60b773d-4148-4d06-8f9c-372fdb8fae78:/dev/nvme0n1p3 ee02ed34-8e06-4278-9248-c2f754b7bf11:/dev/nvme0n1p4]
Aug 05 17:12:52.921524 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:12:52.921472    2571 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Aug 05 17:12:52.927144 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:12:52.927026    2571 manager.go:216] Machine: {Timestamp:2026-08-05 17:12:52.925729578 +0000 UTC m=+0.293095912 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3096111 MemoryCapacity:33164492800 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec21206526f4bd2c7b44cfa23c0e0bb9 SystemUUID:ec212065-26f4-bd2c-7b44-cfa23c0e0bb9 BootID:bd36aca6-72d6-42b0-bbee-dfd1b1d8be8f Filesystems:[{Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582246400 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582246400 Type:vfs Inodes:4048400 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632898560 Type:vfs Inodes:819200 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:c6:02:2c:5e:fb Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:c6:02:2c:5e:fb Speed:0 Mtu:9001} {Name:ovs-system MacAddress:da:d6:ef:5c:cf:8f Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164492800 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Aug 05 17:12:52.927144 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:12:52.927138    2571 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Aug 05 17:12:52.927253 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:12:52.927206    2571 manager.go:232] Version: {KernelVersion:5.14.0-570.131.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260727-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Aug 05 17:12:52.929909 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:12:52.929887    2571 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Aug 05 17:12:52.930032 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:12:52.929913    2571 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-135-2.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Aug 05 17:12:52.930587 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:12:52.930575    2571 topology_manager.go:138] "Creating topology manager with none policy"
Aug 05 17:12:52.930587 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:12:52.930589    2571 container_manager_linux.go:309] "Creating device plugin manager"
Aug 05 17:12:52.930676 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:12:52.930596    2571 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Aug 05 17:12:52.930676 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:12:52.930609    2571 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Aug 05 17:12:52.931452 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:12:52.931441    2571 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Aug 05 17:12:52.936236 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:12:52.936224    2571 state_mem.go:36] "Initialized new in-memory state store"
Aug 05 17:12:52.936355 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:12:52.936343    2571 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Aug 05 17:12:52.936965 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:12:52.936954    2571 kubelet.go:486] "Attempting to sync node with API server"
Aug 05 17:12:52.937013 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:12:52.936970    2571 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Aug 05 17:12:52.937013 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:12:52.936992    2571 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Aug 05 17:12:52.937013 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:12:52.937006    2571 kubelet.go:398] "Adding apiserver pod source"
Aug 05 17:12:52.937137 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:12:52.937018    2571 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Aug 05 17:12:52.937421 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:12:52.937408    2571 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Aug 05 17:12:52.937459 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:12:52.937433    2571 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Aug 05 17:12:52.939321 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:12:52.939291    2571 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.10-2.rhaos4.21.git83ee25b.el9" apiVersion="v1"
Aug 05 17:12:52.941403 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:12:52.941386    2571 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Aug 05 17:12:52.941476 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:12:52.941420    2571 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Aug 05 17:12:52.942985 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:12:52.942972    2571 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Aug 05 17:12:52.943033 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:12:52.942990    2571 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Aug 05 17:12:52.943033 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:12:52.942996    2571 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Aug 05 17:12:52.943033 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:12:52.943001    2571 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Aug 05 17:12:52.943033 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:12:52.943010    2571 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Aug 05 17:12:52.943033 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:12:52.943016    2571 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Aug 05 17:12:52.943033 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:12:52.943024    2571 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Aug 05 17:12:52.943033 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:12:52.943029    2571 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Aug 05 17:12:52.943231 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:12:52.943037    2571 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Aug 05 17:12:52.943231 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:12:52.943045    2571 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Aug 05 17:12:52.943231 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:12:52.943062    2571 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Aug 05 17:12:52.943231 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:12:52.943067    2571 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Aug 05 17:12:52.943880 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:12:52.943871    2571 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Aug 05 17:12:52.943880 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:12:52.943881    2571 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Aug 05 17:12:52.944489 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:12:52.944471    2571 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Aug 05 17:12:52.946890 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:12:52.946875    2571 server.go:1268] "Started kubelet"
Aug 05 17:12:52.947089 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:12:52.947036    2571 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Aug 05 17:12:52.947607 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:12:52.947572    2571 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Aug 05 17:12:52.947688 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:12:52.947615    2571 server_v1.go:49] "podresources" method="list" useActivePods=true
Aug 05 17:12:52.947742 ip-10-0-135-2 systemd[1]: Started Kubernetes Kubelet.
Aug 05 17:12:52.952265 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:12:52.952246    2571 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Aug 05 17:12:52.952838 ip-10-0-135-2 kubenswrapper[2571]: E0805 17:12:52.952816    2571 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Aug 05 17:12:52.952978 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:12:52.952962    2571 server.go:310] "Adding debug handlers to kubelet server"
Aug 05 17:12:52.955244 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:12:52.955227    2571 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-135-2.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Aug 05 17:12:52.955319 ip-10-0-135-2 kubenswrapper[2571]: E0805 17:12:52.955230    2571 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-135-2.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Aug 05 17:12:52.955720 ip-10-0-135-2 kubenswrapper[2571]: E0805 17:12:52.955694    2571 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Aug 05 17:12:52.956506 ip-10-0-135-2 kubenswrapper[2571]: E0805 17:12:52.955718    2571 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-135-2.ec2.internal.18c8f809bc1d7c8b  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-135-2.ec2.internal,UID:ip-10-0-135-2.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-135-2.ec2.internal,},FirstTimestamp:2026-08-05 17:12:52.946844811 +0000 UTC m=+0.314211146,LastTimestamp:2026-08-05 17:12:52.946844811 +0000 UTC m=+0.314211146,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-135-2.ec2.internal,}"
Aug 05 17:12:52.956644 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:12:52.956620    2571 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Aug 05 17:12:52.956644 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:12:52.956635    2571 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Aug 05 17:12:52.956764 ip-10-0-135-2 kubenswrapper[2571]: E0805 17:12:52.956717    2571 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-2.ec2.internal\" not found"
Aug 05 17:12:52.957216 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:12:52.957200    2571 volume_manager.go:311] "The desired_state_of_world populator starts"
Aug 05 17:12:52.957216 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:12:52.957218    2571 volume_manager.go:313] "Starting Kubelet Volume Manager"
Aug 05 17:12:52.957351 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:12:52.957200    2571 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Aug 05 17:12:52.957351 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:12:52.957318    2571 reconstruct.go:97] "Volume reconstruction finished"
Aug 05 17:12:52.957351 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:12:52.957327    2571 reconciler.go:29] "Reconciler: start to sync state"
Aug 05 17:12:52.957559 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:12:52.957543    2571 factory.go:55] Registering systemd factory
Aug 05 17:12:52.957559 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:12:52.957561    2571 factory.go:223] Registration of the systemd container factory successfully
Aug 05 17:12:52.957744 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:12:52.957728    2571 factory.go:153] Registering CRI-O factory
Aug 05 17:12:52.957744 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:12:52.957745    2571 factory.go:223] Registration of the crio container factory successfully
Aug 05 17:12:52.957881 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:12:52.957795    2571 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Aug 05 17:12:52.957881 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:12:52.957821    2571 factory.go:103] Registering Raw factory
Aug 05 17:12:52.957881 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:12:52.957835    2571 manager.go:1196] Started watching for new ooms in manager
Aug 05 17:12:52.958230 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:12:52.958218    2571 manager.go:349] Starting recovery of all containers
Aug 05 17:12:52.959410 ip-10-0-135-2 kubenswrapper[2571]: E0805 17:12:52.959366    2571 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-135-2.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Aug 05 17:12:52.959504 ip-10-0-135-2 kubenswrapper[2571]: E0805 17:12:52.959465    2571 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Aug 05 17:12:52.963653 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:12:52.963605    2571 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Aug 05 17:12:52.965943 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:12:52.965924    2571 manager.go:354] Recovery completed
Aug 05 17:12:52.970219 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:12:52.970203    2571 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Aug 05 17:12:52.971926 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:12:52.971905    2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-2.ec2.internal" event="NodeHasSufficientMemory"
Aug 05 17:12:52.972007 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:12:52.971938    2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-2.ec2.internal" event="NodeHasNoDiskPressure"
Aug 05 17:12:52.972007 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:12:52.971953    2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-2.ec2.internal" event="NodeHasSufficientPID"
Aug 05 17:12:52.972436 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:12:52.972424    2571 cpu_manager.go:222] "Starting CPU manager" policy="none"
Aug 05 17:12:52.972436 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:12:52.972436    2571 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Aug 05 17:12:52.972551 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:12:52.972450    2571 state_mem.go:36] "Initialized new in-memory state store"
Aug 05 17:12:52.972551 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:12:52.972508    2571 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Aug 05 17:12:52.973267 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:12:52.973190    2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-2.ec2.internal" event="NodeHasSufficientMemory"
Aug 05 17:12:52.973267 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:12:52.973215    2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-2.ec2.internal" event="NodeHasNoDiskPressure"
Aug 05 17:12:52.973267 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:12:52.973225    2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-2.ec2.internal" event="NodeHasSufficientPID"
Aug 05 17:12:52.974358 ip-10-0-135-2 kubenswrapper[2571]: E0805 17:12:52.974286    2571 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-135-2.ec2.internal.18c8f809bd9c2cbf  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-135-2.ec2.internal,UID:ip-10-0-135-2.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-135-2.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-135-2.ec2.internal,},FirstTimestamp:2026-08-05 17:12:52.971924671 +0000 UTC m=+0.339291007,LastTimestamp:2026-08-05 17:12:52.971924671 +0000 UTC m=+0.339291007,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-135-2.ec2.internal,}"
Aug 05 17:12:52.974597 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:12:52.974585    2571 policy_none.go:49] "None policy: Start"
Aug 05 17:12:52.974640 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:12:52.974601    2571 memory_manager.go:187] "Starting memorymanager" policy="None"
Aug 05 17:12:52.974640 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:12:52.974611    2571 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Aug 05 17:12:52.976778 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:12:52.976767    2571 policy_none.go:47] "Start"
Aug 05 17:12:52.985886 ip-10-0-135-2 kubenswrapper[2571]: E0805 17:12:52.985831    2571 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-135-2.ec2.internal.18c8f809bd9c7998  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-135-2.ec2.internal,UID:ip-10-0-135-2.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-135-2.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-135-2.ec2.internal,},FirstTimestamp:2026-08-05 17:12:52.971944344 +0000 UTC m=+0.339310684,LastTimestamp:2026-08-05 17:12:52.971944344 +0000 UTC m=+0.339310684,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-135-2.ec2.internal,}"
Aug 05 17:12:52.996531 ip-10-0-135-2 kubenswrapper[2571]: E0805 17:12:52.995913    2571 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-135-2.ec2.internal.18c8f809bd9cb328  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-135-2.ec2.internal,UID:ip-10-0-135-2.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-135-2.ec2.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-135-2.ec2.internal,},FirstTimestamp:2026-08-05 17:12:52.97195908 +0000 UTC m=+0.339325416,LastTimestamp:2026-08-05 17:12:52.97195908 +0000 UTC m=+0.339325416,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-135-2.ec2.internal,}"
Aug 05 17:12:52.996531 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:12:52.996353    2571 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-h88jw"
Aug 05 17:12:53.005627 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:12:53.005609    2571 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-h88jw"
Aug 05 17:12:53.007826 ip-10-0-135-2 kubenswrapper[2571]: E0805 17:12:53.007772    2571 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-135-2.ec2.internal.18c8f809bd9c2cbf\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-135-2.ec2.internal.18c8f809bd9c2cbf  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-135-2.ec2.internal,UID:ip-10-0-135-2.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-135-2.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-135-2.ec2.internal,},FirstTimestamp:2026-08-05 17:12:52.971924671 +0000 UTC m=+0.339291007,LastTimestamp:2026-08-05 17:12:52.973200762 +0000 UTC m=+0.340567097,Count:2,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-135-2.ec2.internal,}"
Aug 05 17:12:53.024900 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:12:53.024884    2571 manager.go:337] "Starting Device Plugin manager"
Aug 05 17:12:53.024963 ip-10-0-135-2 kubenswrapper[2571]: E0805 17:12:53.024912    2571 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Aug 05 17:12:53.024963 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:12:53.024921    2571 server.go:87] "Starting device plugin registration server"
Aug 05 17:12:53.025153 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:12:53.025142    2571 eviction_manager.go:189] "Eviction manager: starting control loop"
Aug 05 17:12:53.025201 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:12:53.025155    2571 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Aug 05 17:12:53.025266 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:12:53.025254    2571 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Aug 05 17:12:53.025368 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:12:53.025323    2571 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Aug 05 17:12:53.025368 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:12:53.025334    2571 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Aug 05 17:12:53.025833 ip-10-0-135-2 kubenswrapper[2571]: E0805 17:12:53.025814    2571 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Aug 05 17:12:53.025899 ip-10-0-135-2 kubenswrapper[2571]: E0805 17:12:53.025857    2571 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-135-2.ec2.internal\" not found"
Aug 05 17:12:53.125649 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:12:53.125597    2571 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Aug 05 17:12:53.127774 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:12:53.126393    2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-2.ec2.internal" event="NodeHasSufficientMemory"
Aug 05 17:12:53.127774 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:12:53.126425    2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-2.ec2.internal" event="NodeHasNoDiskPressure"
Aug 05 17:12:53.127774 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:12:53.126440    2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-2.ec2.internal" event="NodeHasSufficientPID"
Aug 05 17:12:53.127774 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:12:53.126471    2571 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-135-2.ec2.internal"
Aug 05 17:12:53.130910 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:12:53.130895    2571 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Aug 05 17:12:53.130977 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:12:53.130930    2571 status_manager.go:244] "Starting to sync pod status with apiserver"
Aug 05 17:12:53.130977 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:12:53.130948    2571 kubelet.go:2448] "Starting kubelet main sync loop"
Aug 05 17:12:53.131051 ip-10-0-135-2 kubenswrapper[2571]: E0805 17:12:53.130976    2571 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Aug 05 17:12:53.133065 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:12:53.133049    2571 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Aug 05 17:12:53.135006 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:12:53.134990    2571 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-135-2.ec2.internal"
Aug 05 17:12:53.135064 ip-10-0-135-2 kubenswrapper[2571]: E0805 17:12:53.135009    2571 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-135-2.ec2.internal\": node \"ip-10-0-135-2.ec2.internal\" not found"
Aug 05 17:12:53.187455 ip-10-0-135-2 kubenswrapper[2571]: E0805 17:12:53.187435    2571 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-2.ec2.internal\" not found"
Aug 05 17:12:53.231533 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:12:53.231497    2571 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-2.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-135-2.ec2.internal"]
Aug 05 17:12:53.231621 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:12:53.231584    2571 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Aug 05 17:12:53.233015 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:12:53.233000    2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-2.ec2.internal" event="NodeHasSufficientMemory"
Aug 05 17:12:53.233090 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:12:53.233028    2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-2.ec2.internal" event="NodeHasNoDiskPressure"
Aug 05 17:12:53.233090 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:12:53.233039    2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-2.ec2.internal" event="NodeHasSufficientPID"
Aug 05 17:12:53.235255 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:12:53.235242    2571 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Aug 05 17:12:53.235398 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:12:53.235370    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-2.ec2.internal"
Aug 05 17:12:53.235462 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:12:53.235418    2571 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Aug 05 17:12:53.236234 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:12:53.235924    2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-2.ec2.internal" event="NodeHasSufficientMemory"
Aug 05 17:12:53.236234 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:12:53.235955    2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-2.ec2.internal" event="NodeHasNoDiskPressure"
Aug 05 17:12:53.236234 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:12:53.235966    2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-2.ec2.internal" event="NodeHasSufficientPID"
Aug 05 17:12:53.236234 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:12:53.235924    2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-2.ec2.internal" event="NodeHasSufficientMemory"
Aug 05 17:12:53.236234 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:12:53.236040    2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-2.ec2.internal" event="NodeHasNoDiskPressure"
Aug 05 17:12:53.236234 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:12:53.236055    2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-2.ec2.internal" event="NodeHasSufficientPID"
Aug 05 17:12:53.238510 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:12:53.238494    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-2.ec2.internal"
Aug 05 17:12:53.238582 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:12:53.238521    2571 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Aug 05 17:12:53.239426 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:12:53.239388    2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-2.ec2.internal" event="NodeHasSufficientMemory"
Aug 05 17:12:53.239534 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:12:53.239442    2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-2.ec2.internal" event="NodeHasNoDiskPressure"
Aug 05 17:12:53.239534 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:12:53.239458    2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-2.ec2.internal" event="NodeHasSufficientPID"
Aug 05 17:12:53.259516 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:12:53.259495    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/bb3fac1e9119c8f4580f4cf12eb2ffb1-config\") pod \"kube-apiserver-proxy-ip-10-0-135-2.ec2.internal\" (UID: \"bb3fac1e9119c8f4580f4cf12eb2ffb1\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-135-2.ec2.internal"
Aug 05 17:12:53.259588 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:12:53.259521    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/300ebfcee102c70ab72580814e7ccaa4-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-135-2.ec2.internal\" (UID: \"300ebfcee102c70ab72580814e7ccaa4\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-2.ec2.internal"
Aug 05 17:12:53.259588 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:12:53.259540    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/300ebfcee102c70ab72580814e7ccaa4-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-135-2.ec2.internal\" (UID: \"300ebfcee102c70ab72580814e7ccaa4\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-2.ec2.internal"
Aug 05 17:12:53.267182 ip-10-0-135-2 kubenswrapper[2571]: E0805 17:12:53.267163    2571 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-135-2.ec2.internal\" not found" node="ip-10-0-135-2.ec2.internal"
Aug 05 17:12:53.271823 ip-10-0-135-2 kubenswrapper[2571]: E0805 17:12:53.271808    2571 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-135-2.ec2.internal\" not found" node="ip-10-0-135-2.ec2.internal"
Aug 05 17:12:53.288072 ip-10-0-135-2 kubenswrapper[2571]: E0805 17:12:53.288053    2571 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-2.ec2.internal\" not found"
Aug 05 17:12:53.360292 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:12:53.360274    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/300ebfcee102c70ab72580814e7ccaa4-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-135-2.ec2.internal\" (UID: \"300ebfcee102c70ab72580814e7ccaa4\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-2.ec2.internal"
Aug 05 17:12:53.360408 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:12:53.360299    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/bb3fac1e9119c8f4580f4cf12eb2ffb1-config\") pod \"kube-apiserver-proxy-ip-10-0-135-2.ec2.internal\" (UID: \"bb3fac1e9119c8f4580f4cf12eb2ffb1\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-135-2.ec2.internal"
Aug 05 17:12:53.360408 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:12:53.360317    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/300ebfcee102c70ab72580814e7ccaa4-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-135-2.ec2.internal\" (UID: \"300ebfcee102c70ab72580814e7ccaa4\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-2.ec2.internal"
Aug 05 17:12:53.360408 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:12:53.360359    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/300ebfcee102c70ab72580814e7ccaa4-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-135-2.ec2.internal\" (UID: \"300ebfcee102c70ab72580814e7ccaa4\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-2.ec2.internal"
Aug 05 17:12:53.360520 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:12:53.360416    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/300ebfcee102c70ab72580814e7ccaa4-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-135-2.ec2.internal\" (UID: \"300ebfcee102c70ab72580814e7ccaa4\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-2.ec2.internal"
Aug 05 17:12:53.360520 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:12:53.360416    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/bb3fac1e9119c8f4580f4cf12eb2ffb1-config\") pod \"kube-apiserver-proxy-ip-10-0-135-2.ec2.internal\" (UID: \"bb3fac1e9119c8f4580f4cf12eb2ffb1\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-135-2.ec2.internal"
Aug 05 17:12:53.388828 ip-10-0-135-2 kubenswrapper[2571]: E0805 17:12:53.388784    2571 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-2.ec2.internal\" not found"
Aug 05 17:12:53.490221 ip-10-0-135-2 kubenswrapper[2571]: E0805 17:12:53.490187    2571 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-2.ec2.internal\" not found"
Aug 05 17:12:53.569540 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:12:53.569512    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-2.ec2.internal"
Aug 05 17:12:53.575499 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:12:53.575008    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-2.ec2.internal"
Aug 05 17:12:53.591208 ip-10-0-135-2 kubenswrapper[2571]: E0805 17:12:53.591185    2571 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-2.ec2.internal\" not found"
Aug 05 17:12:53.692804 ip-10-0-135-2 kubenswrapper[2571]: E0805 17:12:53.692707    2571 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-2.ec2.internal\" not found"
Aug 05 17:12:53.793495 ip-10-0-135-2 kubenswrapper[2571]: E0805 17:12:53.793458    2571 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-2.ec2.internal\" not found"
Aug 05 17:12:53.895555 ip-10-0-135-2 kubenswrapper[2571]: E0805 17:12:53.895520    2571 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-2.ec2.internal\" not found"
Aug 05 17:12:53.900037 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:12:53.900019    2571 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Aug 05 17:12:53.900211 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:12:53.900188    2571 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Aug 05 17:12:53.996399 ip-10-0-135-2 kubenswrapper[2571]: E0805 17:12:53.996315    2571 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-2.ec2.internal\" not found"
Aug 05 17:12:54.011503 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:12:54.011466    2571 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-08-04 17:07:53 +0000 UTC" deadline="2028-05-13 13:13:11.446580966 +0000 UTC"
Aug 05 17:12:54.011503 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:12:54.011501    2571 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="15524h0m17.435082735s"
Aug 05 17:12:54.097440 ip-10-0-135-2 kubenswrapper[2571]: E0805 17:12:54.097406    2571 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-2.ec2.internal\" not found"
Aug 05 17:12:54.140207 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:12:54.140181    2571 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Aug 05 17:12:54.142412 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:12:54.142392    2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-2.ec2.internal" event="NodeHasSufficientMemory"
Aug 05 17:12:54.142535 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:12:54.142428    2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-2.ec2.internal" event="NodeHasNoDiskPressure"
Aug 05 17:12:54.142535 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:12:54.142441    2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-2.ec2.internal" event="NodeHasSufficientPID"
Aug 05 17:12:54.147738 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:12:54.147720    2571 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Aug 05 17:12:54.198590 ip-10-0-135-2 kubenswrapper[2571]: E0805 17:12:54.198565    2571 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-2.ec2.internal\" not found"
Aug 05 17:12:54.240768 ip-10-0-135-2 kubenswrapper[2571]: W0805 17:12:54.240739    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbb3fac1e9119c8f4580f4cf12eb2ffb1.slice/crio-cd1cdc0023080df0a6663aedac8a8c50521b25858e8c5f4e397d3b56f35851e3 WatchSource:0}: Error finding container cd1cdc0023080df0a6663aedac8a8c50521b25858e8c5f4e397d3b56f35851e3: Status 404 returned error can't find the container with id cd1cdc0023080df0a6663aedac8a8c50521b25858e8c5f4e397d3b56f35851e3
Aug 05 17:12:54.244942 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:12:54.244925    2571 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Aug 05 17:12:54.298721 ip-10-0-135-2 kubenswrapper[2571]: E0805 17:12:54.298696    2571 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-2.ec2.internal\" not found"
Aug 05 17:12:54.358387 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:12:54.358366    2571 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Aug 05 17:12:54.372388 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:12:54.372373    2571 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Aug 05 17:12:54.377844 ip-10-0-135-2 kubenswrapper[2571]: W0805 17:12:54.377821    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod300ebfcee102c70ab72580814e7ccaa4.slice/crio-3e328c61698e8c2430035058081e4517c91b5893c5e264502c215714ce49bb01 WatchSource:0}: Error finding container 3e328c61698e8c2430035058081e4517c91b5893c5e264502c215714ce49bb01: Status 404 returned error can't find the container with id 3e328c61698e8c2430035058081e4517c91b5893c5e264502c215714ce49bb01
Aug 05 17:12:54.458435 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:12:54.458411    2571 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-2.ec2.internal"
Aug 05 17:12:54.490564 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:12:54.490542    2571 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Aug 05 17:12:54.491600 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:12:54.491584    2571 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-2.ec2.internal"
Aug 05 17:12:54.526720 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:12:54.526664    2571 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Aug 05 17:12:54.939684 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:12:54.939602    2571 apiserver.go:52] "Watching apiserver"
Aug 05 17:12:54.949626 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:12:54.949600    2571 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Aug 05 17:12:54.951438 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:12:54.951407    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-multus/multus-vzp52","openshift-network-operator/iptables-alerter-txqn4","openshift-ovn-kubernetes/ovnkube-node-tsjgl","kube-system/global-pull-secret-syncer-9jxp4","kube-system/konnectivity-agent-zmr54","kube-system/kube-apiserver-proxy-ip-10-0-135-2.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7h5q8","openshift-cluster-node-tuning-operator/tuned-nb82s","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-2.ec2.internal","openshift-multus/network-metrics-daemon-677x7","openshift-network-diagnostics/network-check-target-whps7","openshift-dns/node-resolver-qz2cd","openshift-image-registry/node-ca-lrdmn","openshift-multus/multus-additional-cni-plugins-mgfr4"]
Aug 05 17:12:54.954153 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:12:54.954130    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-vzp52"
Aug 05 17:12:54.956561 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:12:54.956520    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-txqn4"
Aug 05 17:12:54.957421 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:12:54.957240    2571 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Aug 05 17:12:54.959100 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:12:54.958885    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-mgfr4"
Aug 05 17:12:54.961280 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:12:54.961209    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-lrdmn"
Aug 05 17:12:54.961686 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:12:54.961663    2571 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Aug 05 17:12:54.962353 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:12:54.962124    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Aug 05 17:12:54.962353 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:12:54.962124    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Aug 05 17:12:54.962353 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:12:54.962236    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Aug 05 17:12:54.962353 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:12:54.962239    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Aug 05 17:12:54.962353 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:12:54.962284    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Aug 05 17:12:54.962353 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:12:54.962293    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-mf6jf\""
Aug 05 17:12:54.962741 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:12:54.962494    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Aug 05 17:12:54.962741 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:12:54.962560    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Aug 05 17:12:54.962741 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:12:54.962616    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Aug 05 17:12:54.962900 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:12:54.962763    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-jg4k4\""
Aug 05 17:12:54.962900 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:12:54.962887    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-hhbh6\""
Aug 05 17:12:54.963010 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:12:54.962907    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Aug 05 17:12:54.965065 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:12:54.964346    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Aug 05 17:12:54.965065 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:12:54.964611    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-brj7n\""
Aug 05 17:12:54.965065 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:12:54.964842    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Aug 05 17:12:54.965065 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:12:54.964921    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Aug 05 17:12:54.965351 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:12:54.965102    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-qz2cd"
Aug 05 17:12:54.967395 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:12:54.967369    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7h5q8"
Aug 05 17:12:54.969184 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:12:54.968605    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Aug 05 17:12:54.969184 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:12:54.968648    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Aug 05 17:12:54.969184 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:12:54.968814    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c7v68\" (UniqueName: \"kubernetes.io/projected/100f6b72-e895-453f-8647-1f615c2fb780-kube-api-access-c7v68\") pod \"multus-additional-cni-plugins-mgfr4\" (UID: \"100f6b72-e895-453f-8647-1f615c2fb780\") " pod="openshift-multus/multus-additional-cni-plugins-mgfr4"
Aug 05 17:12:54.969184 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:12:54.968848    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/99b228eb-7a25-47a1-a90d-06b62e8f17e0-os-release\") pod \"multus-vzp52\" (UID: \"99b228eb-7a25-47a1-a90d-06b62e8f17e0\") " pod="openshift-multus/multus-vzp52"
Aug 05 17:12:54.969184 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:12:54.968876    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/99b228eb-7a25-47a1-a90d-06b62e8f17e0-multus-daemon-config\") pod \"multus-vzp52\" (UID: \"99b228eb-7a25-47a1-a90d-06b62e8f17e0\") " pod="openshift-multus/multus-vzp52"
Aug 05 17:12:54.969184 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:12:54.968852    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-trblj\""
Aug 05 17:12:54.969184 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:12:54.968902    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/100f6b72-e895-453f-8647-1f615c2fb780-system-cni-dir\") pod \"multus-additional-cni-plugins-mgfr4\" (UID: \"100f6b72-e895-453f-8647-1f615c2fb780\") " pod="openshift-multus/multus-additional-cni-plugins-mgfr4"
Aug 05 17:12:54.969184 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:12:54.968929    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/99b228eb-7a25-47a1-a90d-06b62e8f17e0-multus-socket-dir-parent\") pod \"multus-vzp52\" (UID: \"99b228eb-7a25-47a1-a90d-06b62e8f17e0\") " pod="openshift-multus/multus-vzp52"
Aug 05 17:12:54.969184 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:12:54.968952    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/99b228eb-7a25-47a1-a90d-06b62e8f17e0-host-var-lib-cni-bin\") pod \"multus-vzp52\" (UID: \"99b228eb-7a25-47a1-a90d-06b62e8f17e0\") " pod="openshift-multus/multus-vzp52"
Aug 05 17:12:54.969184 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:12:54.968978    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/99b228eb-7a25-47a1-a90d-06b62e8f17e0-multus-conf-dir\") pod \"multus-vzp52\" (UID: \"99b228eb-7a25-47a1-a90d-06b62e8f17e0\") " pod="openshift-multus/multus-vzp52"
Aug 05 17:12:54.969184 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:12:54.969001    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/99b228eb-7a25-47a1-a90d-06b62e8f17e0-cni-binary-copy\") pod \"multus-vzp52\" (UID: \"99b228eb-7a25-47a1-a90d-06b62e8f17e0\") " pod="openshift-multus/multus-vzp52"
Aug 05 17:12:54.969184 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:12:54.969024    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/99b228eb-7a25-47a1-a90d-06b62e8f17e0-host-run-netns\") pod \"multus-vzp52\" (UID: \"99b228eb-7a25-47a1-a90d-06b62e8f17e0\") " pod="openshift-multus/multus-vzp52"
Aug 05 17:12:54.969184 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:12:54.969049    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/99b228eb-7a25-47a1-a90d-06b62e8f17e0-host-run-multus-certs\") pod \"multus-vzp52\" (UID: \"99b228eb-7a25-47a1-a90d-06b62e8f17e0\") " pod="openshift-multus/multus-vzp52"
Aug 05 17:12:54.969184 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:12:54.969072    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/100f6b72-e895-453f-8647-1f615c2fb780-os-release\") pod \"multus-additional-cni-plugins-mgfr4\" (UID: \"100f6b72-e895-453f-8647-1f615c2fb780\") " pod="openshift-multus/multus-additional-cni-plugins-mgfr4"
Aug 05 17:12:54.969184 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:12:54.969112    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/99b228eb-7a25-47a1-a90d-06b62e8f17e0-multus-cni-dir\") pod \"multus-vzp52\" (UID: \"99b228eb-7a25-47a1-a90d-06b62e8f17e0\") " pod="openshift-multus/multus-vzp52"
Aug 05 17:12:54.969184 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:12:54.969143    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/99b228eb-7a25-47a1-a90d-06b62e8f17e0-cnibin\") pod \"multus-vzp52\" (UID: \"99b228eb-7a25-47a1-a90d-06b62e8f17e0\") " pod="openshift-multus/multus-vzp52"
Aug 05 17:12:54.969184 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:12:54.969169    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/99b228eb-7a25-47a1-a90d-06b62e8f17e0-hostroot\") pod \"multus-vzp52\" (UID: \"99b228eb-7a25-47a1-a90d-06b62e8f17e0\") " pod="openshift-multus/multus-vzp52"
Aug 05 17:12:54.969184 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:12:54.969194    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/99b228eb-7a25-47a1-a90d-06b62e8f17e0-etc-kubernetes\") pod \"multus-vzp52\" (UID: \"99b228eb-7a25-47a1-a90d-06b62e8f17e0\") " pod="openshift-multus/multus-vzp52"
Aug 05 17:12:54.970141 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:12:54.969220    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/b934c28e-a416-4ddf-8b90-0c9acdd8ce4c-iptables-alerter-script\") pod \"iptables-alerter-txqn4\" (UID: \"b934c28e-a416-4ddf-8b90-0c9acdd8ce4c\") " pod="openshift-network-operator/iptables-alerter-txqn4"
Aug 05 17:12:54.970141 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:12:54.969243    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gnsw6\" (UniqueName: \"kubernetes.io/projected/b934c28e-a416-4ddf-8b90-0c9acdd8ce4c-kube-api-access-gnsw6\") pod \"iptables-alerter-txqn4\" (UID: \"b934c28e-a416-4ddf-8b90-0c9acdd8ce4c\") " pod="openshift-network-operator/iptables-alerter-txqn4"
Aug 05 17:12:54.970141 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:12:54.969571    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-nb82s"
Aug 05 17:12:54.970141 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:12:54.969639    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-wb6cr\""
Aug 05 17:12:54.970141 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:12:54.969269    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/100f6b72-e895-453f-8647-1f615c2fb780-cnibin\") pod \"multus-additional-cni-plugins-mgfr4\" (UID: \"100f6b72-e895-453f-8647-1f615c2fb780\") " pod="openshift-multus/multus-additional-cni-plugins-mgfr4"
Aug 05 17:12:54.970141 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:12:54.969839    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/100f6b72-e895-453f-8647-1f615c2fb780-cni-binary-copy\") pod \"multus-additional-cni-plugins-mgfr4\" (UID: \"100f6b72-e895-453f-8647-1f615c2fb780\") " pod="openshift-multus/multus-additional-cni-plugins-mgfr4"
Aug 05 17:12:54.970141 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:12:54.969854    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Aug 05 17:12:54.970141 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:12:54.969858    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/99b228eb-7a25-47a1-a90d-06b62e8f17e0-host-var-lib-kubelet\") pod \"multus-vzp52\" (UID: \"99b228eb-7a25-47a1-a90d-06b62e8f17e0\") " pod="openshift-multus/multus-vzp52"
Aug 05 17:12:54.970141 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:12:54.969896    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/100f6b72-e895-453f-8647-1f615c2fb780-tuning-conf-dir\") pod \"multus-additional-cni-plugins-mgfr4\" (UID: \"100f6b72-e895-453f-8647-1f615c2fb780\") " pod="openshift-multus/multus-additional-cni-plugins-mgfr4"
Aug 05 17:12:54.970141 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:12:54.969916    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/100f6b72-e895-453f-8647-1f615c2fb780-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-mgfr4\" (UID: \"100f6b72-e895-453f-8647-1f615c2fb780\") " pod="openshift-multus/multus-additional-cni-plugins-mgfr4"
Aug 05 17:12:54.970141 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:12:54.969932    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/99b228eb-7a25-47a1-a90d-06b62e8f17e0-system-cni-dir\") pod \"multus-vzp52\" (UID: \"99b228eb-7a25-47a1-a90d-06b62e8f17e0\") " pod="openshift-multus/multus-vzp52"
Aug 05 17:12:54.970141 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:12:54.969948    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/99b228eb-7a25-47a1-a90d-06b62e8f17e0-host-run-k8s-cni-cncf-io\") pod \"multus-vzp52\" (UID: \"99b228eb-7a25-47a1-a90d-06b62e8f17e0\") " pod="openshift-multus/multus-vzp52"
Aug 05 17:12:54.970141 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:12:54.969968    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/99b228eb-7a25-47a1-a90d-06b62e8f17e0-host-var-lib-cni-multus\") pod \"multus-vzp52\" (UID: \"99b228eb-7a25-47a1-a90d-06b62e8f17e0\") " pod="openshift-multus/multus-vzp52"
Aug 05 17:12:54.970141 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:12:54.969991    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/b934c28e-a416-4ddf-8b90-0c9acdd8ce4c-host-slash\") pod \"iptables-alerter-txqn4\" (UID: \"b934c28e-a416-4ddf-8b90-0c9acdd8ce4c\") " pod="openshift-network-operator/iptables-alerter-txqn4"
Aug 05 17:12:54.970141 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:12:54.970006    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/100f6b72-e895-453f-8647-1f615c2fb780-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-mgfr4\" (UID: \"100f6b72-e895-453f-8647-1f615c2fb780\") " pod="openshift-multus/multus-additional-cni-plugins-mgfr4"
Aug 05 17:12:54.970141 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:12:54.970021    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bm5cq\" (UniqueName: \"kubernetes.io/projected/99b228eb-7a25-47a1-a90d-06b62e8f17e0-kube-api-access-bm5cq\") pod \"multus-vzp52\" (UID: \"99b228eb-7a25-47a1-a90d-06b62e8f17e0\") " pod="openshift-multus/multus-vzp52"
Aug 05 17:12:54.970141 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:12:54.970138    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Aug 05 17:12:54.971005 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:12:54.970751    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Aug 05 17:12:54.972057 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:12:54.972036    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-677x7"
Aug 05 17:12:54.972156 ip-10-0-135-2 kubenswrapper[2571]: E0805 17:12:54.972115    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-677x7" podUID="da6f032b-35a1-4c51-b115-defaf9b2a9a5"
Aug 05 17:12:54.973085 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:12:54.972963    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Aug 05 17:12:54.973375 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:12:54.973182    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-6rjrh\""
Aug 05 17:12:54.973375 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:12:54.973193    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Aug 05 17:12:54.974682 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:12:54.974636    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-whps7"
Aug 05 17:12:54.974779 ip-10-0-135-2 kubenswrapper[2571]: E0805 17:12:54.974721    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-whps7" podUID="a78df3c5-f92c-4dc6-a56c-a794929aa911"
Aug 05 17:12:54.977806 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:12:54.977731    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-zmr54"
Aug 05 17:12:54.980257 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:12:54.980123    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9jxp4"
Aug 05 17:12:54.980257 ip-10-0-135-2 kubenswrapper[2571]: E0805 17:12:54.980205    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9jxp4" podUID="f2ecc651-01ec-4fa1-a7c1-780b3a963996"
Aug 05 17:12:54.981036 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:12:54.981013    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Aug 05 17:12:54.981616 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:12:54.981347    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Aug 05 17:12:54.981616 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:12:54.981500    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-fd46s\""
Aug 05 17:12:54.983997 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:12:54.982995    2571 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Aug 05 17:12:54.984222 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:12:54.984204    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-tsjgl"
Aug 05 17:12:54.988401 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:12:54.988360    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Aug 05 17:12:54.988545 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:12:54.988477    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Aug 05 17:12:54.988545 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:12:54.988537    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-kssvc\""
Aug 05 17:12:54.988968 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:12:54.988946    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Aug 05 17:12:54.989204 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:12:54.989186    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Aug 05 17:12:54.989298 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:12:54.989223    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Aug 05 17:12:54.990018 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:12:54.989996    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Aug 05 17:12:55.041913 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:12:55.041892    2571 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-rjq74"
Aug 05 17:12:55.053188 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:12:55.053163    2571 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-rjq74"
Aug 05 17:12:55.059063 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:12:55.059043    2571 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Aug 05 17:12:55.070221 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:12:55.070201    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/d662bf46-8fcd-40b5-94fa-eded9b062699-ovnkube-config\") pod \"ovnkube-node-tsjgl\" (UID: \"d662bf46-8fcd-40b5-94fa-eded9b062699\") " pod="openshift-ovn-kubernetes/ovnkube-node-tsjgl"
Aug 05 17:12:55.070319 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:12:55.070236    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/61599c74-5517-43f4-bed8-a232408b44c0-serviceca\") pod \"node-ca-lrdmn\" (UID: \"61599c74-5517-43f4-bed8-a232408b44c0\") " pod="openshift-image-registry/node-ca-lrdmn"
Aug 05 17:12:55.070319 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:12:55.070255    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7xgvj\" (UniqueName: \"kubernetes.io/projected/a78df3c5-f92c-4dc6-a56c-a794929aa911-kube-api-access-7xgvj\") pod \"network-check-target-whps7\" (UID: \"a78df3c5-f92c-4dc6-a56c-a794929aa911\") " pod="openshift-network-diagnostics/network-check-target-whps7"
Aug 05 17:12:55.070319 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:12:55.070270    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/d662bf46-8fcd-40b5-94fa-eded9b062699-ovn-node-metrics-cert\") pod \"ovnkube-node-tsjgl\" (UID: \"d662bf46-8fcd-40b5-94fa-eded9b062699\") " pod="openshift-ovn-kubernetes/ovnkube-node-tsjgl"
Aug 05 17:12:55.070319 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:12:55.070291    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/b934c28e-a416-4ddf-8b90-0c9acdd8ce4c-iptables-alerter-script\") pod \"iptables-alerter-txqn4\" (UID: \"b934c28e-a416-4ddf-8b90-0c9acdd8ce4c\") " pod="openshift-network-operator/iptables-alerter-txqn4"
Aug 05 17:12:55.070319 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:12:55.070306    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/100f6b72-e895-453f-8647-1f615c2fb780-cnibin\") pod \"multus-additional-cni-plugins-mgfr4\" (UID: \"100f6b72-e895-453f-8647-1f615c2fb780\") " pod="openshift-multus/multus-additional-cni-plugins-mgfr4"
Aug 05 17:12:55.070718 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:12:55.070361    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/ac2f6cf0-11ca-47f3-8597-5cc1db578e78-lib-modules\") pod \"tuned-nb82s\" (UID: \"ac2f6cf0-11ca-47f3-8597-5cc1db578e78\") " pod="openshift-cluster-node-tuning-operator/tuned-nb82s"
Aug 05 17:12:55.070718 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:12:55.070410    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/100f6b72-e895-453f-8647-1f615c2fb780-cnibin\") pod \"multus-additional-cni-plugins-mgfr4\" (UID: \"100f6b72-e895-453f-8647-1f615c2fb780\") " pod="openshift-multus/multus-additional-cni-plugins-mgfr4"
Aug 05 17:12:55.070718 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:12:55.070413    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/df7e05c7-620a-4246-bd4b-0e2c2009335b-etc-selinux\") pod \"aws-ebs-csi-driver-node-7h5q8\" (UID: \"df7e05c7-620a-4246-bd4b-0e2c2009335b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7h5q8"
Aug 05 17:12:55.070718 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:12:55.070470    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/b0520f9c-efe2-43b9-8623-411b55bbc874-agent-certs\") pod \"konnectivity-agent-zmr54\" (UID: \"b0520f9c-efe2-43b9-8623-411b55bbc874\") " pod="kube-system/konnectivity-agent-zmr54"
Aug 05 17:12:55.070718 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:12:55.070497    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/b0520f9c-efe2-43b9-8623-411b55bbc874-konnectivity-ca\") pod \"konnectivity-agent-zmr54\" (UID: \"b0520f9c-efe2-43b9-8623-411b55bbc874\") " pod="kube-system/konnectivity-agent-zmr54"
Aug 05 17:12:55.070718 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:12:55.070526    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/99b228eb-7a25-47a1-a90d-06b62e8f17e0-system-cni-dir\") pod \"multus-vzp52\" (UID: \"99b228eb-7a25-47a1-a90d-06b62e8f17e0\") " pod="openshift-multus/multus-vzp52"
Aug 05 17:12:55.070718 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:12:55.070552    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/99b228eb-7a25-47a1-a90d-06b62e8f17e0-host-run-k8s-cni-cncf-io\") pod \"multus-vzp52\" (UID: \"99b228eb-7a25-47a1-a90d-06b62e8f17e0\") " pod="openshift-multus/multus-vzp52"
Aug 05 17:12:55.070718 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:12:55.070585    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/ac2f6cf0-11ca-47f3-8597-5cc1db578e78-host\") pod \"tuned-nb82s\" (UID: \"ac2f6cf0-11ca-47f3-8597-5cc1db578e78\") " pod="openshift-cluster-node-tuning-operator/tuned-nb82s"
Aug 05 17:12:55.070718 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:12:55.070609    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d662bf46-8fcd-40b5-94fa-eded9b062699-run-openvswitch\") pod \"ovnkube-node-tsjgl\" (UID: \"d662bf46-8fcd-40b5-94fa-eded9b062699\") " pod="openshift-ovn-kubernetes/ovnkube-node-tsjgl"
Aug 05 17:12:55.070718 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:12:55.070629    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/99b228eb-7a25-47a1-a90d-06b62e8f17e0-host-run-k8s-cni-cncf-io\") pod \"multus-vzp52\" (UID: \"99b228eb-7a25-47a1-a90d-06b62e8f17e0\") " pod="openshift-multus/multus-vzp52"
Aug 05 17:12:55.070718 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:12:55.070637    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/99b228eb-7a25-47a1-a90d-06b62e8f17e0-system-cni-dir\") pod \"multus-vzp52\" (UID: \"99b228eb-7a25-47a1-a90d-06b62e8f17e0\") " pod="openshift-multus/multus-vzp52"
Aug 05 17:12:55.070718 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:12:55.070634    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/d662bf46-8fcd-40b5-94fa-eded9b062699-ovnkube-script-lib\") pod \"ovnkube-node-tsjgl\" (UID: \"d662bf46-8fcd-40b5-94fa-eded9b062699\") " pod="openshift-ovn-kubernetes/ovnkube-node-tsjgl"
Aug 05 17:12:55.070718 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:12:55.070685    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/df7e05c7-620a-4246-bd4b-0e2c2009335b-device-dir\") pod \"aws-ebs-csi-driver-node-7h5q8\" (UID: \"df7e05c7-620a-4246-bd4b-0e2c2009335b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7h5q8"
Aug 05 17:12:55.070718 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:12:55.070715    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/b934c28e-a416-4ddf-8b90-0c9acdd8ce4c-host-slash\") pod \"iptables-alerter-txqn4\" (UID: \"b934c28e-a416-4ddf-8b90-0c9acdd8ce4c\") " pod="openshift-network-operator/iptables-alerter-txqn4"
Aug 05 17:12:55.071423 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:12:55.070743    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bm5cq\" (UniqueName: \"kubernetes.io/projected/99b228eb-7a25-47a1-a90d-06b62e8f17e0-kube-api-access-bm5cq\") pod \"multus-vzp52\" (UID: \"99b228eb-7a25-47a1-a90d-06b62e8f17e0\") " pod="openshift-multus/multus-vzp52"
Aug 05 17:12:55.071423 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:12:55.070768    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/ac2f6cf0-11ca-47f3-8597-5cc1db578e78-etc-systemd\") pod \"tuned-nb82s\" (UID: \"ac2f6cf0-11ca-47f3-8597-5cc1db578e78\") " pod="openshift-cluster-node-tuning-operator/tuned-nb82s"
Aug 05 17:12:55.071423 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:12:55.070793    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/ac2f6cf0-11ca-47f3-8597-5cc1db578e78-etc-tuned\") pod \"tuned-nb82s\" (UID: \"ac2f6cf0-11ca-47f3-8597-5cc1db578e78\") " pod="openshift-cluster-node-tuning-operator/tuned-nb82s"
Aug 05 17:12:55.071423 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:12:55.070818    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/da6f032b-35a1-4c51-b115-defaf9b2a9a5-metrics-certs\") pod \"network-metrics-daemon-677x7\" (UID: \"da6f032b-35a1-4c51-b115-defaf9b2a9a5\") " pod="openshift-multus/network-metrics-daemon-677x7"
Aug 05 17:12:55.071423 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:12:55.070842    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/d662bf46-8fcd-40b5-94fa-eded9b062699-systemd-units\") pod \"ovnkube-node-tsjgl\" (UID: \"d662bf46-8fcd-40b5-94fa-eded9b062699\") " pod="openshift-ovn-kubernetes/ovnkube-node-tsjgl"
Aug 05 17:12:55.071423 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:12:55.070870    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d662bf46-8fcd-40b5-94fa-eded9b062699-var-lib-openvswitch\") pod \"ovnkube-node-tsjgl\" (UID: \"d662bf46-8fcd-40b5-94fa-eded9b062699\") " pod="openshift-ovn-kubernetes/ovnkube-node-tsjgl"
Aug 05 17:12:55.071423 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:12:55.070897    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/df7e05c7-620a-4246-bd4b-0e2c2009335b-registration-dir\") pod \"aws-ebs-csi-driver-node-7h5q8\" (UID: \"df7e05c7-620a-4246-bd4b-0e2c2009335b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7h5q8"
Aug 05 17:12:55.071423 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:12:55.070915    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/b934c28e-a416-4ddf-8b90-0c9acdd8ce4c-iptables-alerter-script\") pod \"iptables-alerter-txqn4\" (UID: \"b934c28e-a416-4ddf-8b90-0c9acdd8ce4c\") " pod="openshift-network-operator/iptables-alerter-txqn4"
Aug 05 17:12:55.071423 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:12:55.070928    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c7v68\" (UniqueName: \"kubernetes.io/projected/100f6b72-e895-453f-8647-1f615c2fb780-kube-api-access-c7v68\") pod \"multus-additional-cni-plugins-mgfr4\" (UID: \"100f6b72-e895-453f-8647-1f615c2fb780\") " pod="openshift-multus/multus-additional-cni-plugins-mgfr4"
Aug 05 17:12:55.071423 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:12:55.070958    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/99b228eb-7a25-47a1-a90d-06b62e8f17e0-os-release\") pod \"multus-vzp52\" (UID: \"99b228eb-7a25-47a1-a90d-06b62e8f17e0\") " pod="openshift-multus/multus-vzp52"
Aug 05 17:12:55.071423 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:12:55.070898    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/b934c28e-a416-4ddf-8b90-0c9acdd8ce4c-host-slash\") pod \"iptables-alerter-txqn4\" (UID: \"b934c28e-a416-4ddf-8b90-0c9acdd8ce4c\") " pod="openshift-network-operator/iptables-alerter-txqn4"
Aug 05 17:12:55.071423 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:12:55.070982    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/99b228eb-7a25-47a1-a90d-06b62e8f17e0-multus-daemon-config\") pod \"multus-vzp52\" (UID: \"99b228eb-7a25-47a1-a90d-06b62e8f17e0\") " pod="openshift-multus/multus-vzp52"
Aug 05 17:12:55.071423 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:12:55.071005    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/ac2f6cf0-11ca-47f3-8597-5cc1db578e78-run\") pod \"tuned-nb82s\" (UID: \"ac2f6cf0-11ca-47f3-8597-5cc1db578e78\") " pod="openshift-cluster-node-tuning-operator/tuned-nb82s"
Aug 05 17:12:55.071423 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:12:55.071038    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/99b228eb-7a25-47a1-a90d-06b62e8f17e0-os-release\") pod \"multus-vzp52\" (UID: \"99b228eb-7a25-47a1-a90d-06b62e8f17e0\") " pod="openshift-multus/multus-vzp52"
Aug 05 17:12:55.071423 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:12:55.071047    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-td69h\" (UniqueName: \"kubernetes.io/projected/da6f032b-35a1-4c51-b115-defaf9b2a9a5-kube-api-access-td69h\") pod \"network-metrics-daemon-677x7\" (UID: \"da6f032b-35a1-4c51-b115-defaf9b2a9a5\") " pod="openshift-multus/network-metrics-daemon-677x7"
Aug 05 17:12:55.071423 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:12:55.071072    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/d662bf46-8fcd-40b5-94fa-eded9b062699-run-systemd\") pod \"ovnkube-node-tsjgl\" (UID: \"d662bf46-8fcd-40b5-94fa-eded9b062699\") " pod="openshift-ovn-kubernetes/ovnkube-node-tsjgl"
Aug 05 17:12:55.071423 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:12:55.071099    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/d662bf46-8fcd-40b5-94fa-eded9b062699-run-ovn\") pod \"ovnkube-node-tsjgl\" (UID: \"d662bf46-8fcd-40b5-94fa-eded9b062699\") " pod="openshift-ovn-kubernetes/ovnkube-node-tsjgl"
Aug 05 17:12:55.072385 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:12:55.071140    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/99b228eb-7a25-47a1-a90d-06b62e8f17e0-multus-socket-dir-parent\") pod \"multus-vzp52\" (UID: \"99b228eb-7a25-47a1-a90d-06b62e8f17e0\") " pod="openshift-multus/multus-vzp52"
Aug 05 17:12:55.072385 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:12:55.071179    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ac2f6cf0-11ca-47f3-8597-5cc1db578e78-var-lib-kubelet\") pod \"tuned-nb82s\" (UID: \"ac2f6cf0-11ca-47f3-8597-5cc1db578e78\") " pod="openshift-cluster-node-tuning-operator/tuned-nb82s"
Aug 05 17:12:55.072385 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:12:55.071204    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d662bf46-8fcd-40b5-94fa-eded9b062699-etc-openvswitch\") pod \"ovnkube-node-tsjgl\" (UID: \"d662bf46-8fcd-40b5-94fa-eded9b062699\") " pod="openshift-ovn-kubernetes/ovnkube-node-tsjgl"
Aug 05 17:12:55.072385 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:12:55.071227    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/d662bf46-8fcd-40b5-94fa-eded9b062699-log-socket\") pod \"ovnkube-node-tsjgl\" (UID: \"d662bf46-8fcd-40b5-94fa-eded9b062699\") " pod="openshift-ovn-kubernetes/ovnkube-node-tsjgl"
Aug 05 17:12:55.072385 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:12:55.071250    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/d662bf46-8fcd-40b5-94fa-eded9b062699-host-cni-bin\") pod \"ovnkube-node-tsjgl\" (UID: \"d662bf46-8fcd-40b5-94fa-eded9b062699\") " pod="openshift-ovn-kubernetes/ovnkube-node-tsjgl"
Aug 05 17:12:55.072385 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:12:55.071273    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/d662bf46-8fcd-40b5-94fa-eded9b062699-env-overrides\") pod \"ovnkube-node-tsjgl\" (UID: \"d662bf46-8fcd-40b5-94fa-eded9b062699\") " pod="openshift-ovn-kubernetes/ovnkube-node-tsjgl"
Aug 05 17:12:55.072385 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:12:55.071302    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/61599c74-5517-43f4-bed8-a232408b44c0-host\") pod \"node-ca-lrdmn\" (UID: \"61599c74-5517-43f4-bed8-a232408b44c0\") " pod="openshift-image-registry/node-ca-lrdmn"
Aug 05 17:12:55.072385 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:12:55.071307    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/99b228eb-7a25-47a1-a90d-06b62e8f17e0-multus-socket-dir-parent\") pod \"multus-vzp52\" (UID: \"99b228eb-7a25-47a1-a90d-06b62e8f17e0\") " pod="openshift-multus/multus-vzp52"
Aug 05 17:12:55.072385 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:12:55.071322    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/c680deac-4ffe-40ff-b521-80beb96c5196-hosts-file\") pod \"node-resolver-qz2cd\" (UID: \"c680deac-4ffe-40ff-b521-80beb96c5196\") " pod="openshift-dns/node-resolver-qz2cd"
Aug 05 17:12:55.072385 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:12:55.071346    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/ac2f6cf0-11ca-47f3-8597-5cc1db578e78-etc-sysctl-d\") pod \"tuned-nb82s\" (UID: \"ac2f6cf0-11ca-47f3-8597-5cc1db578e78\") " pod="openshift-cluster-node-tuning-operator/tuned-nb82s"
Aug 05 17:12:55.072385 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:12:55.071363    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/f2ecc651-01ec-4fa1-a7c1-780b3a963996-dbus\") pod \"global-pull-secret-syncer-9jxp4\" (UID: \"f2ecc651-01ec-4fa1-a7c1-780b3a963996\") " pod="kube-system/global-pull-secret-syncer-9jxp4"
Aug 05 17:12:55.072385 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:12:55.071380    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/99b228eb-7a25-47a1-a90d-06b62e8f17e0-multus-cni-dir\") pod \"multus-vzp52\" (UID: \"99b228eb-7a25-47a1-a90d-06b62e8f17e0\") " pod="openshift-multus/multus-vzp52"
Aug 05 17:12:55.072385 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:12:55.071395    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/ac2f6cf0-11ca-47f3-8597-5cc1db578e78-etc-sysctl-conf\") pod \"tuned-nb82s\" (UID: \"ac2f6cf0-11ca-47f3-8597-5cc1db578e78\") " pod="openshift-cluster-node-tuning-operator/tuned-nb82s"
Aug 05 17:12:55.072385 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:12:55.071414    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5dmf8\" (UniqueName: \"kubernetes.io/projected/ac2f6cf0-11ca-47f3-8597-5cc1db578e78-kube-api-access-5dmf8\") pod \"tuned-nb82s\" (UID: \"ac2f6cf0-11ca-47f3-8597-5cc1db578e78\") " pod="openshift-cluster-node-tuning-operator/tuned-nb82s"
Aug 05 17:12:55.072385 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:12:55.071452    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rw4qx\" (UniqueName: \"kubernetes.io/projected/d662bf46-8fcd-40b5-94fa-eded9b062699-kube-api-access-rw4qx\") pod \"ovnkube-node-tsjgl\" (UID: \"d662bf46-8fcd-40b5-94fa-eded9b062699\") " pod="openshift-ovn-kubernetes/ovnkube-node-tsjgl"
Aug 05 17:12:55.072385 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:12:55.071477    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/df7e05c7-620a-4246-bd4b-0e2c2009335b-kubelet-dir\") pod \"aws-ebs-csi-driver-node-7h5q8\" (UID: \"df7e05c7-620a-4246-bd4b-0e2c2009335b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7h5q8"
Aug 05 17:12:55.072385 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:12:55.071494    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2d2kp\" (UniqueName: \"kubernetes.io/projected/df7e05c7-620a-4246-bd4b-0e2c2009335b-kube-api-access-2d2kp\") pod \"aws-ebs-csi-driver-node-7h5q8\" (UID: \"df7e05c7-620a-4246-bd4b-0e2c2009335b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7h5q8"
Aug 05 17:12:55.073250 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:12:55.071509    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f2ecc651-01ec-4fa1-a7c1-780b3a963996-original-pull-secret\") pod \"global-pull-secret-syncer-9jxp4\" (UID: \"f2ecc651-01ec-4fa1-a7c1-780b3a963996\") " pod="kube-system/global-pull-secret-syncer-9jxp4"
Aug 05 17:12:55.073250 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:12:55.071528    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/99b228eb-7a25-47a1-a90d-06b62e8f17e0-etc-kubernetes\") pod \"multus-vzp52\" (UID: \"99b228eb-7a25-47a1-a90d-06b62e8f17e0\") " pod="openshift-multus/multus-vzp52"
Aug 05 17:12:55.073250 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:12:55.071542    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/d662bf46-8fcd-40b5-94fa-eded9b062699-host-kubelet\") pod \"ovnkube-node-tsjgl\" (UID: \"d662bf46-8fcd-40b5-94fa-eded9b062699\") " pod="openshift-ovn-kubernetes/ovnkube-node-tsjgl"
Aug 05 17:12:55.073250 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:12:55.071550    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/99b228eb-7a25-47a1-a90d-06b62e8f17e0-multus-cni-dir\") pod \"multus-vzp52\" (UID: \"99b228eb-7a25-47a1-a90d-06b62e8f17e0\") " pod="openshift-multus/multus-vzp52"
Aug 05 17:12:55.073250 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:12:55.071559    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/99b228eb-7a25-47a1-a90d-06b62e8f17e0-multus-daemon-config\") pod \"multus-vzp52\" (UID: \"99b228eb-7a25-47a1-a90d-06b62e8f17e0\") " pod="openshift-multus/multus-vzp52"
Aug 05 17:12:55.073250 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:12:55.071556    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/d662bf46-8fcd-40b5-94fa-eded9b062699-host-run-netns\") pod \"ovnkube-node-tsjgl\" (UID: \"d662bf46-8fcd-40b5-94fa-eded9b062699\") " pod="openshift-ovn-kubernetes/ovnkube-node-tsjgl"
Aug 05 17:12:55.073250 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:12:55.071597    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/99b228eb-7a25-47a1-a90d-06b62e8f17e0-etc-kubernetes\") pod \"multus-vzp52\" (UID: \"99b228eb-7a25-47a1-a90d-06b62e8f17e0\") " pod="openshift-multus/multus-vzp52"
Aug 05 17:12:55.073250 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:12:55.071623    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ncqk2\" (UniqueName: \"kubernetes.io/projected/61599c74-5517-43f4-bed8-a232408b44c0-kube-api-access-ncqk2\") pod \"node-ca-lrdmn\" (UID: \"61599c74-5517-43f4-bed8-a232408b44c0\") " pod="openshift-image-registry/node-ca-lrdmn"
Aug 05 17:12:55.073250 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:12:55.071647    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gnsw6\" (UniqueName: \"kubernetes.io/projected/b934c28e-a416-4ddf-8b90-0c9acdd8ce4c-kube-api-access-gnsw6\") pod \"iptables-alerter-txqn4\" (UID: \"b934c28e-a416-4ddf-8b90-0c9acdd8ce4c\") " pod="openshift-network-operator/iptables-alerter-txqn4"
Aug 05 17:12:55.073250 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:12:55.071685    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/100f6b72-e895-453f-8647-1f615c2fb780-cni-binary-copy\") pod \"multus-additional-cni-plugins-mgfr4\" (UID: \"100f6b72-e895-453f-8647-1f615c2fb780\") " pod="openshift-multus/multus-additional-cni-plugins-mgfr4"
Aug 05 17:12:55.073250 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:12:55.071705    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/99b228eb-7a25-47a1-a90d-06b62e8f17e0-host-var-lib-kubelet\") pod \"multus-vzp52\" (UID: \"99b228eb-7a25-47a1-a90d-06b62e8f17e0\") " pod="openshift-multus/multus-vzp52"
Aug 05 17:12:55.073250 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:12:55.071737    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c680deac-4ffe-40ff-b521-80beb96c5196-tmp-dir\") pod \"node-resolver-qz2cd\" (UID: \"c680deac-4ffe-40ff-b521-80beb96c5196\") " pod="openshift-dns/node-resolver-qz2cd"
Aug 05 17:12:55.073250 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:12:55.071765    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ac2f6cf0-11ca-47f3-8597-5cc1db578e78-tmp\") pod \"tuned-nb82s\" (UID: \"ac2f6cf0-11ca-47f3-8597-5cc1db578e78\") " pod="openshift-cluster-node-tuning-operator/tuned-nb82s"
Aug 05 17:12:55.073250 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:12:55.071790    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/f2ecc651-01ec-4fa1-a7c1-780b3a963996-kubelet-config\") pod \"global-pull-secret-syncer-9jxp4\" (UID: \"f2ecc651-01ec-4fa1-a7c1-780b3a963996\") " pod="kube-system/global-pull-secret-syncer-9jxp4"
Aug 05 17:12:55.073250 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:12:55.071824    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/99b228eb-7a25-47a1-a90d-06b62e8f17e0-host-var-lib-kubelet\") pod \"multus-vzp52\" (UID: \"99b228eb-7a25-47a1-a90d-06b62e8f17e0\") " pod="openshift-multus/multus-vzp52"
Aug 05 17:12:55.073250 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:12:55.071867    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/100f6b72-e895-453f-8647-1f615c2fb780-tuning-conf-dir\") pod \"multus-additional-cni-plugins-mgfr4\" (UID: \"100f6b72-e895-453f-8647-1f615c2fb780\") " pod="openshift-multus/multus-additional-cni-plugins-mgfr4"
Aug 05 17:12:55.073250 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:12:55.071895    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/100f6b72-e895-453f-8647-1f615c2fb780-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-mgfr4\" (UID: \"100f6b72-e895-453f-8647-1f615c2fb780\") " pod="openshift-multus/multus-additional-cni-plugins-mgfr4"
Aug 05 17:12:55.074211 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:12:55.071933    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/99b228eb-7a25-47a1-a90d-06b62e8f17e0-host-var-lib-cni-multus\") pod \"multus-vzp52\" (UID: \"99b228eb-7a25-47a1-a90d-06b62e8f17e0\") " pod="openshift-multus/multus-vzp52"
Aug 05 17:12:55.074211 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:12:55.071965    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ac2f6cf0-11ca-47f3-8597-5cc1db578e78-etc-kubernetes\") pod \"tuned-nb82s\" (UID: \"ac2f6cf0-11ca-47f3-8597-5cc1db578e78\") " pod="openshift-cluster-node-tuning-operator/tuned-nb82s"
Aug 05 17:12:55.074211 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:12:55.071994    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d662bf46-8fcd-40b5-94fa-eded9b062699-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-tsjgl\" (UID: \"d662bf46-8fcd-40b5-94fa-eded9b062699\") " pod="openshift-ovn-kubernetes/ovnkube-node-tsjgl"
Aug 05 17:12:55.074211 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:12:55.072011    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/99b228eb-7a25-47a1-a90d-06b62e8f17e0-host-var-lib-cni-multus\") pod \"multus-vzp52\" (UID: \"99b228eb-7a25-47a1-a90d-06b62e8f17e0\") " pod="openshift-multus/multus-vzp52"
Aug 05 17:12:55.074211 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:12:55.072022    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/df7e05c7-620a-4246-bd4b-0e2c2009335b-sys-fs\") pod \"aws-ebs-csi-driver-node-7h5q8\" (UID: \"df7e05c7-620a-4246-bd4b-0e2c2009335b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7h5q8"
Aug 05 17:12:55.074211 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:12:55.072055    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/100f6b72-e895-453f-8647-1f615c2fb780-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-mgfr4\" (UID: \"100f6b72-e895-453f-8647-1f615c2fb780\") " pod="openshift-multus/multus-additional-cni-plugins-mgfr4"
Aug 05 17:12:55.074211 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:12:55.072103    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/d662bf46-8fcd-40b5-94fa-eded9b062699-node-log\") pod \"ovnkube-node-tsjgl\" (UID: \"d662bf46-8fcd-40b5-94fa-eded9b062699\") " pod="openshift-ovn-kubernetes/ovnkube-node-tsjgl"
Aug 05 17:12:55.074211 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:12:55.072134    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d662bf46-8fcd-40b5-94fa-eded9b062699-host-run-ovn-kubernetes\") pod \"ovnkube-node-tsjgl\" (UID: \"d662bf46-8fcd-40b5-94fa-eded9b062699\") " pod="openshift-ovn-kubernetes/ovnkube-node-tsjgl"
Aug 05 17:12:55.074211 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:12:55.072162    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/ac2f6cf0-11ca-47f3-8597-5cc1db578e78-etc-sysconfig\") pod \"tuned-nb82s\" (UID: \"ac2f6cf0-11ca-47f3-8597-5cc1db578e78\") " pod="openshift-cluster-node-tuning-operator/tuned-nb82s"
Aug 05 17:12:55.074211 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:12:55.072161    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/100f6b72-e895-453f-8647-1f615c2fb780-tuning-conf-dir\") pod \"multus-additional-cni-plugins-mgfr4\" (UID: \"100f6b72-e895-453f-8647-1f615c2fb780\") " pod="openshift-multus/multus-additional-cni-plugins-mgfr4"
Aug 05 17:12:55.074211 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:12:55.072199    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/df7e05c7-620a-4246-bd4b-0e2c2009335b-socket-dir\") pod \"aws-ebs-csi-driver-node-7h5q8\" (UID: \"df7e05c7-620a-4246-bd4b-0e2c2009335b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7h5q8"
Aug 05 17:12:55.074211 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:12:55.072218    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/100f6b72-e895-453f-8647-1f615c2fb780-cni-binary-copy\") pod \"multus-additional-cni-plugins-mgfr4\" (UID: \"100f6b72-e895-453f-8647-1f615c2fb780\") " pod="openshift-multus/multus-additional-cni-plugins-mgfr4"
Aug 05 17:12:55.074211 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:12:55.072230    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/100f6b72-e895-453f-8647-1f615c2fb780-system-cni-dir\") pod \"multus-additional-cni-plugins-mgfr4\" (UID: \"100f6b72-e895-453f-8647-1f615c2fb780\") " pod="openshift-multus/multus-additional-cni-plugins-mgfr4"
Aug 05 17:12:55.074211 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:12:55.072272    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/99b228eb-7a25-47a1-a90d-06b62e8f17e0-host-var-lib-cni-bin\") pod \"multus-vzp52\" (UID: \"99b228eb-7a25-47a1-a90d-06b62e8f17e0\") " pod="openshift-multus/multus-vzp52"
Aug 05 17:12:55.074211 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:12:55.072310    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/100f6b72-e895-453f-8647-1f615c2fb780-system-cni-dir\") pod \"multus-additional-cni-plugins-mgfr4\" (UID: \"100f6b72-e895-453f-8647-1f615c2fb780\") " pod="openshift-multus/multus-additional-cni-plugins-mgfr4"
Aug 05 17:12:55.074211 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:12:55.072323    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ac2f6cf0-11ca-47f3-8597-5cc1db578e78-sys\") pod \"tuned-nb82s\" (UID: \"ac2f6cf0-11ca-47f3-8597-5cc1db578e78\") " pod="openshift-cluster-node-tuning-operator/tuned-nb82s"
Aug 05 17:12:55.075104 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:12:55.072332    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/100f6b72-e895-453f-8647-1f615c2fb780-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-mgfr4\" (UID: \"100f6b72-e895-453f-8647-1f615c2fb780\") " pod="openshift-multus/multus-additional-cni-plugins-mgfr4"
Aug 05 17:12:55.075104 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:12:55.072339    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/99b228eb-7a25-47a1-a90d-06b62e8f17e0-host-var-lib-cni-bin\") pod \"multus-vzp52\" (UID: \"99b228eb-7a25-47a1-a90d-06b62e8f17e0\") " pod="openshift-multus/multus-vzp52"
Aug 05 17:12:55.075104 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:12:55.072351    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/99b228eb-7a25-47a1-a90d-06b62e8f17e0-multus-conf-dir\") pod \"multus-vzp52\" (UID: \"99b228eb-7a25-47a1-a90d-06b62e8f17e0\") " pod="openshift-multus/multus-vzp52"
Aug 05 17:12:55.075104 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:12:55.072385    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/99b228eb-7a25-47a1-a90d-06b62e8f17e0-cni-binary-copy\") pod \"multus-vzp52\" (UID: \"99b228eb-7a25-47a1-a90d-06b62e8f17e0\") " pod="openshift-multus/multus-vzp52"
Aug 05 17:12:55.075104 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:12:55.072451    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/99b228eb-7a25-47a1-a90d-06b62e8f17e0-multus-conf-dir\") pod \"multus-vzp52\" (UID: \"99b228eb-7a25-47a1-a90d-06b62e8f17e0\") " pod="openshift-multus/multus-vzp52"
Aug 05 17:12:55.075104 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:12:55.072472    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/99b228eb-7a25-47a1-a90d-06b62e8f17e0-host-run-netns\") pod \"multus-vzp52\" (UID: \"99b228eb-7a25-47a1-a90d-06b62e8f17e0\") " pod="openshift-multus/multus-vzp52"
Aug 05 17:12:55.075104 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:12:55.072496    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/99b228eb-7a25-47a1-a90d-06b62e8f17e0-host-run-multus-certs\") pod \"multus-vzp52\" (UID: \"99b228eb-7a25-47a1-a90d-06b62e8f17e0\") " pod="openshift-multus/multus-vzp52"
Aug 05 17:12:55.075104 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:12:55.072514    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rf7fb\" (UniqueName: \"kubernetes.io/projected/c680deac-4ffe-40ff-b521-80beb96c5196-kube-api-access-rf7fb\") pod \"node-resolver-qz2cd\" (UID: \"c680deac-4ffe-40ff-b521-80beb96c5196\") " pod="openshift-dns/node-resolver-qz2cd"
Aug 05 17:12:55.075104 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:12:55.072530    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/ac2f6cf0-11ca-47f3-8597-5cc1db578e78-etc-modprobe-d\") pod \"tuned-nb82s\" (UID: \"ac2f6cf0-11ca-47f3-8597-5cc1db578e78\") " pod="openshift-cluster-node-tuning-operator/tuned-nb82s"
Aug 05 17:12:55.075104 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:12:55.072554    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/d662bf46-8fcd-40b5-94fa-eded9b062699-host-cni-netd\") pod \"ovnkube-node-tsjgl\" (UID: \"d662bf46-8fcd-40b5-94fa-eded9b062699\") " pod="openshift-ovn-kubernetes/ovnkube-node-tsjgl"
Aug 05 17:12:55.075104 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:12:55.072571    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/99b228eb-7a25-47a1-a90d-06b62e8f17e0-host-run-multus-certs\") pod \"multus-vzp52\" (UID: \"99b228eb-7a25-47a1-a90d-06b62e8f17e0\") " pod="openshift-multus/multus-vzp52"
Aug 05 17:12:55.075104 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:12:55.072582    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/100f6b72-e895-453f-8647-1f615c2fb780-os-release\") pod \"multus-additional-cni-plugins-mgfr4\" (UID: \"100f6b72-e895-453f-8647-1f615c2fb780\") " pod="openshift-multus/multus-additional-cni-plugins-mgfr4"
Aug 05 17:12:55.075104 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:12:55.072609    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/99b228eb-7a25-47a1-a90d-06b62e8f17e0-cnibin\") pod \"multus-vzp52\" (UID: \"99b228eb-7a25-47a1-a90d-06b62e8f17e0\") " pod="openshift-multus/multus-vzp52"
Aug 05 17:12:55.075104 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:12:55.072635    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/99b228eb-7a25-47a1-a90d-06b62e8f17e0-hostroot\") pod \"multus-vzp52\" (UID: \"99b228eb-7a25-47a1-a90d-06b62e8f17e0\") " pod="openshift-multus/multus-vzp52"
Aug 05 17:12:55.075104 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:12:55.072660    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/d662bf46-8fcd-40b5-94fa-eded9b062699-host-slash\") pod \"ovnkube-node-tsjgl\" (UID: \"d662bf46-8fcd-40b5-94fa-eded9b062699\") " pod="openshift-ovn-kubernetes/ovnkube-node-tsjgl"
Aug 05 17:12:55.075104 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:12:55.072668    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/100f6b72-e895-453f-8647-1f615c2fb780-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-mgfr4\" (UID: \"100f6b72-e895-453f-8647-1f615c2fb780\") " pod="openshift-multus/multus-additional-cni-plugins-mgfr4"
Aug 05 17:12:55.075104 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:12:55.072611    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/99b228eb-7a25-47a1-a90d-06b62e8f17e0-host-run-netns\") pod \"multus-vzp52\" (UID: \"99b228eb-7a25-47a1-a90d-06b62e8f17e0\") " pod="openshift-multus/multus-vzp52"
Aug 05 17:12:55.076007 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:12:55.072673    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/100f6b72-e895-453f-8647-1f615c2fb780-os-release\") pod \"multus-additional-cni-plugins-mgfr4\" (UID: \"100f6b72-e895-453f-8647-1f615c2fb780\") " pod="openshift-multus/multus-additional-cni-plugins-mgfr4"
Aug 05 17:12:55.076007 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:12:55.072745    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/99b228eb-7a25-47a1-a90d-06b62e8f17e0-cnibin\") pod \"multus-vzp52\" (UID: \"99b228eb-7a25-47a1-a90d-06b62e8f17e0\") " pod="openshift-multus/multus-vzp52"
Aug 05 17:12:55.076007 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:12:55.072773    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/99b228eb-7a25-47a1-a90d-06b62e8f17e0-hostroot\") pod \"multus-vzp52\" (UID: \"99b228eb-7a25-47a1-a90d-06b62e8f17e0\") " pod="openshift-multus/multus-vzp52"
Aug 05 17:12:55.076007 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:12:55.072844    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/99b228eb-7a25-47a1-a90d-06b62e8f17e0-cni-binary-copy\") pod \"multus-vzp52\" (UID: \"99b228eb-7a25-47a1-a90d-06b62e8f17e0\") " pod="openshift-multus/multus-vzp52"
Aug 05 17:12:55.079451 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:12:55.079425    2571 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Aug 05 17:12:55.082979 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:12:55.082960    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gnsw6\" (UniqueName: \"kubernetes.io/projected/b934c28e-a416-4ddf-8b90-0c9acdd8ce4c-kube-api-access-gnsw6\") pod \"iptables-alerter-txqn4\" (UID: \"b934c28e-a416-4ddf-8b90-0c9acdd8ce4c\") " pod="openshift-network-operator/iptables-alerter-txqn4"
Aug 05 17:12:55.083101 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:12:55.083040    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c7v68\" (UniqueName: \"kubernetes.io/projected/100f6b72-e895-453f-8647-1f615c2fb780-kube-api-access-c7v68\") pod \"multus-additional-cni-plugins-mgfr4\" (UID: \"100f6b72-e895-453f-8647-1f615c2fb780\") " pod="openshift-multus/multus-additional-cni-plugins-mgfr4"
Aug 05 17:12:55.083101 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:12:55.083040    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bm5cq\" (UniqueName: \"kubernetes.io/projected/99b228eb-7a25-47a1-a90d-06b62e8f17e0-kube-api-access-bm5cq\") pod \"multus-vzp52\" (UID: \"99b228eb-7a25-47a1-a90d-06b62e8f17e0\") " pod="openshift-multus/multus-vzp52"
Aug 05 17:12:55.135163 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:12:55.135112    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-2.ec2.internal" event={"ID":"bb3fac1e9119c8f4580f4cf12eb2ffb1","Type":"ContainerStarted","Data":"cd1cdc0023080df0a6663aedac8a8c50521b25858e8c5f4e397d3b56f35851e3"}
Aug 05 17:12:55.136257 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:12:55.136233    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-2.ec2.internal" event={"ID":"300ebfcee102c70ab72580814e7ccaa4","Type":"ContainerStarted","Data":"3e328c61698e8c2430035058081e4517c91b5893c5e264502c215714ce49bb01"}
Aug 05 17:12:55.173545 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:12:55.173514    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/ac2f6cf0-11ca-47f3-8597-5cc1db578e78-etc-systemd\") pod \"tuned-nb82s\" (UID: \"ac2f6cf0-11ca-47f3-8597-5cc1db578e78\") " pod="openshift-cluster-node-tuning-operator/tuned-nb82s"
Aug 05 17:12:55.173545 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:12:55.173549    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/ac2f6cf0-11ca-47f3-8597-5cc1db578e78-etc-tuned\") pod \"tuned-nb82s\" (UID: \"ac2f6cf0-11ca-47f3-8597-5cc1db578e78\") " pod="openshift-cluster-node-tuning-operator/tuned-nb82s"
Aug 05 17:12:55.173725 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:12:55.173566    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/da6f032b-35a1-4c51-b115-defaf9b2a9a5-metrics-certs\") pod \"network-metrics-daemon-677x7\" (UID: \"da6f032b-35a1-4c51-b115-defaf9b2a9a5\") " pod="openshift-multus/network-metrics-daemon-677x7"
Aug 05 17:12:55.173725 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:12:55.173582    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/d662bf46-8fcd-40b5-94fa-eded9b062699-systemd-units\") pod \"ovnkube-node-tsjgl\" (UID: \"d662bf46-8fcd-40b5-94fa-eded9b062699\") " pod="openshift-ovn-kubernetes/ovnkube-node-tsjgl"
Aug 05 17:12:55.173725 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:12:55.173602    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d662bf46-8fcd-40b5-94fa-eded9b062699-var-lib-openvswitch\") pod \"ovnkube-node-tsjgl\" (UID: \"d662bf46-8fcd-40b5-94fa-eded9b062699\") " pod="openshift-ovn-kubernetes/ovnkube-node-tsjgl"
Aug 05 17:12:55.173725 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:12:55.173626    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/df7e05c7-620a-4246-bd4b-0e2c2009335b-registration-dir\") pod \"aws-ebs-csi-driver-node-7h5q8\" (UID: \"df7e05c7-620a-4246-bd4b-0e2c2009335b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7h5q8"
Aug 05 17:12:55.173725 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:12:55.173648    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/ac2f6cf0-11ca-47f3-8597-5cc1db578e78-run\") pod \"tuned-nb82s\" (UID: \"ac2f6cf0-11ca-47f3-8597-5cc1db578e78\") " pod="openshift-cluster-node-tuning-operator/tuned-nb82s"
Aug 05 17:12:55.173725 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:12:55.173649    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/ac2f6cf0-11ca-47f3-8597-5cc1db578e78-etc-systemd\") pod \"tuned-nb82s\" (UID: \"ac2f6cf0-11ca-47f3-8597-5cc1db578e78\") " pod="openshift-cluster-node-tuning-operator/tuned-nb82s"
Aug 05 17:12:55.173725 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:12:55.173669    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-td69h\" (UniqueName: \"kubernetes.io/projected/da6f032b-35a1-4c51-b115-defaf9b2a9a5-kube-api-access-td69h\") pod \"network-metrics-daemon-677x7\" (UID: \"da6f032b-35a1-4c51-b115-defaf9b2a9a5\") " pod="openshift-multus/network-metrics-daemon-677x7"
Aug 05 17:12:55.173725 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:12:55.173672    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/d662bf46-8fcd-40b5-94fa-eded9b062699-systemd-units\") pod \"ovnkube-node-tsjgl\" (UID: \"d662bf46-8fcd-40b5-94fa-eded9b062699\") " pod="openshift-ovn-kubernetes/ovnkube-node-tsjgl"
Aug 05 17:12:55.173725 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:12:55.173679    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d662bf46-8fcd-40b5-94fa-eded9b062699-var-lib-openvswitch\") pod \"ovnkube-node-tsjgl\" (UID: \"d662bf46-8fcd-40b5-94fa-eded9b062699\") " pod="openshift-ovn-kubernetes/ovnkube-node-tsjgl"
Aug 05 17:12:55.173725 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:12:55.173693    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/d662bf46-8fcd-40b5-94fa-eded9b062699-run-systemd\") pod \"ovnkube-node-tsjgl\" (UID: \"d662bf46-8fcd-40b5-94fa-eded9b062699\") " pod="openshift-ovn-kubernetes/ovnkube-node-tsjgl"
Aug 05 17:12:55.173725 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:12:55.173726    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/df7e05c7-620a-4246-bd4b-0e2c2009335b-registration-dir\") pod \"aws-ebs-csi-driver-node-7h5q8\" (UID: \"df7e05c7-620a-4246-bd4b-0e2c2009335b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7h5q8"
Aug 05 17:12:55.173725 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:12:55.173730    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/d662bf46-8fcd-40b5-94fa-eded9b062699-run-ovn\") pod \"ovnkube-node-tsjgl\" (UID: \"d662bf46-8fcd-40b5-94fa-eded9b062699\") " pod="openshift-ovn-kubernetes/ovnkube-node-tsjgl"
Aug 05 17:12:55.174346 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:12:55.173740    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/d662bf46-8fcd-40b5-94fa-eded9b062699-run-systemd\") pod \"ovnkube-node-tsjgl\" (UID: \"d662bf46-8fcd-40b5-94fa-eded9b062699\") " pod="openshift-ovn-kubernetes/ovnkube-node-tsjgl"
Aug 05 17:12:55.174346 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:12:55.173773    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/ac2f6cf0-11ca-47f3-8597-5cc1db578e78-run\") pod \"tuned-nb82s\" (UID: \"ac2f6cf0-11ca-47f3-8597-5cc1db578e78\") " pod="openshift-cluster-node-tuning-operator/tuned-nb82s"
Aug 05 17:12:55.174346 ip-10-0-135-2 kubenswrapper[2571]: E0805 17:12:55.173776    2571 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Aug 05 17:12:55.174346 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:12:55.173810    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ac2f6cf0-11ca-47f3-8597-5cc1db578e78-var-lib-kubelet\") pod \"tuned-nb82s\" (UID: \"ac2f6cf0-11ca-47f3-8597-5cc1db578e78\") " pod="openshift-cluster-node-tuning-operator/tuned-nb82s"
Aug 05 17:12:55.174346 ip-10-0-135-2 kubenswrapper[2571]: E0805 17:12:55.173853    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/da6f032b-35a1-4c51-b115-defaf9b2a9a5-metrics-certs podName:da6f032b-35a1-4c51-b115-defaf9b2a9a5 nodeName:}" failed. No retries permitted until 2026-08-05 17:12:55.673828135 +0000 UTC m=+3.041194466 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/da6f032b-35a1-4c51-b115-defaf9b2a9a5-metrics-certs") pod "network-metrics-daemon-677x7" (UID: "da6f032b-35a1-4c51-b115-defaf9b2a9a5") : object "openshift-multus"/"metrics-daemon-secret" not registered
Aug 05 17:12:55.174346 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:12:55.173858    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/d662bf46-8fcd-40b5-94fa-eded9b062699-run-ovn\") pod \"ovnkube-node-tsjgl\" (UID: \"d662bf46-8fcd-40b5-94fa-eded9b062699\") " pod="openshift-ovn-kubernetes/ovnkube-node-tsjgl"
Aug 05 17:12:55.174346 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:12:55.173874    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d662bf46-8fcd-40b5-94fa-eded9b062699-etc-openvswitch\") pod \"ovnkube-node-tsjgl\" (UID: \"d662bf46-8fcd-40b5-94fa-eded9b062699\") " pod="openshift-ovn-kubernetes/ovnkube-node-tsjgl"
Aug 05 17:12:55.174346 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:12:55.173906    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/d662bf46-8fcd-40b5-94fa-eded9b062699-log-socket\") pod \"ovnkube-node-tsjgl\" (UID: \"d662bf46-8fcd-40b5-94fa-eded9b062699\") " pod="openshift-ovn-kubernetes/ovnkube-node-tsjgl"
Aug 05 17:12:55.174346 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:12:55.173925    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/d662bf46-8fcd-40b5-94fa-eded9b062699-host-cni-bin\") pod \"ovnkube-node-tsjgl\" (UID: \"d662bf46-8fcd-40b5-94fa-eded9b062699\") " pod="openshift-ovn-kubernetes/ovnkube-node-tsjgl"
Aug 05 17:12:55.174346 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:12:55.173929    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d662bf46-8fcd-40b5-94fa-eded9b062699-etc-openvswitch\") pod \"ovnkube-node-tsjgl\" (UID: \"d662bf46-8fcd-40b5-94fa-eded9b062699\") " pod="openshift-ovn-kubernetes/ovnkube-node-tsjgl"
Aug 05 17:12:55.174346 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:12:55.173947    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/d662bf46-8fcd-40b5-94fa-eded9b062699-env-overrides\") pod \"ovnkube-node-tsjgl\" (UID: \"d662bf46-8fcd-40b5-94fa-eded9b062699\") " pod="openshift-ovn-kubernetes/ovnkube-node-tsjgl"
Aug 05 17:12:55.174346 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:12:55.173941    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ac2f6cf0-11ca-47f3-8597-5cc1db578e78-var-lib-kubelet\") pod \"tuned-nb82s\" (UID: \"ac2f6cf0-11ca-47f3-8597-5cc1db578e78\") " pod="openshift-cluster-node-tuning-operator/tuned-nb82s"
Aug 05 17:12:55.174346 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:12:55.173968    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/61599c74-5517-43f4-bed8-a232408b44c0-host\") pod \"node-ca-lrdmn\" (UID: \"61599c74-5517-43f4-bed8-a232408b44c0\") " pod="openshift-image-registry/node-ca-lrdmn"
Aug 05 17:12:55.174346 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:12:55.173985    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/c680deac-4ffe-40ff-b521-80beb96c5196-hosts-file\") pod \"node-resolver-qz2cd\" (UID: \"c680deac-4ffe-40ff-b521-80beb96c5196\") " pod="openshift-dns/node-resolver-qz2cd"
Aug 05 17:12:55.174346 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:12:55.173992    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/d662bf46-8fcd-40b5-94fa-eded9b062699-log-socket\") pod \"ovnkube-node-tsjgl\" (UID: \"d662bf46-8fcd-40b5-94fa-eded9b062699\") " pod="openshift-ovn-kubernetes/ovnkube-node-tsjgl"
Aug 05 17:12:55.174346 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:12:55.174003    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/d662bf46-8fcd-40b5-94fa-eded9b062699-host-cni-bin\") pod \"ovnkube-node-tsjgl\" (UID: \"d662bf46-8fcd-40b5-94fa-eded9b062699\") " pod="openshift-ovn-kubernetes/ovnkube-node-tsjgl"
Aug 05 17:12:55.174346 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:12:55.174000    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/ac2f6cf0-11ca-47f3-8597-5cc1db578e78-etc-sysctl-d\") pod \"tuned-nb82s\" (UID: \"ac2f6cf0-11ca-47f3-8597-5cc1db578e78\") " pod="openshift-cluster-node-tuning-operator/tuned-nb82s"
Aug 05 17:12:55.174346 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:12:55.174049    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/61599c74-5517-43f4-bed8-a232408b44c0-host\") pod \"node-ca-lrdmn\" (UID: \"61599c74-5517-43f4-bed8-a232408b44c0\") " pod="openshift-image-registry/node-ca-lrdmn"
Aug 05 17:12:55.175395 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:12:55.174062    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/f2ecc651-01ec-4fa1-a7c1-780b3a963996-dbus\") pod \"global-pull-secret-syncer-9jxp4\" (UID: \"f2ecc651-01ec-4fa1-a7c1-780b3a963996\") " pod="kube-system/global-pull-secret-syncer-9jxp4"
Aug 05 17:12:55.175395 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:12:55.174068    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/c680deac-4ffe-40ff-b521-80beb96c5196-hosts-file\") pod \"node-resolver-qz2cd\" (UID: \"c680deac-4ffe-40ff-b521-80beb96c5196\") " pod="openshift-dns/node-resolver-qz2cd"
Aug 05 17:12:55.175395 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:12:55.174126    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/f2ecc651-01ec-4fa1-a7c1-780b3a963996-dbus\") pod \"global-pull-secret-syncer-9jxp4\" (UID: \"f2ecc651-01ec-4fa1-a7c1-780b3a963996\") " pod="kube-system/global-pull-secret-syncer-9jxp4"
Aug 05 17:12:55.175395 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:12:55.174153    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/ac2f6cf0-11ca-47f3-8597-5cc1db578e78-etc-sysctl-conf\") pod \"tuned-nb82s\" (UID: \"ac2f6cf0-11ca-47f3-8597-5cc1db578e78\") " pod="openshift-cluster-node-tuning-operator/tuned-nb82s"
Aug 05 17:12:55.175395 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:12:55.174180    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5dmf8\" (UniqueName: \"kubernetes.io/projected/ac2f6cf0-11ca-47f3-8597-5cc1db578e78-kube-api-access-5dmf8\") pod \"tuned-nb82s\" (UID: \"ac2f6cf0-11ca-47f3-8597-5cc1db578e78\") " pod="openshift-cluster-node-tuning-operator/tuned-nb82s"
Aug 05 17:12:55.175395 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:12:55.174217    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rw4qx\" (UniqueName: \"kubernetes.io/projected/d662bf46-8fcd-40b5-94fa-eded9b062699-kube-api-access-rw4qx\") pod \"ovnkube-node-tsjgl\" (UID: \"d662bf46-8fcd-40b5-94fa-eded9b062699\") " pod="openshift-ovn-kubernetes/ovnkube-node-tsjgl"
Aug 05 17:12:55.175395 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:12:55.174232    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/ac2f6cf0-11ca-47f3-8597-5cc1db578e78-etc-sysctl-d\") pod \"tuned-nb82s\" (UID: \"ac2f6cf0-11ca-47f3-8597-5cc1db578e78\") " pod="openshift-cluster-node-tuning-operator/tuned-nb82s"
Aug 05 17:12:55.175395 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:12:55.174286    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/ac2f6cf0-11ca-47f3-8597-5cc1db578e78-etc-sysctl-conf\") pod \"tuned-nb82s\" (UID: \"ac2f6cf0-11ca-47f3-8597-5cc1db578e78\") " pod="openshift-cluster-node-tuning-operator/tuned-nb82s"
Aug 05 17:12:55.175395 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:12:55.174292    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/df7e05c7-620a-4246-bd4b-0e2c2009335b-kubelet-dir\") pod \"aws-ebs-csi-driver-node-7h5q8\" (UID: \"df7e05c7-620a-4246-bd4b-0e2c2009335b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7h5q8"
Aug 05 17:12:55.175395 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:12:55.174238    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/df7e05c7-620a-4246-bd4b-0e2c2009335b-kubelet-dir\") pod \"aws-ebs-csi-driver-node-7h5q8\" (UID: \"df7e05c7-620a-4246-bd4b-0e2c2009335b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7h5q8"
Aug 05 17:12:55.175395 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:12:55.174356    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2d2kp\" (UniqueName: \"kubernetes.io/projected/df7e05c7-620a-4246-bd4b-0e2c2009335b-kube-api-access-2d2kp\") pod \"aws-ebs-csi-driver-node-7h5q8\" (UID: \"df7e05c7-620a-4246-bd4b-0e2c2009335b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7h5q8"
Aug 05 17:12:55.175395 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:12:55.174385    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f2ecc651-01ec-4fa1-a7c1-780b3a963996-original-pull-secret\") pod \"global-pull-secret-syncer-9jxp4\" (UID: \"f2ecc651-01ec-4fa1-a7c1-780b3a963996\") " pod="kube-system/global-pull-secret-syncer-9jxp4"
Aug 05 17:12:55.175395 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:12:55.174427    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/d662bf46-8fcd-40b5-94fa-eded9b062699-env-overrides\") pod \"ovnkube-node-tsjgl\" (UID: \"d662bf46-8fcd-40b5-94fa-eded9b062699\") " pod="openshift-ovn-kubernetes/ovnkube-node-tsjgl"
Aug 05 17:12:55.175395 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:12:55.174434    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/d662bf46-8fcd-40b5-94fa-eded9b062699-host-kubelet\") pod \"ovnkube-node-tsjgl\" (UID: \"d662bf46-8fcd-40b5-94fa-eded9b062699\") " pod="openshift-ovn-kubernetes/ovnkube-node-tsjgl"
Aug 05 17:12:55.175395 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:12:55.174461    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/d662bf46-8fcd-40b5-94fa-eded9b062699-host-run-netns\") pod \"ovnkube-node-tsjgl\" (UID: \"d662bf46-8fcd-40b5-94fa-eded9b062699\") " pod="openshift-ovn-kubernetes/ovnkube-node-tsjgl"
Aug 05 17:12:55.175395 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:12:55.174485    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ncqk2\" (UniqueName: \"kubernetes.io/projected/61599c74-5517-43f4-bed8-a232408b44c0-kube-api-access-ncqk2\") pod \"node-ca-lrdmn\" (UID: \"61599c74-5517-43f4-bed8-a232408b44c0\") " pod="openshift-image-registry/node-ca-lrdmn"
Aug 05 17:12:55.175395 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:12:55.174518    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c680deac-4ffe-40ff-b521-80beb96c5196-tmp-dir\") pod \"node-resolver-qz2cd\" (UID: \"c680deac-4ffe-40ff-b521-80beb96c5196\") " pod="openshift-dns/node-resolver-qz2cd"
Aug 05 17:12:55.176279 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:12:55.174546    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ac2f6cf0-11ca-47f3-8597-5cc1db578e78-tmp\") pod \"tuned-nb82s\" (UID: \"ac2f6cf0-11ca-47f3-8597-5cc1db578e78\") " pod="openshift-cluster-node-tuning-operator/tuned-nb82s"
Aug 05 17:12:55.176279 ip-10-0-135-2 kubenswrapper[2571]: E0805 17:12:55.174564    2571 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Aug 05 17:12:55.176279 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:12:55.174591    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/d662bf46-8fcd-40b5-94fa-eded9b062699-host-run-netns\") pod \"ovnkube-node-tsjgl\" (UID: \"d662bf46-8fcd-40b5-94fa-eded9b062699\") " pod="openshift-ovn-kubernetes/ovnkube-node-tsjgl"
Aug 05 17:12:55.176279 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:12:55.174624    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/f2ecc651-01ec-4fa1-a7c1-780b3a963996-kubelet-config\") pod \"global-pull-secret-syncer-9jxp4\" (UID: \"f2ecc651-01ec-4fa1-a7c1-780b3a963996\") " pod="kube-system/global-pull-secret-syncer-9jxp4"
Aug 05 17:12:55.176279 ip-10-0-135-2 kubenswrapper[2571]: E0805 17:12:55.174636    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f2ecc651-01ec-4fa1-a7c1-780b3a963996-original-pull-secret podName:f2ecc651-01ec-4fa1-a7c1-780b3a963996 nodeName:}" failed. No retries permitted until 2026-08-05 17:12:55.674616095 +0000 UTC m=+3.041982423 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/f2ecc651-01ec-4fa1-a7c1-780b3a963996-original-pull-secret") pod "global-pull-secret-syncer-9jxp4" (UID: "f2ecc651-01ec-4fa1-a7c1-780b3a963996") : object "kube-system"/"original-pull-secret" not registered
Aug 05 17:12:55.176279 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:12:55.174667    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/f2ecc651-01ec-4fa1-a7c1-780b3a963996-kubelet-config\") pod \"global-pull-secret-syncer-9jxp4\" (UID: \"f2ecc651-01ec-4fa1-a7c1-780b3a963996\") " pod="kube-system/global-pull-secret-syncer-9jxp4"
Aug 05 17:12:55.176279 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:12:55.174688    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ac2f6cf0-11ca-47f3-8597-5cc1db578e78-etc-kubernetes\") pod \"tuned-nb82s\" (UID: \"ac2f6cf0-11ca-47f3-8597-5cc1db578e78\") " pod="openshift-cluster-node-tuning-operator/tuned-nb82s"
Aug 05 17:12:55.176279 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:12:55.174709    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/d662bf46-8fcd-40b5-94fa-eded9b062699-host-kubelet\") pod \"ovnkube-node-tsjgl\" (UID: \"d662bf46-8fcd-40b5-94fa-eded9b062699\") " pod="openshift-ovn-kubernetes/ovnkube-node-tsjgl"
Aug 05 17:12:55.176279 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:12:55.174725    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d662bf46-8fcd-40b5-94fa-eded9b062699-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-tsjgl\" (UID: \"d662bf46-8fcd-40b5-94fa-eded9b062699\") " pod="openshift-ovn-kubernetes/ovnkube-node-tsjgl"
Aug 05 17:12:55.176279 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:12:55.174755    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/df7e05c7-620a-4246-bd4b-0e2c2009335b-sys-fs\") pod \"aws-ebs-csi-driver-node-7h5q8\" (UID: \"df7e05c7-620a-4246-bd4b-0e2c2009335b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7h5q8"
Aug 05 17:12:55.176279 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:12:55.174765    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ac2f6cf0-11ca-47f3-8597-5cc1db578e78-etc-kubernetes\") pod \"tuned-nb82s\" (UID: \"ac2f6cf0-11ca-47f3-8597-5cc1db578e78\") " pod="openshift-cluster-node-tuning-operator/tuned-nb82s"
Aug 05 17:12:55.176279 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:12:55.174787    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/d662bf46-8fcd-40b5-94fa-eded9b062699-node-log\") pod \"ovnkube-node-tsjgl\" (UID: \"d662bf46-8fcd-40b5-94fa-eded9b062699\") " pod="openshift-ovn-kubernetes/ovnkube-node-tsjgl"
Aug 05 17:12:55.176279 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:12:55.174819    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d662bf46-8fcd-40b5-94fa-eded9b062699-host-run-ovn-kubernetes\") pod \"ovnkube-node-tsjgl\" (UID: \"d662bf46-8fcd-40b5-94fa-eded9b062699\") " pod="openshift-ovn-kubernetes/ovnkube-node-tsjgl"
Aug 05 17:12:55.176279 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:12:55.174822    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/df7e05c7-620a-4246-bd4b-0e2c2009335b-sys-fs\") pod \"aws-ebs-csi-driver-node-7h5q8\" (UID: \"df7e05c7-620a-4246-bd4b-0e2c2009335b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7h5q8"
Aug 05 17:12:55.176279 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:12:55.174832    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d662bf46-8fcd-40b5-94fa-eded9b062699-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-tsjgl\" (UID: \"d662bf46-8fcd-40b5-94fa-eded9b062699\") " pod="openshift-ovn-kubernetes/ovnkube-node-tsjgl"
Aug 05 17:12:55.176279 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:12:55.174869    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/ac2f6cf0-11ca-47f3-8597-5cc1db578e78-etc-sysconfig\") pod \"tuned-nb82s\" (UID: \"ac2f6cf0-11ca-47f3-8597-5cc1db578e78\") " pod="openshift-cluster-node-tuning-operator/tuned-nb82s"
Aug 05 17:12:55.176279 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:12:55.174885    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d662bf46-8fcd-40b5-94fa-eded9b062699-host-run-ovn-kubernetes\") pod \"ovnkube-node-tsjgl\" (UID: \"d662bf46-8fcd-40b5-94fa-eded9b062699\") " pod="openshift-ovn-kubernetes/ovnkube-node-tsjgl"
Aug 05 17:12:55.177207 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:12:55.174891    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/d662bf46-8fcd-40b5-94fa-eded9b062699-node-log\") pod \"ovnkube-node-tsjgl\" (UID: \"d662bf46-8fcd-40b5-94fa-eded9b062699\") " pod="openshift-ovn-kubernetes/ovnkube-node-tsjgl"
Aug 05 17:12:55.177207 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:12:55.174897    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/df7e05c7-620a-4246-bd4b-0e2c2009335b-socket-dir\") pod \"aws-ebs-csi-driver-node-7h5q8\" (UID: \"df7e05c7-620a-4246-bd4b-0e2c2009335b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7h5q8"
Aug 05 17:12:55.177207 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:12:55.174932    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ac2f6cf0-11ca-47f3-8597-5cc1db578e78-sys\") pod \"tuned-nb82s\" (UID: \"ac2f6cf0-11ca-47f3-8597-5cc1db578e78\") " pod="openshift-cluster-node-tuning-operator/tuned-nb82s"
Aug 05 17:12:55.177207 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:12:55.174947    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/ac2f6cf0-11ca-47f3-8597-5cc1db578e78-etc-sysconfig\") pod \"tuned-nb82s\" (UID: \"ac2f6cf0-11ca-47f3-8597-5cc1db578e78\") " pod="openshift-cluster-node-tuning-operator/tuned-nb82s"
Aug 05 17:12:55.177207 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:12:55.174966    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rf7fb\" (UniqueName: \"kubernetes.io/projected/c680deac-4ffe-40ff-b521-80beb96c5196-kube-api-access-rf7fb\") pod \"node-resolver-qz2cd\" (UID: \"c680deac-4ffe-40ff-b521-80beb96c5196\") " pod="openshift-dns/node-resolver-qz2cd"
Aug 05 17:12:55.177207 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:12:55.174971    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c680deac-4ffe-40ff-b521-80beb96c5196-tmp-dir\") pod \"node-resolver-qz2cd\" (UID: \"c680deac-4ffe-40ff-b521-80beb96c5196\") " pod="openshift-dns/node-resolver-qz2cd"
Aug 05 17:12:55.177207 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:12:55.174991    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/ac2f6cf0-11ca-47f3-8597-5cc1db578e78-etc-modprobe-d\") pod \"tuned-nb82s\" (UID: \"ac2f6cf0-11ca-47f3-8597-5cc1db578e78\") " pod="openshift-cluster-node-tuning-operator/tuned-nb82s"
Aug 05 17:12:55.177207 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:12:55.175002    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ac2f6cf0-11ca-47f3-8597-5cc1db578e78-sys\") pod \"tuned-nb82s\" (UID: \"ac2f6cf0-11ca-47f3-8597-5cc1db578e78\") " pod="openshift-cluster-node-tuning-operator/tuned-nb82s"
Aug 05 17:12:55.177207 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:12:55.175018    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/d662bf46-8fcd-40b5-94fa-eded9b062699-host-cni-netd\") pod \"ovnkube-node-tsjgl\" (UID: \"d662bf46-8fcd-40b5-94fa-eded9b062699\") " pod="openshift-ovn-kubernetes/ovnkube-node-tsjgl"
Aug 05 17:12:55.177207 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:12:55.175020    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/df7e05c7-620a-4246-bd4b-0e2c2009335b-socket-dir\") pod \"aws-ebs-csi-driver-node-7h5q8\" (UID: \"df7e05c7-620a-4246-bd4b-0e2c2009335b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7h5q8"
Aug 05 17:12:55.177207 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:12:55.175049    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/d662bf46-8fcd-40b5-94fa-eded9b062699-host-slash\") pod \"ovnkube-node-tsjgl\" (UID: \"d662bf46-8fcd-40b5-94fa-eded9b062699\") " pod="openshift-ovn-kubernetes/ovnkube-node-tsjgl"
Aug 05 17:12:55.177207 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:12:55.175075    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/d662bf46-8fcd-40b5-94fa-eded9b062699-ovnkube-config\") pod \"ovnkube-node-tsjgl\" (UID: \"d662bf46-8fcd-40b5-94fa-eded9b062699\") " pod="openshift-ovn-kubernetes/ovnkube-node-tsjgl"
Aug 05 17:12:55.177207 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:12:55.175102    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/d662bf46-8fcd-40b5-94fa-eded9b062699-host-cni-netd\") pod \"ovnkube-node-tsjgl\" (UID: \"d662bf46-8fcd-40b5-94fa-eded9b062699\") " pod="openshift-ovn-kubernetes/ovnkube-node-tsjgl"
Aug 05 17:12:55.177207 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:12:55.175115    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/61599c74-5517-43f4-bed8-a232408b44c0-serviceca\") pod \"node-ca-lrdmn\" (UID: \"61599c74-5517-43f4-bed8-a232408b44c0\") " pod="openshift-image-registry/node-ca-lrdmn"
Aug 05 17:12:55.177207 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:12:55.175134    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/ac2f6cf0-11ca-47f3-8597-5cc1db578e78-etc-modprobe-d\") pod \"tuned-nb82s\" (UID: \"ac2f6cf0-11ca-47f3-8597-5cc1db578e78\") " pod="openshift-cluster-node-tuning-operator/tuned-nb82s"
Aug 05 17:12:55.177207 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:12:55.175142    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7xgvj\" (UniqueName: \"kubernetes.io/projected/a78df3c5-f92c-4dc6-a56c-a794929aa911-kube-api-access-7xgvj\") pod \"network-check-target-whps7\" (UID: \"a78df3c5-f92c-4dc6-a56c-a794929aa911\") " pod="openshift-network-diagnostics/network-check-target-whps7"
Aug 05 17:12:55.177207 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:12:55.175149    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/d662bf46-8fcd-40b5-94fa-eded9b062699-host-slash\") pod \"ovnkube-node-tsjgl\" (UID: \"d662bf46-8fcd-40b5-94fa-eded9b062699\") " pod="openshift-ovn-kubernetes/ovnkube-node-tsjgl"
Aug 05 17:12:55.177207 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:12:55.175303    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/d662bf46-8fcd-40b5-94fa-eded9b062699-ovn-node-metrics-cert\") pod \"ovnkube-node-tsjgl\" (UID: \"d662bf46-8fcd-40b5-94fa-eded9b062699\") " pod="openshift-ovn-kubernetes/ovnkube-node-tsjgl"
Aug 05 17:12:55.178190 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:12:55.175340    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/ac2f6cf0-11ca-47f3-8597-5cc1db578e78-lib-modules\") pod \"tuned-nb82s\" (UID: \"ac2f6cf0-11ca-47f3-8597-5cc1db578e78\") " pod="openshift-cluster-node-tuning-operator/tuned-nb82s"
Aug 05 17:12:55.178190 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:12:55.175365    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/df7e05c7-620a-4246-bd4b-0e2c2009335b-etc-selinux\") pod \"aws-ebs-csi-driver-node-7h5q8\" (UID: \"df7e05c7-620a-4246-bd4b-0e2c2009335b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7h5q8"
Aug 05 17:12:55.178190 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:12:55.175557    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/b0520f9c-efe2-43b9-8623-411b55bbc874-agent-certs\") pod \"konnectivity-agent-zmr54\" (UID: \"b0520f9c-efe2-43b9-8623-411b55bbc874\") " pod="kube-system/konnectivity-agent-zmr54"
Aug 05 17:12:55.178190 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:12:55.175593    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/b0520f9c-efe2-43b9-8623-411b55bbc874-konnectivity-ca\") pod \"konnectivity-agent-zmr54\" (UID: \"b0520f9c-efe2-43b9-8623-411b55bbc874\") " pod="kube-system/konnectivity-agent-zmr54"
Aug 05 17:12:55.178190 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:12:55.175621    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/ac2f6cf0-11ca-47f3-8597-5cc1db578e78-host\") pod \"tuned-nb82s\" (UID: \"ac2f6cf0-11ca-47f3-8597-5cc1db578e78\") " pod="openshift-cluster-node-tuning-operator/tuned-nb82s"
Aug 05 17:12:55.178190 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:12:55.175647    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d662bf46-8fcd-40b5-94fa-eded9b062699-run-openvswitch\") pod \"ovnkube-node-tsjgl\" (UID: \"d662bf46-8fcd-40b5-94fa-eded9b062699\") " pod="openshift-ovn-kubernetes/ovnkube-node-tsjgl"
Aug 05 17:12:55.178190 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:12:55.175672    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/61599c74-5517-43f4-bed8-a232408b44c0-serviceca\") pod \"node-ca-lrdmn\" (UID: \"61599c74-5517-43f4-bed8-a232408b44c0\") " pod="openshift-image-registry/node-ca-lrdmn"
Aug 05 17:12:55.178190 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:12:55.175727    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d662bf46-8fcd-40b5-94fa-eded9b062699-run-openvswitch\") pod \"ovnkube-node-tsjgl\" (UID: \"d662bf46-8fcd-40b5-94fa-eded9b062699\") " pod="openshift-ovn-kubernetes/ovnkube-node-tsjgl"
Aug 05 17:12:55.178190 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:12:55.175765    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/d662bf46-8fcd-40b5-94fa-eded9b062699-ovnkube-config\") pod \"ovnkube-node-tsjgl\" (UID: \"d662bf46-8fcd-40b5-94fa-eded9b062699\") " pod="openshift-ovn-kubernetes/ovnkube-node-tsjgl"
Aug 05 17:12:55.178190 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:12:55.175802    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/df7e05c7-620a-4246-bd4b-0e2c2009335b-etc-selinux\") pod \"aws-ebs-csi-driver-node-7h5q8\" (UID: \"df7e05c7-620a-4246-bd4b-0e2c2009335b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7h5q8"
Aug 05 17:12:55.178190 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:12:55.176194    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/ac2f6cf0-11ca-47f3-8597-5cc1db578e78-etc-tuned\") pod \"tuned-nb82s\" (UID: \"ac2f6cf0-11ca-47f3-8597-5cc1db578e78\") " pod="openshift-cluster-node-tuning-operator/tuned-nb82s"
Aug 05 17:12:55.178190 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:12:55.175718    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/d662bf46-8fcd-40b5-94fa-eded9b062699-ovnkube-script-lib\") pod \"ovnkube-node-tsjgl\" (UID: \"d662bf46-8fcd-40b5-94fa-eded9b062699\") " pod="openshift-ovn-kubernetes/ovnkube-node-tsjgl"
Aug 05 17:12:55.178190 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:12:55.176262    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/df7e05c7-620a-4246-bd4b-0e2c2009335b-device-dir\") pod \"aws-ebs-csi-driver-node-7h5q8\" (UID: \"df7e05c7-620a-4246-bd4b-0e2c2009335b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7h5q8"
Aug 05 17:12:55.178190 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:12:55.176346    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/df7e05c7-620a-4246-bd4b-0e2c2009335b-device-dir\") pod \"aws-ebs-csi-driver-node-7h5q8\" (UID: \"df7e05c7-620a-4246-bd4b-0e2c2009335b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7h5q8"
Aug 05 17:12:55.178190 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:12:55.176473    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/ac2f6cf0-11ca-47f3-8597-5cc1db578e78-lib-modules\") pod \"tuned-nb82s\" (UID: \"ac2f6cf0-11ca-47f3-8597-5cc1db578e78\") " pod="openshift-cluster-node-tuning-operator/tuned-nb82s"
Aug 05 17:12:55.178190 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:12:55.176823    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/b0520f9c-efe2-43b9-8623-411b55bbc874-konnectivity-ca\") pod \"konnectivity-agent-zmr54\" (UID: \"b0520f9c-efe2-43b9-8623-411b55bbc874\") " pod="kube-system/konnectivity-agent-zmr54"
Aug 05 17:12:55.178190 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:12:55.176930    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/ac2f6cf0-11ca-47f3-8597-5cc1db578e78-host\") pod \"tuned-nb82s\" (UID: \"ac2f6cf0-11ca-47f3-8597-5cc1db578e78\") " pod="openshift-cluster-node-tuning-operator/tuned-nb82s"
Aug 05 17:12:55.178190 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:12:55.176957    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/d662bf46-8fcd-40b5-94fa-eded9b062699-ovnkube-script-lib\") pod \"ovnkube-node-tsjgl\" (UID: \"d662bf46-8fcd-40b5-94fa-eded9b062699\") " pod="openshift-ovn-kubernetes/ovnkube-node-tsjgl"
Aug 05 17:12:55.179125 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:12:55.177379    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ac2f6cf0-11ca-47f3-8597-5cc1db578e78-tmp\") pod \"tuned-nb82s\" (UID: \"ac2f6cf0-11ca-47f3-8597-5cc1db578e78\") " pod="openshift-cluster-node-tuning-operator/tuned-nb82s"
Aug 05 17:12:55.179194 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:12:55.179176    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/b0520f9c-efe2-43b9-8623-411b55bbc874-agent-certs\") pod \"konnectivity-agent-zmr54\" (UID: \"b0520f9c-efe2-43b9-8623-411b55bbc874\") " pod="kube-system/konnectivity-agent-zmr54"
Aug 05 17:12:55.179468 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:12:55.179440    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/d662bf46-8fcd-40b5-94fa-eded9b062699-ovn-node-metrics-cert\") pod \"ovnkube-node-tsjgl\" (UID: \"d662bf46-8fcd-40b5-94fa-eded9b062699\") " pod="openshift-ovn-kubernetes/ovnkube-node-tsjgl"
Aug 05 17:12:55.205114 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:12:55.205061    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-td69h\" (UniqueName: \"kubernetes.io/projected/da6f032b-35a1-4c51-b115-defaf9b2a9a5-kube-api-access-td69h\") pod \"network-metrics-daemon-677x7\" (UID: \"da6f032b-35a1-4c51-b115-defaf9b2a9a5\") " pod="openshift-multus/network-metrics-daemon-677x7"
Aug 05 17:12:55.206198 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:12:55.206179    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ncqk2\" (UniqueName: \"kubernetes.io/projected/61599c74-5517-43f4-bed8-a232408b44c0-kube-api-access-ncqk2\") pod \"node-ca-lrdmn\" (UID: \"61599c74-5517-43f4-bed8-a232408b44c0\") " pod="openshift-image-registry/node-ca-lrdmn"
Aug 05 17:12:55.206654 ip-10-0-135-2 kubenswrapper[2571]: E0805 17:12:55.206638    2571 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Aug 05 17:12:55.206731 ip-10-0-135-2 kubenswrapper[2571]: E0805 17:12:55.206659    2571 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Aug 05 17:12:55.206731 ip-10-0-135-2 kubenswrapper[2571]: E0805 17:12:55.206673    2571 projected.go:196] Error preparing data for projected volume kube-api-access-7xgvj for pod openshift-network-diagnostics/network-check-target-whps7: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Aug 05 17:12:55.206829 ip-10-0-135-2 kubenswrapper[2571]: E0805 17:12:55.206731    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a78df3c5-f92c-4dc6-a56c-a794929aa911-kube-api-access-7xgvj podName:a78df3c5-f92c-4dc6-a56c-a794929aa911 nodeName:}" failed. No retries permitted until 2026-08-05 17:12:55.706718285 +0000 UTC m=+3.074084608 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-7xgvj" (UniqueName: "kubernetes.io/projected/a78df3c5-f92c-4dc6-a56c-a794929aa911-kube-api-access-7xgvj") pod "network-check-target-whps7" (UID: "a78df3c5-f92c-4dc6-a56c-a794929aa911") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Aug 05 17:12:55.211707 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:12:55.211313    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2d2kp\" (UniqueName: \"kubernetes.io/projected/df7e05c7-620a-4246-bd4b-0e2c2009335b-kube-api-access-2d2kp\") pod \"aws-ebs-csi-driver-node-7h5q8\" (UID: \"df7e05c7-620a-4246-bd4b-0e2c2009335b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7h5q8"
Aug 05 17:12:55.211707 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:12:55.211657    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5dmf8\" (UniqueName: \"kubernetes.io/projected/ac2f6cf0-11ca-47f3-8597-5cc1db578e78-kube-api-access-5dmf8\") pod \"tuned-nb82s\" (UID: \"ac2f6cf0-11ca-47f3-8597-5cc1db578e78\") " pod="openshift-cluster-node-tuning-operator/tuned-nb82s"
Aug 05 17:12:55.212018 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:12:55.212002    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rw4qx\" (UniqueName: \"kubernetes.io/projected/d662bf46-8fcd-40b5-94fa-eded9b062699-kube-api-access-rw4qx\") pod \"ovnkube-node-tsjgl\" (UID: \"d662bf46-8fcd-40b5-94fa-eded9b062699\") " pod="openshift-ovn-kubernetes/ovnkube-node-tsjgl"
Aug 05 17:12:55.213131 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:12:55.213102    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rf7fb\" (UniqueName: \"kubernetes.io/projected/c680deac-4ffe-40ff-b521-80beb96c5196-kube-api-access-rf7fb\") pod \"node-resolver-qz2cd\" (UID: \"c680deac-4ffe-40ff-b521-80beb96c5196\") " pod="openshift-dns/node-resolver-qz2cd"
Aug 05 17:12:55.267271 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:12:55.267242    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-vzp52"
Aug 05 17:12:55.278359 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:12:55.278336    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-txqn4"
Aug 05 17:12:55.288744 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:12:55.288720    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-mgfr4"
Aug 05 17:12:55.294530 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:12:55.294509    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-lrdmn"
Aug 05 17:12:55.302538 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:12:55.302517    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-qz2cd"
Aug 05 17:12:55.311271 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:12:55.311238    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7h5q8"
Aug 05 17:12:55.318028 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:12:55.317998    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-nb82s"
Aug 05 17:12:55.324789 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:12:55.324772    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-zmr54"
Aug 05 17:12:55.331625 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:12:55.331606    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-tsjgl"
Aug 05 17:12:55.681157 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:12:55.681082    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/da6f032b-35a1-4c51-b115-defaf9b2a9a5-metrics-certs\") pod \"network-metrics-daemon-677x7\" (UID: \"da6f032b-35a1-4c51-b115-defaf9b2a9a5\") " pod="openshift-multus/network-metrics-daemon-677x7"
Aug 05 17:12:55.681157 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:12:55.681132    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f2ecc651-01ec-4fa1-a7c1-780b3a963996-original-pull-secret\") pod \"global-pull-secret-syncer-9jxp4\" (UID: \"f2ecc651-01ec-4fa1-a7c1-780b3a963996\") " pod="kube-system/global-pull-secret-syncer-9jxp4"
Aug 05 17:12:55.681345 ip-10-0-135-2 kubenswrapper[2571]: E0805 17:12:55.681231    2571 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Aug 05 17:12:55.681345 ip-10-0-135-2 kubenswrapper[2571]: E0805 17:12:55.681240    2571 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Aug 05 17:12:55.681345 ip-10-0-135-2 kubenswrapper[2571]: E0805 17:12:55.681284    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f2ecc651-01ec-4fa1-a7c1-780b3a963996-original-pull-secret podName:f2ecc651-01ec-4fa1-a7c1-780b3a963996 nodeName:}" failed. No retries permitted until 2026-08-05 17:12:56.681271819 +0000 UTC m=+4.048638145 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/f2ecc651-01ec-4fa1-a7c1-780b3a963996-original-pull-secret") pod "global-pull-secret-syncer-9jxp4" (UID: "f2ecc651-01ec-4fa1-a7c1-780b3a963996") : object "kube-system"/"original-pull-secret" not registered
Aug 05 17:12:55.681345 ip-10-0-135-2 kubenswrapper[2571]: E0805 17:12:55.681301    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/da6f032b-35a1-4c51-b115-defaf9b2a9a5-metrics-certs podName:da6f032b-35a1-4c51-b115-defaf9b2a9a5 nodeName:}" failed. No retries permitted until 2026-08-05 17:12:56.681292921 +0000 UTC m=+4.048659244 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/da6f032b-35a1-4c51-b115-defaf9b2a9a5-metrics-certs") pod "network-metrics-daemon-677x7" (UID: "da6f032b-35a1-4c51-b115-defaf9b2a9a5") : object "openshift-multus"/"metrics-daemon-secret" not registered
Aug 05 17:12:55.781859 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:12:55.781826    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7xgvj\" (UniqueName: \"kubernetes.io/projected/a78df3c5-f92c-4dc6-a56c-a794929aa911-kube-api-access-7xgvj\") pod \"network-check-target-whps7\" (UID: \"a78df3c5-f92c-4dc6-a56c-a794929aa911\") " pod="openshift-network-diagnostics/network-check-target-whps7"
Aug 05 17:12:55.782027 ip-10-0-135-2 kubenswrapper[2571]: E0805 17:12:55.781958    2571 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Aug 05 17:12:55.782027 ip-10-0-135-2 kubenswrapper[2571]: E0805 17:12:55.781982    2571 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Aug 05 17:12:55.782027 ip-10-0-135-2 kubenswrapper[2571]: E0805 17:12:55.781994    2571 projected.go:196] Error preparing data for projected volume kube-api-access-7xgvj for pod openshift-network-diagnostics/network-check-target-whps7: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Aug 05 17:12:55.782158 ip-10-0-135-2 kubenswrapper[2571]: E0805 17:12:55.782047    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a78df3c5-f92c-4dc6-a56c-a794929aa911-kube-api-access-7xgvj podName:a78df3c5-f92c-4dc6-a56c-a794929aa911 nodeName:}" failed. No retries permitted until 2026-08-05 17:12:56.782033498 +0000 UTC m=+4.149399820 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-7xgvj" (UniqueName: "kubernetes.io/projected/a78df3c5-f92c-4dc6-a56c-a794929aa911-kube-api-access-7xgvj") pod "network-check-target-whps7" (UID: "a78df3c5-f92c-4dc6-a56c-a794929aa911") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Aug 05 17:12:55.854228 ip-10-0-135-2 kubenswrapper[2571]: W0805 17:12:55.854199    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod100f6b72_e895_453f_8647_1f615c2fb780.slice/crio-e652ebf1a9342b4f9b4c1d019926ee7a30fcab16fde6bcd46e06d3bacd4b6488 WatchSource:0}: Error finding container e652ebf1a9342b4f9b4c1d019926ee7a30fcab16fde6bcd46e06d3bacd4b6488: Status 404 returned error can't find the container with id e652ebf1a9342b4f9b4c1d019926ee7a30fcab16fde6bcd46e06d3bacd4b6488
Aug 05 17:12:55.855208 ip-10-0-135-2 kubenswrapper[2571]: W0805 17:12:55.855179    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd662bf46_8fcd_40b5_94fa_eded9b062699.slice/crio-d26d437d7fb89471f0134543f7cc082144c6c05aac287283d0c999fd8d24224a WatchSource:0}: Error finding container d26d437d7fb89471f0134543f7cc082144c6c05aac287283d0c999fd8d24224a: Status 404 returned error can't find the container with id d26d437d7fb89471f0134543f7cc082144c6c05aac287283d0c999fd8d24224a
Aug 05 17:12:55.860534 ip-10-0-135-2 kubenswrapper[2571]: W0805 17:12:55.860514    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod99b228eb_7a25_47a1_a90d_06b62e8f17e0.slice/crio-e75b489363d0c64e07603101b4bc2083f3e1424836a0bbad65d545e0960bbdfd WatchSource:0}: Error finding container e75b489363d0c64e07603101b4bc2083f3e1424836a0bbad65d545e0960bbdfd: Status 404 returned error can't find the container with id e75b489363d0c64e07603101b4bc2083f3e1424836a0bbad65d545e0960bbdfd
Aug 05 17:12:55.861785 ip-10-0-135-2 kubenswrapper[2571]: W0805 17:12:55.861764    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod61599c74_5517_43f4_bed8_a232408b44c0.slice/crio-4bb808333aa408b3b30f4859e74bda6e4eef30eca23b6c0b3a56c1a5b30e15a9 WatchSource:0}: Error finding container 4bb808333aa408b3b30f4859e74bda6e4eef30eca23b6c0b3a56c1a5b30e15a9: Status 404 returned error can't find the container with id 4bb808333aa408b3b30f4859e74bda6e4eef30eca23b6c0b3a56c1a5b30e15a9
Aug 05 17:12:55.863088 ip-10-0-135-2 kubenswrapper[2571]: W0805 17:12:55.863061    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc680deac_4ffe_40ff_b521_80beb96c5196.slice/crio-3a2a7b816a4c8709eae577c6de826f43fb6b96deae26f19bec00749edebf6469 WatchSource:0}: Error finding container 3a2a7b816a4c8709eae577c6de826f43fb6b96deae26f19bec00749edebf6469: Status 404 returned error can't find the container with id 3a2a7b816a4c8709eae577c6de826f43fb6b96deae26f19bec00749edebf6469
Aug 05 17:12:55.864494 ip-10-0-135-2 kubenswrapper[2571]: W0805 17:12:55.864463    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb934c28e_a416_4ddf_8b90_0c9acdd8ce4c.slice/crio-0b20aeb48deac37beeade38d4e9a4372e030d8baaa04862d2b7ee85ec4899a1c WatchSource:0}: Error finding container 0b20aeb48deac37beeade38d4e9a4372e030d8baaa04862d2b7ee85ec4899a1c: Status 404 returned error can't find the container with id 0b20aeb48deac37beeade38d4e9a4372e030d8baaa04862d2b7ee85ec4899a1c
Aug 05 17:12:55.865811 ip-10-0-135-2 kubenswrapper[2571]: W0805 17:12:55.865792    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddf7e05c7_620a_4246_bd4b_0e2c2009335b.slice/crio-c7d56e788dadfd0a652ea35f2b410715a44f058c864ed454610d7035493f6cbe WatchSource:0}: Error finding container c7d56e788dadfd0a652ea35f2b410715a44f058c864ed454610d7035493f6cbe: Status 404 returned error can't find the container with id c7d56e788dadfd0a652ea35f2b410715a44f058c864ed454610d7035493f6cbe
Aug 05 17:12:55.866324 ip-10-0-135-2 kubenswrapper[2571]: W0805 17:12:55.866300    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb0520f9c_efe2_43b9_8623_411b55bbc874.slice/crio-e6584614a73bfc106040564eb4ee37d5f322c84e97ef3a57a6242dbd12d8db89 WatchSource:0}: Error finding container e6584614a73bfc106040564eb4ee37d5f322c84e97ef3a57a6242dbd12d8db89: Status 404 returned error can't find the container with id e6584614a73bfc106040564eb4ee37d5f322c84e97ef3a57a6242dbd12d8db89
Aug 05 17:12:55.867164 ip-10-0-135-2 kubenswrapper[2571]: W0805 17:12:55.867139    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podac2f6cf0_11ca_47f3_8597_5cc1db578e78.slice/crio-71ca0eea53ca417ac71d8561d21f2fbe9b65ff033f71c016e9ded69162a967fe WatchSource:0}: Error finding container 71ca0eea53ca417ac71d8561d21f2fbe9b65ff033f71c016e9ded69162a967fe: Status 404 returned error can't find the container with id 71ca0eea53ca417ac71d8561d21f2fbe9b65ff033f71c016e9ded69162a967fe
Aug 05 17:12:56.056453 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:12:56.056413    2571 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-08-04 17:07:55 +0000 UTC" deadline="2028-03-25 08:38:54.956404743 +0000 UTC"
Aug 05 17:12:56.056453 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:12:56.056448    2571 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14343h25m58.899961673s"
Aug 05 17:12:56.139794 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:12:56.139760    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7h5q8" event={"ID":"df7e05c7-620a-4246-bd4b-0e2c2009335b","Type":"ContainerStarted","Data":"c7d56e788dadfd0a652ea35f2b410715a44f058c864ed454610d7035493f6cbe"}
Aug 05 17:12:56.141178 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:12:56.141139    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-vzp52" event={"ID":"99b228eb-7a25-47a1-a90d-06b62e8f17e0","Type":"ContainerStarted","Data":"e75b489363d0c64e07603101b4bc2083f3e1424836a0bbad65d545e0960bbdfd"}
Aug 05 17:12:56.142544 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:12:56.142514    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-lrdmn" event={"ID":"61599c74-5517-43f4-bed8-a232408b44c0","Type":"ContainerStarted","Data":"4bb808333aa408b3b30f4859e74bda6e4eef30eca23b6c0b3a56c1a5b30e15a9"}
Aug 05 17:12:56.144202 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:12:56.144176    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-tsjgl" event={"ID":"d662bf46-8fcd-40b5-94fa-eded9b062699","Type":"ContainerStarted","Data":"d26d437d7fb89471f0134543f7cc082144c6c05aac287283d0c999fd8d24224a"}
Aug 05 17:12:56.145409 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:12:56.145358    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-txqn4" event={"ID":"b934c28e-a416-4ddf-8b90-0c9acdd8ce4c","Type":"ContainerStarted","Data":"0b20aeb48deac37beeade38d4e9a4372e030d8baaa04862d2b7ee85ec4899a1c"}
Aug 05 17:12:56.146567 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:12:56.146543    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-qz2cd" event={"ID":"c680deac-4ffe-40ff-b521-80beb96c5196","Type":"ContainerStarted","Data":"3a2a7b816a4c8709eae577c6de826f43fb6b96deae26f19bec00749edebf6469"}
Aug 05 17:12:56.147782 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:12:56.147760    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-mgfr4" event={"ID":"100f6b72-e895-453f-8647-1f615c2fb780","Type":"ContainerStarted","Data":"e652ebf1a9342b4f9b4c1d019926ee7a30fcab16fde6bcd46e06d3bacd4b6488"}
Aug 05 17:12:56.149926 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:12:56.149894    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-2.ec2.internal" event={"ID":"bb3fac1e9119c8f4580f4cf12eb2ffb1","Type":"ContainerStarted","Data":"0d80b14538eb4a8ba41adaae968615359a6796d3d2fa3def020cbe6ed27147f9"}
Aug 05 17:12:56.151187 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:12:56.151162    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-nb82s" event={"ID":"ac2f6cf0-11ca-47f3-8597-5cc1db578e78","Type":"ContainerStarted","Data":"71ca0eea53ca417ac71d8561d21f2fbe9b65ff033f71c016e9ded69162a967fe"}
Aug 05 17:12:56.152453 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:12:56.152432    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-zmr54" event={"ID":"b0520f9c-efe2-43b9-8623-411b55bbc874","Type":"ContainerStarted","Data":"e6584614a73bfc106040564eb4ee37d5f322c84e97ef3a57a6242dbd12d8db89"}
Aug 05 17:12:56.689525 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:12:56.688782    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/da6f032b-35a1-4c51-b115-defaf9b2a9a5-metrics-certs\") pod \"network-metrics-daemon-677x7\" (UID: \"da6f032b-35a1-4c51-b115-defaf9b2a9a5\") " pod="openshift-multus/network-metrics-daemon-677x7"
Aug 05 17:12:56.689525 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:12:56.688844    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f2ecc651-01ec-4fa1-a7c1-780b3a963996-original-pull-secret\") pod \"global-pull-secret-syncer-9jxp4\" (UID: \"f2ecc651-01ec-4fa1-a7c1-780b3a963996\") " pod="kube-system/global-pull-secret-syncer-9jxp4"
Aug 05 17:12:56.689525 ip-10-0-135-2 kubenswrapper[2571]: E0805 17:12:56.688975    2571 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Aug 05 17:12:56.689525 ip-10-0-135-2 kubenswrapper[2571]: E0805 17:12:56.689034    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f2ecc651-01ec-4fa1-a7c1-780b3a963996-original-pull-secret podName:f2ecc651-01ec-4fa1-a7c1-780b3a963996 nodeName:}" failed. No retries permitted until 2026-08-05 17:12:58.689016677 +0000 UTC m=+6.056383003 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/f2ecc651-01ec-4fa1-a7c1-780b3a963996-original-pull-secret") pod "global-pull-secret-syncer-9jxp4" (UID: "f2ecc651-01ec-4fa1-a7c1-780b3a963996") : object "kube-system"/"original-pull-secret" not registered
Aug 05 17:12:56.689525 ip-10-0-135-2 kubenswrapper[2571]: E0805 17:12:56.689416    2571 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Aug 05 17:12:56.689525 ip-10-0-135-2 kubenswrapper[2571]: E0805 17:12:56.689483    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/da6f032b-35a1-4c51-b115-defaf9b2a9a5-metrics-certs podName:da6f032b-35a1-4c51-b115-defaf9b2a9a5 nodeName:}" failed. No retries permitted until 2026-08-05 17:12:58.689466352 +0000 UTC m=+6.056832681 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/da6f032b-35a1-4c51-b115-defaf9b2a9a5-metrics-certs") pod "network-metrics-daemon-677x7" (UID: "da6f032b-35a1-4c51-b115-defaf9b2a9a5") : object "openshift-multus"/"metrics-daemon-secret" not registered
Aug 05 17:12:56.789978 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:12:56.789938    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7xgvj\" (UniqueName: \"kubernetes.io/projected/a78df3c5-f92c-4dc6-a56c-a794929aa911-kube-api-access-7xgvj\") pod \"network-check-target-whps7\" (UID: \"a78df3c5-f92c-4dc6-a56c-a794929aa911\") " pod="openshift-network-diagnostics/network-check-target-whps7"
Aug 05 17:12:56.790232 ip-10-0-135-2 kubenswrapper[2571]: E0805 17:12:56.790145    2571 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Aug 05 17:12:56.790232 ip-10-0-135-2 kubenswrapper[2571]: E0805 17:12:56.790178    2571 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Aug 05 17:12:56.790232 ip-10-0-135-2 kubenswrapper[2571]: E0805 17:12:56.790191    2571 projected.go:196] Error preparing data for projected volume kube-api-access-7xgvj for pod openshift-network-diagnostics/network-check-target-whps7: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Aug 05 17:12:56.790430 ip-10-0-135-2 kubenswrapper[2571]: E0805 17:12:56.790249    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a78df3c5-f92c-4dc6-a56c-a794929aa911-kube-api-access-7xgvj podName:a78df3c5-f92c-4dc6-a56c-a794929aa911 nodeName:}" failed. No retries permitted until 2026-08-05 17:12:58.79023062 +0000 UTC m=+6.157596949 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-7xgvj" (UniqueName: "kubernetes.io/projected/a78df3c5-f92c-4dc6-a56c-a794929aa911-kube-api-access-7xgvj") pod "network-check-target-whps7" (UID: "a78df3c5-f92c-4dc6-a56c-a794929aa911") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Aug 05 17:12:57.057610 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:12:57.057564    2571 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-08-04 17:07:55 +0000 UTC" deadline="2028-03-26 19:06:21.997927849 +0000 UTC"
Aug 05 17:12:57.057610 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:12:57.057607    2571 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14377h53m24.940325378s"
Aug 05 17:12:57.131290 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:12:57.131256    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-whps7"
Aug 05 17:12:57.131471 ip-10-0-135-2 kubenswrapper[2571]: E0805 17:12:57.131418    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-whps7" podUID="a78df3c5-f92c-4dc6-a56c-a794929aa911"
Aug 05 17:12:57.131893 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:12:57.131870    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9jxp4"
Aug 05 17:12:57.131992 ip-10-0-135-2 kubenswrapper[2571]: E0805 17:12:57.131975    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9jxp4" podUID="f2ecc651-01ec-4fa1-a7c1-780b3a963996"
Aug 05 17:12:57.132187 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:12:57.132168    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-677x7"
Aug 05 17:12:57.132306 ip-10-0-135-2 kubenswrapper[2571]: E0805 17:12:57.132284    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-677x7" podUID="da6f032b-35a1-4c51-b115-defaf9b2a9a5"
Aug 05 17:12:58.710044 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:12:58.709724    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/da6f032b-35a1-4c51-b115-defaf9b2a9a5-metrics-certs\") pod \"network-metrics-daemon-677x7\" (UID: \"da6f032b-35a1-4c51-b115-defaf9b2a9a5\") " pod="openshift-multus/network-metrics-daemon-677x7"
Aug 05 17:12:58.710547 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:12:58.710050    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f2ecc651-01ec-4fa1-a7c1-780b3a963996-original-pull-secret\") pod \"global-pull-secret-syncer-9jxp4\" (UID: \"f2ecc651-01ec-4fa1-a7c1-780b3a963996\") " pod="kube-system/global-pull-secret-syncer-9jxp4"
Aug 05 17:12:58.710547 ip-10-0-135-2 kubenswrapper[2571]: E0805 17:12:58.710198    2571 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Aug 05 17:12:58.710547 ip-10-0-135-2 kubenswrapper[2571]: E0805 17:12:58.710261    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f2ecc651-01ec-4fa1-a7c1-780b3a963996-original-pull-secret podName:f2ecc651-01ec-4fa1-a7c1-780b3a963996 nodeName:}" failed. No retries permitted until 2026-08-05 17:13:02.710240277 +0000 UTC m=+10.077606605 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/f2ecc651-01ec-4fa1-a7c1-780b3a963996-original-pull-secret") pod "global-pull-secret-syncer-9jxp4" (UID: "f2ecc651-01ec-4fa1-a7c1-780b3a963996") : object "kube-system"/"original-pull-secret" not registered
Aug 05 17:12:58.710761 ip-10-0-135-2 kubenswrapper[2571]: E0805 17:12:58.710730    2571 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Aug 05 17:12:58.710819 ip-10-0-135-2 kubenswrapper[2571]: E0805 17:12:58.710799    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/da6f032b-35a1-4c51-b115-defaf9b2a9a5-metrics-certs podName:da6f032b-35a1-4c51-b115-defaf9b2a9a5 nodeName:}" failed. No retries permitted until 2026-08-05 17:13:02.710778253 +0000 UTC m=+10.078144592 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/da6f032b-35a1-4c51-b115-defaf9b2a9a5-metrics-certs") pod "network-metrics-daemon-677x7" (UID: "da6f032b-35a1-4c51-b115-defaf9b2a9a5") : object "openshift-multus"/"metrics-daemon-secret" not registered
Aug 05 17:12:58.811511 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:12:58.811232    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7xgvj\" (UniqueName: \"kubernetes.io/projected/a78df3c5-f92c-4dc6-a56c-a794929aa911-kube-api-access-7xgvj\") pod \"network-check-target-whps7\" (UID: \"a78df3c5-f92c-4dc6-a56c-a794929aa911\") " pod="openshift-network-diagnostics/network-check-target-whps7"
Aug 05 17:12:58.811511 ip-10-0-135-2 kubenswrapper[2571]: E0805 17:12:58.811386    2571 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Aug 05 17:12:58.811511 ip-10-0-135-2 kubenswrapper[2571]: E0805 17:12:58.811423    2571 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Aug 05 17:12:58.811511 ip-10-0-135-2 kubenswrapper[2571]: E0805 17:12:58.811436    2571 projected.go:196] Error preparing data for projected volume kube-api-access-7xgvj for pod openshift-network-diagnostics/network-check-target-whps7: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Aug 05 17:12:58.811511 ip-10-0-135-2 kubenswrapper[2571]: E0805 17:12:58.811494    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a78df3c5-f92c-4dc6-a56c-a794929aa911-kube-api-access-7xgvj podName:a78df3c5-f92c-4dc6-a56c-a794929aa911 nodeName:}" failed. No retries permitted until 2026-08-05 17:13:02.811475138 +0000 UTC m=+10.178841475 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-7xgvj" (UniqueName: "kubernetes.io/projected/a78df3c5-f92c-4dc6-a56c-a794929aa911-kube-api-access-7xgvj") pod "network-check-target-whps7" (UID: "a78df3c5-f92c-4dc6-a56c-a794929aa911") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Aug 05 17:12:59.131965 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:12:59.131886    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9jxp4"
Aug 05 17:12:59.132154 ip-10-0-135-2 kubenswrapper[2571]: E0805 17:12:59.132031    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9jxp4" podUID="f2ecc651-01ec-4fa1-a7c1-780b3a963996"
Aug 05 17:12:59.134798 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:12:59.134773    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-677x7"
Aug 05 17:12:59.134925 ip-10-0-135-2 kubenswrapper[2571]: E0805 17:12:59.134890    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-677x7" podUID="da6f032b-35a1-4c51-b115-defaf9b2a9a5"
Aug 05 17:12:59.134925 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:12:59.134896    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-whps7"
Aug 05 17:12:59.135044 ip-10-0-135-2 kubenswrapper[2571]: E0805 17:12:59.134993    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-whps7" podUID="a78df3c5-f92c-4dc6-a56c-a794929aa911"
Aug 05 17:12:59.170324 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:12:59.169864    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-2.ec2.internal" event={"ID":"300ebfcee102c70ab72580814e7ccaa4","Type":"ContainerStarted","Data":"eefe6ff912662bc2a4e37b8a01754677eed0dc45df91365fdefa06ca9a274adf"}
Aug 05 17:12:59.191822 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:12:59.191397    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-2.ec2.internal" podStartSLOduration=4.191376879 podStartE2EDuration="4.191376879s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-08-05 17:12:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-05 17:12:56.174949994 +0000 UTC m=+3.542316340" watchObservedRunningTime="2026-08-05 17:12:59.191376879 +0000 UTC m=+6.558743233"
Aug 05 17:13:01.132012 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:13:01.131376    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-677x7"
Aug 05 17:13:01.132012 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:13:01.131459    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-whps7"
Aug 05 17:13:01.132012 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:13:01.131459    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9jxp4"
Aug 05 17:13:01.132012 ip-10-0-135-2 kubenswrapper[2571]: E0805 17:13:01.131592    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-677x7" podUID="da6f032b-35a1-4c51-b115-defaf9b2a9a5"
Aug 05 17:13:01.132012 ip-10-0-135-2 kubenswrapper[2571]: E0805 17:13:01.131673    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-whps7" podUID="a78df3c5-f92c-4dc6-a56c-a794929aa911"
Aug 05 17:13:01.132012 ip-10-0-135-2 kubenswrapper[2571]: E0805 17:13:01.131816    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9jxp4" podUID="f2ecc651-01ec-4fa1-a7c1-780b3a963996"
Aug 05 17:13:02.742258 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:13:02.742210    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f2ecc651-01ec-4fa1-a7c1-780b3a963996-original-pull-secret\") pod \"global-pull-secret-syncer-9jxp4\" (UID: \"f2ecc651-01ec-4fa1-a7c1-780b3a963996\") " pod="kube-system/global-pull-secret-syncer-9jxp4"
Aug 05 17:13:02.742843 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:13:02.742319    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/da6f032b-35a1-4c51-b115-defaf9b2a9a5-metrics-certs\") pod \"network-metrics-daemon-677x7\" (UID: \"da6f032b-35a1-4c51-b115-defaf9b2a9a5\") " pod="openshift-multus/network-metrics-daemon-677x7"
Aug 05 17:13:02.742843 ip-10-0-135-2 kubenswrapper[2571]: E0805 17:13:02.742403    2571 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Aug 05 17:13:02.742843 ip-10-0-135-2 kubenswrapper[2571]: E0805 17:13:02.742454    2571 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Aug 05 17:13:02.742843 ip-10-0-135-2 kubenswrapper[2571]: E0805 17:13:02.742486    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f2ecc651-01ec-4fa1-a7c1-780b3a963996-original-pull-secret podName:f2ecc651-01ec-4fa1-a7c1-780b3a963996 nodeName:}" failed. No retries permitted until 2026-08-05 17:13:10.74246184 +0000 UTC m=+18.109828181 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/f2ecc651-01ec-4fa1-a7c1-780b3a963996-original-pull-secret") pod "global-pull-secret-syncer-9jxp4" (UID: "f2ecc651-01ec-4fa1-a7c1-780b3a963996") : object "kube-system"/"original-pull-secret" not registered
Aug 05 17:13:02.742843 ip-10-0-135-2 kubenswrapper[2571]: E0805 17:13:02.742508    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/da6f032b-35a1-4c51-b115-defaf9b2a9a5-metrics-certs podName:da6f032b-35a1-4c51-b115-defaf9b2a9a5 nodeName:}" failed. No retries permitted until 2026-08-05 17:13:10.742497547 +0000 UTC m=+18.109863876 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/da6f032b-35a1-4c51-b115-defaf9b2a9a5-metrics-certs") pod "network-metrics-daemon-677x7" (UID: "da6f032b-35a1-4c51-b115-defaf9b2a9a5") : object "openshift-multus"/"metrics-daemon-secret" not registered
Aug 05 17:13:02.842934 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:13:02.842849    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7xgvj\" (UniqueName: \"kubernetes.io/projected/a78df3c5-f92c-4dc6-a56c-a794929aa911-kube-api-access-7xgvj\") pod \"network-check-target-whps7\" (UID: \"a78df3c5-f92c-4dc6-a56c-a794929aa911\") " pod="openshift-network-diagnostics/network-check-target-whps7"
Aug 05 17:13:02.843102 ip-10-0-135-2 kubenswrapper[2571]: E0805 17:13:02.842984    2571 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Aug 05 17:13:02.843102 ip-10-0-135-2 kubenswrapper[2571]: E0805 17:13:02.843009    2571 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Aug 05 17:13:02.843102 ip-10-0-135-2 kubenswrapper[2571]: E0805 17:13:02.843020    2571 projected.go:196] Error preparing data for projected volume kube-api-access-7xgvj for pod openshift-network-diagnostics/network-check-target-whps7: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Aug 05 17:13:02.843102 ip-10-0-135-2 kubenswrapper[2571]: E0805 17:13:02.843073    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a78df3c5-f92c-4dc6-a56c-a794929aa911-kube-api-access-7xgvj podName:a78df3c5-f92c-4dc6-a56c-a794929aa911 nodeName:}" failed. No retries permitted until 2026-08-05 17:13:10.843051897 +0000 UTC m=+18.210418231 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-7xgvj" (UniqueName: "kubernetes.io/projected/a78df3c5-f92c-4dc6-a56c-a794929aa911-kube-api-access-7xgvj") pod "network-check-target-whps7" (UID: "a78df3c5-f92c-4dc6-a56c-a794929aa911") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Aug 05 17:13:03.133559 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:13:03.133002    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-677x7"
Aug 05 17:13:03.133559 ip-10-0-135-2 kubenswrapper[2571]: E0805 17:13:03.133136    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-677x7" podUID="da6f032b-35a1-4c51-b115-defaf9b2a9a5"
Aug 05 17:13:03.133559 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:13:03.133551    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-whps7"
Aug 05 17:13:03.133935 ip-10-0-135-2 kubenswrapper[2571]: E0805 17:13:03.133640    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-whps7" podUID="a78df3c5-f92c-4dc6-a56c-a794929aa911"
Aug 05 17:13:03.133935 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:13:03.133784    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9jxp4"
Aug 05 17:13:03.133935 ip-10-0-135-2 kubenswrapper[2571]: E0805 17:13:03.133868    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9jxp4" podUID="f2ecc651-01ec-4fa1-a7c1-780b3a963996"
Aug 05 17:13:04.184143 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:13:04.184111    2571 generic.go:356] "Generic (PLEG): container finished" podID="300ebfcee102c70ab72580814e7ccaa4" containerID="eefe6ff912662bc2a4e37b8a01754677eed0dc45df91365fdefa06ca9a274adf" exitCode=0
Aug 05 17:13:04.184607 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:13:04.184160    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-2.ec2.internal" event={"ID":"300ebfcee102c70ab72580814e7ccaa4","Type":"ContainerDied","Data":"eefe6ff912662bc2a4e37b8a01754677eed0dc45df91365fdefa06ca9a274adf"}
Aug 05 17:13:05.132212 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:13:05.132183    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9jxp4"
Aug 05 17:13:05.132415 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:13:05.132198    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-677x7"
Aug 05 17:13:05.132415 ip-10-0-135-2 kubenswrapper[2571]: E0805 17:13:05.132312    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9jxp4" podUID="f2ecc651-01ec-4fa1-a7c1-780b3a963996"
Aug 05 17:13:05.132415 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:13:05.132323    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-whps7"
Aug 05 17:13:05.132574 ip-10-0-135-2 kubenswrapper[2571]: E0805 17:13:05.132438    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-677x7" podUID="da6f032b-35a1-4c51-b115-defaf9b2a9a5"
Aug 05 17:13:05.132574 ip-10-0-135-2 kubenswrapper[2571]: E0805 17:13:05.132519    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-whps7" podUID="a78df3c5-f92c-4dc6-a56c-a794929aa911"
Aug 05 17:13:07.132056 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:13:07.132024    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-677x7"
Aug 05 17:13:07.132510 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:13:07.132024    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9jxp4"
Aug 05 17:13:07.132510 ip-10-0-135-2 kubenswrapper[2571]: E0805 17:13:07.132127    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-677x7" podUID="da6f032b-35a1-4c51-b115-defaf9b2a9a5"
Aug 05 17:13:07.132510 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:13:07.132142    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-whps7"
Aug 05 17:13:07.132510 ip-10-0-135-2 kubenswrapper[2571]: E0805 17:13:07.132261    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9jxp4" podUID="f2ecc651-01ec-4fa1-a7c1-780b3a963996"
Aug 05 17:13:07.132510 ip-10-0-135-2 kubenswrapper[2571]: E0805 17:13:07.132320    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-whps7" podUID="a78df3c5-f92c-4dc6-a56c-a794929aa911"
Aug 05 17:13:09.131697 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:13:09.131659    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-677x7"
Aug 05 17:13:09.132194 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:13:09.131708    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9jxp4"
Aug 05 17:13:09.132194 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:13:09.131662    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-whps7"
Aug 05 17:13:09.132194 ip-10-0-135-2 kubenswrapper[2571]: E0805 17:13:09.131814    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-677x7" podUID="da6f032b-35a1-4c51-b115-defaf9b2a9a5"
Aug 05 17:13:09.132194 ip-10-0-135-2 kubenswrapper[2571]: E0805 17:13:09.131898    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-whps7" podUID="a78df3c5-f92c-4dc6-a56c-a794929aa911"
Aug 05 17:13:09.132194 ip-10-0-135-2 kubenswrapper[2571]: E0805 17:13:09.131957    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9jxp4" podUID="f2ecc651-01ec-4fa1-a7c1-780b3a963996"
Aug 05 17:13:10.801264 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:13:10.801219    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/da6f032b-35a1-4c51-b115-defaf9b2a9a5-metrics-certs\") pod \"network-metrics-daemon-677x7\" (UID: \"da6f032b-35a1-4c51-b115-defaf9b2a9a5\") " pod="openshift-multus/network-metrics-daemon-677x7"
Aug 05 17:13:10.801770 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:13:10.801284    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f2ecc651-01ec-4fa1-a7c1-780b3a963996-original-pull-secret\") pod \"global-pull-secret-syncer-9jxp4\" (UID: \"f2ecc651-01ec-4fa1-a7c1-780b3a963996\") " pod="kube-system/global-pull-secret-syncer-9jxp4"
Aug 05 17:13:10.801770 ip-10-0-135-2 kubenswrapper[2571]: E0805 17:13:10.801409    2571 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Aug 05 17:13:10.801770 ip-10-0-135-2 kubenswrapper[2571]: E0805 17:13:10.801481    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/da6f032b-35a1-4c51-b115-defaf9b2a9a5-metrics-certs podName:da6f032b-35a1-4c51-b115-defaf9b2a9a5 nodeName:}" failed. No retries permitted until 2026-08-05 17:13:26.801459643 +0000 UTC m=+34.168825979 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/da6f032b-35a1-4c51-b115-defaf9b2a9a5-metrics-certs") pod "network-metrics-daemon-677x7" (UID: "da6f032b-35a1-4c51-b115-defaf9b2a9a5") : object "openshift-multus"/"metrics-daemon-secret" not registered
Aug 05 17:13:10.801770 ip-10-0-135-2 kubenswrapper[2571]: E0805 17:13:10.801411    2571 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Aug 05 17:13:10.801770 ip-10-0-135-2 kubenswrapper[2571]: E0805 17:13:10.801577    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f2ecc651-01ec-4fa1-a7c1-780b3a963996-original-pull-secret podName:f2ecc651-01ec-4fa1-a7c1-780b3a963996 nodeName:}" failed. No retries permitted until 2026-08-05 17:13:26.801561085 +0000 UTC m=+34.168927410 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/f2ecc651-01ec-4fa1-a7c1-780b3a963996-original-pull-secret") pod "global-pull-secret-syncer-9jxp4" (UID: "f2ecc651-01ec-4fa1-a7c1-780b3a963996") : object "kube-system"/"original-pull-secret" not registered
Aug 05 17:13:10.902106 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:13:10.902075    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7xgvj\" (UniqueName: \"kubernetes.io/projected/a78df3c5-f92c-4dc6-a56c-a794929aa911-kube-api-access-7xgvj\") pod \"network-check-target-whps7\" (UID: \"a78df3c5-f92c-4dc6-a56c-a794929aa911\") " pod="openshift-network-diagnostics/network-check-target-whps7"
Aug 05 17:13:10.902269 ip-10-0-135-2 kubenswrapper[2571]: E0805 17:13:10.902252    2571 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Aug 05 17:13:10.902325 ip-10-0-135-2 kubenswrapper[2571]: E0805 17:13:10.902272    2571 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Aug 05 17:13:10.902325 ip-10-0-135-2 kubenswrapper[2571]: E0805 17:13:10.902286    2571 projected.go:196] Error preparing data for projected volume kube-api-access-7xgvj for pod openshift-network-diagnostics/network-check-target-whps7: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Aug 05 17:13:10.902454 ip-10-0-135-2 kubenswrapper[2571]: E0805 17:13:10.902345    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a78df3c5-f92c-4dc6-a56c-a794929aa911-kube-api-access-7xgvj podName:a78df3c5-f92c-4dc6-a56c-a794929aa911 nodeName:}" failed. No retries permitted until 2026-08-05 17:13:26.902328522 +0000 UTC m=+34.269694848 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-7xgvj" (UniqueName: "kubernetes.io/projected/a78df3c5-f92c-4dc6-a56c-a794929aa911-kube-api-access-7xgvj") pod "network-check-target-whps7" (UID: "a78df3c5-f92c-4dc6-a56c-a794929aa911") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Aug 05 17:13:11.132117 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:13:11.132043    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-whps7"
Aug 05 17:13:11.132117 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:13:11.132059    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9jxp4"
Aug 05 17:13:11.132325 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:13:11.132043    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-677x7"
Aug 05 17:13:11.132325 ip-10-0-135-2 kubenswrapper[2571]: E0805 17:13:11.132165    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-whps7" podUID="a78df3c5-f92c-4dc6-a56c-a794929aa911"
Aug 05 17:13:11.132325 ip-10-0-135-2 kubenswrapper[2571]: E0805 17:13:11.132240    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-677x7" podUID="da6f032b-35a1-4c51-b115-defaf9b2a9a5"
Aug 05 17:13:11.132497 ip-10-0-135-2 kubenswrapper[2571]: E0805 17:13:11.132320    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9jxp4" podUID="f2ecc651-01ec-4fa1-a7c1-780b3a963996"
Aug 05 17:13:13.133035 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:13:13.133003    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9jxp4"
Aug 05 17:13:13.133531 ip-10-0-135-2 kubenswrapper[2571]: E0805 17:13:13.133118    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9jxp4" podUID="f2ecc651-01ec-4fa1-a7c1-780b3a963996"
Aug 05 17:13:13.133531 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:13:13.133170    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-677x7"
Aug 05 17:13:13.133531 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:13:13.133303    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-whps7"
Aug 05 17:13:13.133531 ip-10-0-135-2 kubenswrapper[2571]: E0805 17:13:13.133342    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-677x7" podUID="da6f032b-35a1-4c51-b115-defaf9b2a9a5"
Aug 05 17:13:13.133531 ip-10-0-135-2 kubenswrapper[2571]: E0805 17:13:13.133388    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-whps7" podUID="a78df3c5-f92c-4dc6-a56c-a794929aa911"
Aug 05 17:13:15.131651 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:13:15.131625    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-whps7"
Aug 05 17:13:15.132002 ip-10-0-135-2 kubenswrapper[2571]: E0805 17:13:15.131738    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-whps7" podUID="a78df3c5-f92c-4dc6-a56c-a794929aa911"
Aug 05 17:13:15.132085 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:13:15.132065    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-677x7"
Aug 05 17:13:15.132197 ip-10-0-135-2 kubenswrapper[2571]: E0805 17:13:15.132175    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-677x7" podUID="da6f032b-35a1-4c51-b115-defaf9b2a9a5"
Aug 05 17:13:15.132250 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:13:15.131631    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9jxp4"
Aug 05 17:13:15.132307 ip-10-0-135-2 kubenswrapper[2571]: E0805 17:13:15.132265    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9jxp4" podUID="f2ecc651-01ec-4fa1-a7c1-780b3a963996"
Aug 05 17:13:15.216897 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:13:15.216014    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-zmr54" event={"ID":"b0520f9c-efe2-43b9-8623-411b55bbc874","Type":"ContainerStarted","Data":"b3b3d1686e51cb4d308260bc56e101d063e1df3606f7b93977e418f2edbcb52e"}
Aug 05 17:13:15.238781 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:13:15.237917    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-zmr54" podStartSLOduration=5.813946446 podStartE2EDuration="21.237897682s" totalImagesPullingTime="15.423951236s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-05 17:12:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-05 17:13:15.237524879 +0000 UTC m=+22.604891223" watchObservedRunningTime="2026-08-05 17:13:15.237897682 +0000 UTC m=+22.605264025"
Aug 05 17:13:16.236547 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:13:16.236335    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7h5q8" event={"ID":"df7e05c7-620a-4246-bd4b-0e2c2009335b","Type":"ContainerStarted","Data":"a6760ba536a0bcc21f9e1a2016fe58bd85e157eb8a36485a0bd8535e072fe202"}
Aug 05 17:13:16.237819 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:13:16.237799    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-vzp52" event={"ID":"99b228eb-7a25-47a1-a90d-06b62e8f17e0","Type":"ContainerStarted","Data":"df3d71ff4789f5885034340bd3a02b179d1911aea55f1a86bdcee0d950a86676"}
Aug 05 17:13:16.239230 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:13:16.239208    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-lrdmn" event={"ID":"61599c74-5517-43f4-bed8-a232408b44c0","Type":"ContainerStarted","Data":"b16710b36b7a66a4c9561ee8a96dc2d0b94685928a0d7dd5ef81756a2f4e4fa5"}
Aug 05 17:13:16.242154 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:13:16.242132    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-tsjgl" event={"ID":"d662bf46-8fcd-40b5-94fa-eded9b062699","Type":"ContainerStarted","Data":"835cace052d0345d45821214ab5a09f9135cb5ef1f19cda928c7c229b654ddfe"}
Aug 05 17:13:16.242256 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:13:16.242188    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-tsjgl" event={"ID":"d662bf46-8fcd-40b5-94fa-eded9b062699","Type":"ContainerStarted","Data":"958ff6ab9828a2f39424da4dd13d6705172a1b6cac0410bb33911c235cfa6e86"}
Aug 05 17:13:16.242256 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:13:16.242225    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-tsjgl" event={"ID":"d662bf46-8fcd-40b5-94fa-eded9b062699","Type":"ContainerStarted","Data":"69bb403f87954c5b8457dec25b339c6f3d9a3a50639231f4447c6ea9b053d56b"}
Aug 05 17:13:16.242389 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:13:16.242270    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-tsjgl" event={"ID":"d662bf46-8fcd-40b5-94fa-eded9b062699","Type":"ContainerStarted","Data":"dd50060315a7da03b05c7a1e258fd77ef2fe84b1cb6127143d7a90458ecd3512"}
Aug 05 17:13:16.242389 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:13:16.242300    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-tsjgl" event={"ID":"d662bf46-8fcd-40b5-94fa-eded9b062699","Type":"ContainerStarted","Data":"76e99ff1b8070aeb9343520035d57c6194a0c0a96f73cf9d5a116e5da029a1bf"}
Aug 05 17:13:16.242389 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:13:16.242329    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-tsjgl" event={"ID":"d662bf46-8fcd-40b5-94fa-eded9b062699","Type":"ContainerStarted","Data":"b076823dc5a9c4ba442d6c23f797f88386a388f8ad2bdb072e92282108a93055"}
Aug 05 17:13:16.243918 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:13:16.243895    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-2.ec2.internal" event={"ID":"300ebfcee102c70ab72580814e7ccaa4","Type":"ContainerStarted","Data":"8e1b2e0293d8ba41d0bb657b513cffe0426a6ab3c731afc2c0084491e35181d1"}
Aug 05 17:13:16.246616 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:13:16.246585    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-qz2cd" event={"ID":"c680deac-4ffe-40ff-b521-80beb96c5196","Type":"ContainerStarted","Data":"5080c482e17cdf37e402614f5cc6aab7514680bbc1dbb311909f3a74bd18316f"}
Aug 05 17:13:16.248019 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:13:16.247995    2571 generic.go:356] "Generic (PLEG): container finished" podID="100f6b72-e895-453f-8647-1f615c2fb780" containerID="eb5fe12f39e925defb340cdc6486c97f313f3769803a94b83808f346cef39185" exitCode=0
Aug 05 17:13:16.248105 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:13:16.248071    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-mgfr4" event={"ID":"100f6b72-e895-453f-8647-1f615c2fb780","Type":"ContainerDied","Data":"eb5fe12f39e925defb340cdc6486c97f313f3769803a94b83808f346cef39185"}
Aug 05 17:13:16.249494 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:13:16.249464    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-nb82s" event={"ID":"ac2f6cf0-11ca-47f3-8597-5cc1db578e78","Type":"ContainerStarted","Data":"f06d039b116e9d735e04844b12ee6dc41657fd1e5afe9abc7826bc1a70389e4e"}
Aug 05 17:13:16.303701 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:13:16.303619    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-2.ec2.internal" podStartSLOduration=21.303604636 podStartE2EDuration="21.303604636s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-08-05 17:12:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-05 17:13:16.303485479 +0000 UTC m=+23.670851821" watchObservedRunningTime="2026-08-05 17:13:16.303604636 +0000 UTC m=+23.670970980"
Aug 05 17:13:16.303955 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:13:16.303927    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-vzp52" podStartSLOduration=4.073576779 podStartE2EDuration="23.303920535s" totalImagesPullingTime="19.230343756s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-08-05 17:12:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-05 17:13:16.267507041 +0000 UTC m=+23.634873384" watchObservedRunningTime="2026-08-05 17:13:16.303920535 +0000 UTC m=+23.671286923"
Aug 05 17:13:16.355292 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:13:16.355259    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-lrdmn" podStartSLOduration=4.187586709 podStartE2EDuration="23.355248242s" totalImagesPullingTime="19.167661533s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-08-05 17:12:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-05 17:13:16.35507398 +0000 UTC m=+23.722440322" watchObservedRunningTime="2026-08-05 17:13:16.355248242 +0000 UTC m=+23.722614585"
Aug 05 17:13:16.383367 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:13:16.383337    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-qz2cd" podStartSLOduration=4.217437433 podStartE2EDuration="23.383328075s" totalImagesPullingTime="19.165890642s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-08-05 17:12:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-05 17:13:16.383276865 +0000 UTC m=+23.750643207" watchObservedRunningTime="2026-08-05 17:13:16.383328075 +0000 UTC m=+23.750694417"
Aug 05 17:13:16.486447 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:13:16.486428    2571 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Aug 05 17:13:16.931100 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:13:16.931068    2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-zmr54"
Aug 05 17:13:16.931778 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:13:16.931757    2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-zmr54"
Aug 05 17:13:16.948646 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:13:16.948589    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-nb82s" podStartSLOduration=3.7896410339999997 podStartE2EDuration="22.948568294s" totalImagesPullingTime="19.15892726s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-08-05 17:12:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-05 17:13:16.412687493 +0000 UTC m=+23.780053836" watchObservedRunningTime="2026-08-05 17:13:16.948568294 +0000 UTC m=+24.315934638"
Aug 05 17:13:17.041120 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:13:17.040963    2571 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-08-05T17:13:16.4864436Z","UUID":"0681f639-cde5-44be-9135-075d8ea27bd2","Handler":null,"Name":"","Endpoint":""}
Aug 05 17:13:17.043079 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:13:17.043051    2571 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Aug 05 17:13:17.043213 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:13:17.043092    2571 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Aug 05 17:13:17.132006 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:13:17.131977    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-whps7"
Aug 05 17:13:17.132170 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:13:17.132017    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9jxp4"
Aug 05 17:13:17.132170 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:13:17.131977    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-677x7"
Aug 05 17:13:17.132170 ip-10-0-135-2 kubenswrapper[2571]: E0805 17:13:17.132151    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-whps7" podUID="a78df3c5-f92c-4dc6-a56c-a794929aa911"
Aug 05 17:13:17.132332 ip-10-0-135-2 kubenswrapper[2571]: E0805 17:13:17.132297    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-677x7" podUID="da6f032b-35a1-4c51-b115-defaf9b2a9a5"
Aug 05 17:13:17.132417 ip-10-0-135-2 kubenswrapper[2571]: E0805 17:13:17.132398    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9jxp4" podUID="f2ecc651-01ec-4fa1-a7c1-780b3a963996"
Aug 05 17:13:17.254526 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:13:17.254445    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7h5q8" event={"ID":"df7e05c7-620a-4246-bd4b-0e2c2009335b","Type":"ContainerStarted","Data":"3f5535179778e90fd493d1807d142007b834c87b708666dcca79c368e3aca511"}
Aug 05 17:13:17.256171 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:13:17.256136    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-txqn4" event={"ID":"b934c28e-a416-4ddf-8b90-0c9acdd8ce4c","Type":"ContainerStarted","Data":"e8ea27ce7accdd6e1330e6905bf547780f77c15bf1f920c1402938fb16edb59d"}
Aug 05 17:13:17.279630 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:13:17.279582    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-txqn4" podStartSLOduration=5.140790371 podStartE2EDuration="24.279561713s" totalImagesPullingTime="19.138771342s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-08-05 17:12:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-05 17:13:17.279562713 +0000 UTC m=+24.646929057" watchObservedRunningTime="2026-08-05 17:13:17.279561713 +0000 UTC m=+24.646928061"
Aug 05 17:13:18.260288 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:13:18.260251    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7h5q8" event={"ID":"df7e05c7-620a-4246-bd4b-0e2c2009335b","Type":"ContainerStarted","Data":"80e776075057e8ddc6ca13fafcb32adfb94cff1cc9d7a08fb8ea085db86d3c14"}
Aug 05 17:13:18.264123 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:13:18.264096    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-tsjgl" event={"ID":"d662bf46-8fcd-40b5-94fa-eded9b062699","Type":"ContainerStarted","Data":"b21b21ff1c0b960562597d8ea80602ab62142907a82bb27c937c35d5cef7e16c"}
Aug 05 17:13:18.264247 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:13:18.264144    2571 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness"
Aug 05 17:13:18.284715 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:13:18.284678    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7h5q8" podStartSLOduration=2.857439297 podStartE2EDuration="24.284665228s" totalImagesPullingTime="21.427225931s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-08-05 17:12:54 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-05 17:13:18.284128144 +0000 UTC m=+25.651494491" watchObservedRunningTime="2026-08-05 17:13:18.284665228 +0000 UTC m=+25.652031571"
Aug 05 17:13:19.131767 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:13:19.131741    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-677x7"
Aug 05 17:13:19.131767 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:13:19.131752    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9jxp4"
Aug 05 17:13:19.131981 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:13:19.131741    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-whps7"
Aug 05 17:13:19.131981 ip-10-0-135-2 kubenswrapper[2571]: E0805 17:13:19.131870    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-677x7" podUID="da6f032b-35a1-4c51-b115-defaf9b2a9a5"
Aug 05 17:13:19.131981 ip-10-0-135-2 kubenswrapper[2571]: E0805 17:13:19.131945    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-whps7" podUID="a78df3c5-f92c-4dc6-a56c-a794929aa911"
Aug 05 17:13:19.132131 ip-10-0-135-2 kubenswrapper[2571]: E0805 17:13:19.131991    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9jxp4" podUID="f2ecc651-01ec-4fa1-a7c1-780b3a963996"
Aug 05 17:13:21.131516 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:13:21.131323    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9jxp4"
Aug 05 17:13:21.132302 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:13:21.131331    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-677x7"
Aug 05 17:13:21.132302 ip-10-0-135-2 kubenswrapper[2571]: E0805 17:13:21.131627    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9jxp4" podUID="f2ecc651-01ec-4fa1-a7c1-780b3a963996"
Aug 05 17:13:21.132302 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:13:21.131392    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-whps7"
Aug 05 17:13:21.132302 ip-10-0-135-2 kubenswrapper[2571]: E0805 17:13:21.131661    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-677x7" podUID="da6f032b-35a1-4c51-b115-defaf9b2a9a5"
Aug 05 17:13:21.132302 ip-10-0-135-2 kubenswrapper[2571]: E0805 17:13:21.131743    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-whps7" podUID="a78df3c5-f92c-4dc6-a56c-a794929aa911"
Aug 05 17:13:21.274265 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:13:21.274232    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-tsjgl" event={"ID":"d662bf46-8fcd-40b5-94fa-eded9b062699","Type":"ContainerStarted","Data":"6635b934946ee2e25e8f518a52715e51d5023c33b20a99f4409d2d2c9389567c"}
Aug 05 17:13:21.274510 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:13:21.274485    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-tsjgl"
Aug 05 17:13:21.274649 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:13:21.274554    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-tsjgl"
Aug 05 17:13:21.275998 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:13:21.275977    2571 generic.go:356] "Generic (PLEG): container finished" podID="100f6b72-e895-453f-8647-1f615c2fb780" containerID="f2ae15687cf5f802562ba6d281bc082ca106f6f6b617f0c70dc7e820bc383a6d" exitCode=0
Aug 05 17:13:21.276115 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:13:21.276010    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-mgfr4" event={"ID":"100f6b72-e895-453f-8647-1f615c2fb780","Type":"ContainerDied","Data":"f2ae15687cf5f802562ba6d281bc082ca106f6f6b617f0c70dc7e820bc383a6d"}
Aug 05 17:13:21.290172 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:13:21.290154    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-tsjgl"
Aug 05 17:13:21.314703 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:13:21.314665    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-tsjgl" podStartSLOduration=8.071695767 podStartE2EDuration="27.314653557s" totalImagesPullingTime="19.24295779s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-08-05 17:12:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-05 17:13:21.314188378 +0000 UTC m=+28.681554721" watchObservedRunningTime="2026-08-05 17:13:21.314653557 +0000 UTC m=+28.682019900"
Aug 05 17:13:22.253853 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:13:22.253817    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-9jxp4"]
Aug 05 17:13:22.254339 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:13:22.253972    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9jxp4"
Aug 05 17:13:22.254339 ip-10-0-135-2 kubenswrapper[2571]: E0805 17:13:22.254063    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9jxp4" podUID="f2ecc651-01ec-4fa1-a7c1-780b3a963996"
Aug 05 17:13:22.256223 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:13:22.256201    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-677x7"]
Aug 05 17:13:22.256350 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:13:22.256301    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-677x7"
Aug 05 17:13:22.256443 ip-10-0-135-2 kubenswrapper[2571]: E0805 17:13:22.256416    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-677x7" podUID="da6f032b-35a1-4c51-b115-defaf9b2a9a5"
Aug 05 17:13:22.267344 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:13:22.267285    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-whps7"]
Aug 05 17:13:22.267467 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:13:22.267367    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-whps7"
Aug 05 17:13:22.267467 ip-10-0-135-2 kubenswrapper[2571]: E0805 17:13:22.267450    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-whps7" podUID="a78df3c5-f92c-4dc6-a56c-a794929aa911"
Aug 05 17:13:22.278537 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:13:22.278503    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-tsjgl"
Aug 05 17:13:22.294779 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:13:22.294758    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-tsjgl"
Aug 05 17:13:23.282203 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:13:23.282179    2571 generic.go:356] "Generic (PLEG): container finished" podID="100f6b72-e895-453f-8647-1f615c2fb780" containerID="cbc964e0d24664c23b66dce3b302503825f11acee82ae2cfb1a9d58e61ba50e7" exitCode=0
Aug 05 17:13:23.282516 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:13:23.282262    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-mgfr4" event={"ID":"100f6b72-e895-453f-8647-1f615c2fb780","Type":"ContainerDied","Data":"cbc964e0d24664c23b66dce3b302503825f11acee82ae2cfb1a9d58e61ba50e7"}
Aug 05 17:13:24.131889 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:13:24.131697    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-whps7"
Aug 05 17:13:24.132031 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:13:24.131697    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9jxp4"
Aug 05 17:13:24.132031 ip-10-0-135-2 kubenswrapper[2571]: E0805 17:13:24.131995    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-whps7" podUID="a78df3c5-f92c-4dc6-a56c-a794929aa911"
Aug 05 17:13:24.132149 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:13:24.131698    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-677x7"
Aug 05 17:13:24.132149 ip-10-0-135-2 kubenswrapper[2571]: E0805 17:13:24.132110    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9jxp4" podUID="f2ecc651-01ec-4fa1-a7c1-780b3a963996"
Aug 05 17:13:24.132149 ip-10-0-135-2 kubenswrapper[2571]: E0805 17:13:24.132138    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-677x7" podUID="da6f032b-35a1-4c51-b115-defaf9b2a9a5"
Aug 05 17:13:24.328519 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:13:24.328484    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-zmr54"
Aug 05 17:13:24.328846 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:13:24.328604    2571 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness"
Aug 05 17:13:24.329275 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:13:24.329240    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-zmr54"
Aug 05 17:13:25.288293 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:13:25.288262    2571 generic.go:356] "Generic (PLEG): container finished" podID="100f6b72-e895-453f-8647-1f615c2fb780" containerID="84cb5d94e864f6fb1625124503248154a1c507907051d500ab0239deefc3ce4d" exitCode=0
Aug 05 17:13:25.288472 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:13:25.288352    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-mgfr4" event={"ID":"100f6b72-e895-453f-8647-1f615c2fb780","Type":"ContainerDied","Data":"84cb5d94e864f6fb1625124503248154a1c507907051d500ab0239deefc3ce4d"}
Aug 05 17:13:26.131463 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:13:26.131429    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-whps7"
Aug 05 17:13:26.132009 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:13:26.131532    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9jxp4"
Aug 05 17:13:26.132009 ip-10-0-135-2 kubenswrapper[2571]: E0805 17:13:26.131544    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-whps7" podUID="a78df3c5-f92c-4dc6-a56c-a794929aa911"
Aug 05 17:13:26.132009 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:13:26.131573    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-677x7"
Aug 05 17:13:26.132009 ip-10-0-135-2 kubenswrapper[2571]: E0805 17:13:26.131671    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9jxp4" podUID="f2ecc651-01ec-4fa1-a7c1-780b3a963996"
Aug 05 17:13:26.132429 ip-10-0-135-2 kubenswrapper[2571]: E0805 17:13:26.132371    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-677x7" podUID="da6f032b-35a1-4c51-b115-defaf9b2a9a5"
Aug 05 17:13:26.832660 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:13:26.832629    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/da6f032b-35a1-4c51-b115-defaf9b2a9a5-metrics-certs\") pod \"network-metrics-daemon-677x7\" (UID: \"da6f032b-35a1-4c51-b115-defaf9b2a9a5\") " pod="openshift-multus/network-metrics-daemon-677x7"
Aug 05 17:13:26.832833 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:13:26.832671    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f2ecc651-01ec-4fa1-a7c1-780b3a963996-original-pull-secret\") pod \"global-pull-secret-syncer-9jxp4\" (UID: \"f2ecc651-01ec-4fa1-a7c1-780b3a963996\") " pod="kube-system/global-pull-secret-syncer-9jxp4"
Aug 05 17:13:26.832833 ip-10-0-135-2 kubenswrapper[2571]: E0805 17:13:26.832802    2571 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Aug 05 17:13:26.832947 ip-10-0-135-2 kubenswrapper[2571]: E0805 17:13:26.832879    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/da6f032b-35a1-4c51-b115-defaf9b2a9a5-metrics-certs podName:da6f032b-35a1-4c51-b115-defaf9b2a9a5 nodeName:}" failed. No retries permitted until 2026-08-05 17:13:58.832858434 +0000 UTC m=+66.200224761 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/da6f032b-35a1-4c51-b115-defaf9b2a9a5-metrics-certs") pod "network-metrics-daemon-677x7" (UID: "da6f032b-35a1-4c51-b115-defaf9b2a9a5") : object "openshift-multus"/"metrics-daemon-secret" not registered
Aug 05 17:13:26.832947 ip-10-0-135-2 kubenswrapper[2571]: E0805 17:13:26.832891    2571 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Aug 05 17:13:26.833037 ip-10-0-135-2 kubenswrapper[2571]: E0805 17:13:26.832947    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f2ecc651-01ec-4fa1-a7c1-780b3a963996-original-pull-secret podName:f2ecc651-01ec-4fa1-a7c1-780b3a963996 nodeName:}" failed. No retries permitted until 2026-08-05 17:13:58.83293117 +0000 UTC m=+66.200297495 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/f2ecc651-01ec-4fa1-a7c1-780b3a963996-original-pull-secret") pod "global-pull-secret-syncer-9jxp4" (UID: "f2ecc651-01ec-4fa1-a7c1-780b3a963996") : object "kube-system"/"original-pull-secret" not registered
Aug 05 17:13:26.933494 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:13:26.933458    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7xgvj\" (UniqueName: \"kubernetes.io/projected/a78df3c5-f92c-4dc6-a56c-a794929aa911-kube-api-access-7xgvj\") pod \"network-check-target-whps7\" (UID: \"a78df3c5-f92c-4dc6-a56c-a794929aa911\") " pod="openshift-network-diagnostics/network-check-target-whps7"
Aug 05 17:13:26.933652 ip-10-0-135-2 kubenswrapper[2571]: E0805 17:13:26.933626    2571 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Aug 05 17:13:26.933652 ip-10-0-135-2 kubenswrapper[2571]: E0805 17:13:26.933646    2571 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Aug 05 17:13:26.933775 ip-10-0-135-2 kubenswrapper[2571]: E0805 17:13:26.933657    2571 projected.go:196] Error preparing data for projected volume kube-api-access-7xgvj for pod openshift-network-diagnostics/network-check-target-whps7: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Aug 05 17:13:26.933775 ip-10-0-135-2 kubenswrapper[2571]: E0805 17:13:26.933717    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a78df3c5-f92c-4dc6-a56c-a794929aa911-kube-api-access-7xgvj podName:a78df3c5-f92c-4dc6-a56c-a794929aa911 nodeName:}" failed. No retries permitted until 2026-08-05 17:13:58.933698532 +0000 UTC m=+66.301064876 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-7xgvj" (UniqueName: "kubernetes.io/projected/a78df3c5-f92c-4dc6-a56c-a794929aa911-kube-api-access-7xgvj") pod "network-check-target-whps7" (UID: "a78df3c5-f92c-4dc6-a56c-a794929aa911") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Aug 05 17:13:27.937737 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:13:27.937659    2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-2.ec2.internal" event="NodeReady"
Aug 05 17:13:27.938219 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:13:27.937797    2571 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Aug 05 17:13:28.008145 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:13:28.008117    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-gn486"]
Aug 05 17:13:28.012169 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:13:28.012148    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-phtkl"]
Aug 05 17:13:28.012312 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:13:28.012264    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-gn486"
Aug 05 17:13:28.014737 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:13:28.014713    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-jxp5d\""
Aug 05 17:13:28.014952 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:13:28.014926    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Aug 05 17:13:28.015079 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:13:28.015064    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Aug 05 17:13:28.015409 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:13:28.015386    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-phtkl"
Aug 05 17:13:28.017511 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:13:28.017486    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Aug 05 17:13:28.017667 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:13:28.017649    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-7qph7\""
Aug 05 17:13:28.017914 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:13:28.017895    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Aug 05 17:13:28.018190 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:13:28.018174    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Aug 05 17:13:28.025125 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:13:28.025074    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-gn486"]
Aug 05 17:13:28.028597 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:13:28.028479    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-phtkl"]
Aug 05 17:13:28.131573 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:13:28.131541    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-677x7"
Aug 05 17:13:28.131735 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:13:28.131653    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9jxp4"
Aug 05 17:13:28.131854 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:13:28.131825    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-whps7"
Aug 05 17:13:28.134819 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:13:28.134799    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Aug 05 17:13:28.135175 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:13:28.135154    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Aug 05 17:13:28.135513 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:13:28.135497    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-z9xpb\""
Aug 05 17:13:28.135610 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:13:28.135568    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Aug 05 17:13:28.135681 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:13:28.135647    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-g8bnx\""
Aug 05 17:13:28.135878 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:13:28.135862    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Aug 05 17:13:28.142668 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:13:28.142643    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b59bced4-389d-494c-8376-d050e02710c4-cert\") pod \"ingress-canary-phtkl\" (UID: \"b59bced4-389d-494c-8376-d050e02710c4\") " pod="openshift-ingress-canary/ingress-canary-phtkl"
Aug 05 17:13:28.142763 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:13:28.142697    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hxmwd\" (UniqueName: \"kubernetes.io/projected/972f76e8-cfb8-4568-81e4-a90a0e5171d8-kube-api-access-hxmwd\") pod \"dns-default-gn486\" (UID: \"972f76e8-cfb8-4568-81e4-a90a0e5171d8\") " pod="openshift-dns/dns-default-gn486"
Aug 05 17:13:28.142763 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:13:28.142752    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-76g9p\" (UniqueName: \"kubernetes.io/projected/b59bced4-389d-494c-8376-d050e02710c4-kube-api-access-76g9p\") pod \"ingress-canary-phtkl\" (UID: \"b59bced4-389d-494c-8376-d050e02710c4\") " pod="openshift-ingress-canary/ingress-canary-phtkl"
Aug 05 17:13:28.142899 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:13:28.142874    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/972f76e8-cfb8-4568-81e4-a90a0e5171d8-config-volume\") pod \"dns-default-gn486\" (UID: \"972f76e8-cfb8-4568-81e4-a90a0e5171d8\") " pod="openshift-dns/dns-default-gn486"
Aug 05 17:13:28.142954 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:13:28.142909    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/972f76e8-cfb8-4568-81e4-a90a0e5171d8-metrics-tls\") pod \"dns-default-gn486\" (UID: \"972f76e8-cfb8-4568-81e4-a90a0e5171d8\") " pod="openshift-dns/dns-default-gn486"
Aug 05 17:13:28.143004 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:13:28.142952    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/972f76e8-cfb8-4568-81e4-a90a0e5171d8-tmp-dir\") pod \"dns-default-gn486\" (UID: \"972f76e8-cfb8-4568-81e4-a90a0e5171d8\") " pod="openshift-dns/dns-default-gn486"
Aug 05 17:13:28.243450 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:13:28.243412    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/972f76e8-cfb8-4568-81e4-a90a0e5171d8-config-volume\") pod \"dns-default-gn486\" (UID: \"972f76e8-cfb8-4568-81e4-a90a0e5171d8\") " pod="openshift-dns/dns-default-gn486"
Aug 05 17:13:28.243450 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:13:28.243448    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/972f76e8-cfb8-4568-81e4-a90a0e5171d8-metrics-tls\") pod \"dns-default-gn486\" (UID: \"972f76e8-cfb8-4568-81e4-a90a0e5171d8\") " pod="openshift-dns/dns-default-gn486"
Aug 05 17:13:28.243668 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:13:28.243485    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/972f76e8-cfb8-4568-81e4-a90a0e5171d8-tmp-dir\") pod \"dns-default-gn486\" (UID: \"972f76e8-cfb8-4568-81e4-a90a0e5171d8\") " pod="openshift-dns/dns-default-gn486"
Aug 05 17:13:28.243668 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:13:28.243507    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b59bced4-389d-494c-8376-d050e02710c4-cert\") pod \"ingress-canary-phtkl\" (UID: \"b59bced4-389d-494c-8376-d050e02710c4\") " pod="openshift-ingress-canary/ingress-canary-phtkl"
Aug 05 17:13:28.243668 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:13:28.243542    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hxmwd\" (UniqueName: \"kubernetes.io/projected/972f76e8-cfb8-4568-81e4-a90a0e5171d8-kube-api-access-hxmwd\") pod \"dns-default-gn486\" (UID: \"972f76e8-cfb8-4568-81e4-a90a0e5171d8\") " pod="openshift-dns/dns-default-gn486"
Aug 05 17:13:28.243668 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:13:28.243570    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-76g9p\" (UniqueName: \"kubernetes.io/projected/b59bced4-389d-494c-8376-d050e02710c4-kube-api-access-76g9p\") pod \"ingress-canary-phtkl\" (UID: \"b59bced4-389d-494c-8376-d050e02710c4\") " pod="openshift-ingress-canary/ingress-canary-phtkl"
Aug 05 17:13:28.243668 ip-10-0-135-2 kubenswrapper[2571]: E0805 17:13:28.243599    2571 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Aug 05 17:13:28.243668 ip-10-0-135-2 kubenswrapper[2571]: E0805 17:13:28.243669    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/972f76e8-cfb8-4568-81e4-a90a0e5171d8-metrics-tls podName:972f76e8-cfb8-4568-81e4-a90a0e5171d8 nodeName:}" failed. No retries permitted until 2026-08-05 17:13:28.743648677 +0000 UTC m=+36.111015014 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/972f76e8-cfb8-4568-81e4-a90a0e5171d8-metrics-tls") pod "dns-default-gn486" (UID: "972f76e8-cfb8-4568-81e4-a90a0e5171d8") : secret "dns-default-metrics-tls" not found
Aug 05 17:13:28.244053 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:13:28.244031    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/972f76e8-cfb8-4568-81e4-a90a0e5171d8-tmp-dir\") pod \"dns-default-gn486\" (UID: \"972f76e8-cfb8-4568-81e4-a90a0e5171d8\") " pod="openshift-dns/dns-default-gn486"
Aug 05 17:13:28.244122 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:13:28.244074    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/972f76e8-cfb8-4568-81e4-a90a0e5171d8-config-volume\") pod \"dns-default-gn486\" (UID: \"972f76e8-cfb8-4568-81e4-a90a0e5171d8\") " pod="openshift-dns/dns-default-gn486"
Aug 05 17:13:28.248099 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:13:28.248074    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b59bced4-389d-494c-8376-d050e02710c4-cert\") pod \"ingress-canary-phtkl\" (UID: \"b59bced4-389d-494c-8376-d050e02710c4\") " pod="openshift-ingress-canary/ingress-canary-phtkl"
Aug 05 17:13:28.254604 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:13:28.254579    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-76g9p\" (UniqueName: \"kubernetes.io/projected/b59bced4-389d-494c-8376-d050e02710c4-kube-api-access-76g9p\") pod \"ingress-canary-phtkl\" (UID: \"b59bced4-389d-494c-8376-d050e02710c4\") " pod="openshift-ingress-canary/ingress-canary-phtkl"
Aug 05 17:13:28.255060 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:13:28.255035    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hxmwd\" (UniqueName: \"kubernetes.io/projected/972f76e8-cfb8-4568-81e4-a90a0e5171d8-kube-api-access-hxmwd\") pod \"dns-default-gn486\" (UID: \"972f76e8-cfb8-4568-81e4-a90a0e5171d8\") " pod="openshift-dns/dns-default-gn486"
Aug 05 17:13:28.330860 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:13:28.330750    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-phtkl"
Aug 05 17:13:28.746959 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:13:28.746926    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/972f76e8-cfb8-4568-81e4-a90a0e5171d8-metrics-tls\") pod \"dns-default-gn486\" (UID: \"972f76e8-cfb8-4568-81e4-a90a0e5171d8\") " pod="openshift-dns/dns-default-gn486"
Aug 05 17:13:28.747122 ip-10-0-135-2 kubenswrapper[2571]: E0805 17:13:28.747093    2571 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Aug 05 17:13:28.747184 ip-10-0-135-2 kubenswrapper[2571]: E0805 17:13:28.747165    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/972f76e8-cfb8-4568-81e4-a90a0e5171d8-metrics-tls podName:972f76e8-cfb8-4568-81e4-a90a0e5171d8 nodeName:}" failed. No retries permitted until 2026-08-05 17:13:29.747147064 +0000 UTC m=+37.114513396 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/972f76e8-cfb8-4568-81e4-a90a0e5171d8-metrics-tls") pod "dns-default-gn486" (UID: "972f76e8-cfb8-4568-81e4-a90a0e5171d8") : secret "dns-default-metrics-tls" not found
Aug 05 17:13:29.280369 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:13:29.280338    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-phtkl"]
Aug 05 17:13:29.283402 ip-10-0-135-2 kubenswrapper[2571]: W0805 17:13:29.283368    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb59bced4_389d_494c_8376_d050e02710c4.slice/crio-ab9e95dde3ab5626f11cb6e4c6c98e3854a08a77617816594606af06b66a3ff3 WatchSource:0}: Error finding container ab9e95dde3ab5626f11cb6e4c6c98e3854a08a77617816594606af06b66a3ff3: Status 404 returned error can't find the container with id ab9e95dde3ab5626f11cb6e4c6c98e3854a08a77617816594606af06b66a3ff3
Aug 05 17:13:29.298172 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:13:29.298147    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-phtkl" event={"ID":"b59bced4-389d-494c-8376-d050e02710c4","Type":"ContainerStarted","Data":"ab9e95dde3ab5626f11cb6e4c6c98e3854a08a77617816594606af06b66a3ff3"}
Aug 05 17:13:29.753857 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:13:29.753821    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/972f76e8-cfb8-4568-81e4-a90a0e5171d8-metrics-tls\") pod \"dns-default-gn486\" (UID: \"972f76e8-cfb8-4568-81e4-a90a0e5171d8\") " pod="openshift-dns/dns-default-gn486"
Aug 05 17:13:29.754023 ip-10-0-135-2 kubenswrapper[2571]: E0805 17:13:29.753972    2571 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Aug 05 17:13:29.754075 ip-10-0-135-2 kubenswrapper[2571]: E0805 17:13:29.754053    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/972f76e8-cfb8-4568-81e4-a90a0e5171d8-metrics-tls podName:972f76e8-cfb8-4568-81e4-a90a0e5171d8 nodeName:}" failed. No retries permitted until 2026-08-05 17:13:31.754034203 +0000 UTC m=+39.121400542 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/972f76e8-cfb8-4568-81e4-a90a0e5171d8-metrics-tls") pod "dns-default-gn486" (UID: "972f76e8-cfb8-4568-81e4-a90a0e5171d8") : secret "dns-default-metrics-tls" not found
Aug 05 17:13:30.303659 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:13:30.303622    2571 generic.go:356] "Generic (PLEG): container finished" podID="100f6b72-e895-453f-8647-1f615c2fb780" containerID="1b8dd5f72951e9180dd50fd22ed1c605bb4ee5cf7dbd02a5ebd7e65bfd2ed705" exitCode=0
Aug 05 17:13:30.304078 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:13:30.303671    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-mgfr4" event={"ID":"100f6b72-e895-453f-8647-1f615c2fb780","Type":"ContainerDied","Data":"1b8dd5f72951e9180dd50fd22ed1c605bb4ee5cf7dbd02a5ebd7e65bfd2ed705"}
Aug 05 17:13:31.310392 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:13:31.310341    2571 generic.go:356] "Generic (PLEG): container finished" podID="100f6b72-e895-453f-8647-1f615c2fb780" containerID="6acbaeb87b032ff7bc5ac5caac7e0cde4d912eee4de86acf4522874483d3f295" exitCode=0
Aug 05 17:13:31.310782 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:13:31.310417    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-mgfr4" event={"ID":"100f6b72-e895-453f-8647-1f615c2fb780","Type":"ContainerDied","Data":"6acbaeb87b032ff7bc5ac5caac7e0cde4d912eee4de86acf4522874483d3f295"}
Aug 05 17:13:31.770307 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:13:31.770284    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/972f76e8-cfb8-4568-81e4-a90a0e5171d8-metrics-tls\") pod \"dns-default-gn486\" (UID: \"972f76e8-cfb8-4568-81e4-a90a0e5171d8\") " pod="openshift-dns/dns-default-gn486"
Aug 05 17:13:31.770442 ip-10-0-135-2 kubenswrapper[2571]: E0805 17:13:31.770429    2571 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Aug 05 17:13:31.770489 ip-10-0-135-2 kubenswrapper[2571]: E0805 17:13:31.770480    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/972f76e8-cfb8-4568-81e4-a90a0e5171d8-metrics-tls podName:972f76e8-cfb8-4568-81e4-a90a0e5171d8 nodeName:}" failed. No retries permitted until 2026-08-05 17:13:35.770465513 +0000 UTC m=+43.137831840 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/972f76e8-cfb8-4568-81e4-a90a0e5171d8-metrics-tls") pod "dns-default-gn486" (UID: "972f76e8-cfb8-4568-81e4-a90a0e5171d8") : secret "dns-default-metrics-tls" not found
Aug 05 17:13:32.314646 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:13:32.314578    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-phtkl" event={"ID":"b59bced4-389d-494c-8376-d050e02710c4","Type":"ContainerStarted","Data":"89533ea5080bf37025c55f87bc0d37d0ed2a81696f9b96808d96eca27158d7d0"}
Aug 05 17:13:32.317800 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:13:32.317777    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-mgfr4" event={"ID":"100f6b72-e895-453f-8647-1f615c2fb780","Type":"ContainerStarted","Data":"6b42f3ce570ef169703f57b2f248f8b2f39e096cdb46704ef8595af337d13d47"}
Aug 05 17:13:32.333126 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:13:32.333085    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-phtkl" podStartSLOduration=2.596166529 podStartE2EDuration="5.333073395s" totalImagesPullingTime="2.736906866s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-05 17:13:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-05 17:13:32.332534123 +0000 UTC m=+39.699900465" watchObservedRunningTime="2026-08-05 17:13:32.333073395 +0000 UTC m=+39.700439763"
Aug 05 17:13:32.359591 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:13:32.359536    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-mgfr4" podStartSLOduration=9.246668572 podStartE2EDuration="39.359516388s" totalImagesPullingTime="30.112847816s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-08-05 17:12:53 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-05 17:13:32.358283286 +0000 UTC m=+39.725649661" watchObservedRunningTime="2026-08-05 17:13:32.359516388 +0000 UTC m=+39.726882732"
Aug 05 17:13:35.298572 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:13:35.298536    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7b74c45fdd-9tcbw"]
Aug 05 17:13:35.330333 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:13:35.330309    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7b74c45fdd-9tcbw"]
Aug 05 17:13:35.330478 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:13:35.330416    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7b74c45fdd-9tcbw"
Aug 05 17:13:35.334464 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:13:35.334441    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-open-cluster-management.io-proxy-agent-signer-client-cert\""
Aug 05 17:13:35.334584 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:13:35.334568    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\""
Aug 05 17:13:35.334816 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:13:35.334803    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-service-proxy-server-certificates\""
Aug 05 17:13:35.334859 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:13:35.334830    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\""
Aug 05 17:13:35.335567 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:13:35.335532    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\""
Aug 05 17:13:35.335567 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:13:35.335554    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-hub-kubeconfig\""
Aug 05 17:13:35.335727 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:13:35.335622    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-ca\""
Aug 05 17:13:35.494557 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:13:35.494526    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/83dbc4d1-49f4-4f07-97d2-80443a2c873f-hub\") pod \"cluster-proxy-proxy-agent-7b74c45fdd-9tcbw\" (UID: \"83dbc4d1-49f4-4f07-97d2-80443a2c873f\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7b74c45fdd-9tcbw"
Aug 05 17:13:35.494557 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:13:35.494562    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/83dbc4d1-49f4-4f07-97d2-80443a2c873f-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-7b74c45fdd-9tcbw\" (UID: \"83dbc4d1-49f4-4f07-97d2-80443a2c873f\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7b74c45fdd-9tcbw"
Aug 05 17:13:35.494788 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:13:35.494606    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/83dbc4d1-49f4-4f07-97d2-80443a2c873f-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-7b74c45fdd-9tcbw\" (UID: \"83dbc4d1-49f4-4f07-97d2-80443a2c873f\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7b74c45fdd-9tcbw"
Aug 05 17:13:35.494788 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:13:35.494623    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5z6zx\" (UniqueName: \"kubernetes.io/projected/83dbc4d1-49f4-4f07-97d2-80443a2c873f-kube-api-access-5z6zx\") pod \"cluster-proxy-proxy-agent-7b74c45fdd-9tcbw\" (UID: \"83dbc4d1-49f4-4f07-97d2-80443a2c873f\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7b74c45fdd-9tcbw"
Aug 05 17:13:35.494788 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:13:35.494649    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/83dbc4d1-49f4-4f07-97d2-80443a2c873f-ca\") pod \"cluster-proxy-proxy-agent-7b74c45fdd-9tcbw\" (UID: \"83dbc4d1-49f4-4f07-97d2-80443a2c873f\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7b74c45fdd-9tcbw"
Aug 05 17:13:35.494788 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:13:35.494681    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/83dbc4d1-49f4-4f07-97d2-80443a2c873f-additional-service-ca\") pod \"cluster-proxy-proxy-agent-7b74c45fdd-9tcbw\" (UID: \"83dbc4d1-49f4-4f07-97d2-80443a2c873f\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7b74c45fdd-9tcbw"
Aug 05 17:13:35.595481 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:13:35.595409    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/83dbc4d1-49f4-4f07-97d2-80443a2c873f-hub\") pod \"cluster-proxy-proxy-agent-7b74c45fdd-9tcbw\" (UID: \"83dbc4d1-49f4-4f07-97d2-80443a2c873f\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7b74c45fdd-9tcbw"
Aug 05 17:13:35.595481 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:13:35.595443    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/83dbc4d1-49f4-4f07-97d2-80443a2c873f-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-7b74c45fdd-9tcbw\" (UID: \"83dbc4d1-49f4-4f07-97d2-80443a2c873f\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7b74c45fdd-9tcbw"
Aug 05 17:13:35.595680 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:13:35.595581    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/83dbc4d1-49f4-4f07-97d2-80443a2c873f-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-7b74c45fdd-9tcbw\" (UID: \"83dbc4d1-49f4-4f07-97d2-80443a2c873f\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7b74c45fdd-9tcbw"
Aug 05 17:13:35.595680 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:13:35.595610    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5z6zx\" (UniqueName: \"kubernetes.io/projected/83dbc4d1-49f4-4f07-97d2-80443a2c873f-kube-api-access-5z6zx\") pod \"cluster-proxy-proxy-agent-7b74c45fdd-9tcbw\" (UID: \"83dbc4d1-49f4-4f07-97d2-80443a2c873f\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7b74c45fdd-9tcbw"
Aug 05 17:13:35.595680 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:13:35.595639    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/83dbc4d1-49f4-4f07-97d2-80443a2c873f-ca\") pod \"cluster-proxy-proxy-agent-7b74c45fdd-9tcbw\" (UID: \"83dbc4d1-49f4-4f07-97d2-80443a2c873f\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7b74c45fdd-9tcbw"
Aug 05 17:13:35.595680 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:13:35.595658    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/83dbc4d1-49f4-4f07-97d2-80443a2c873f-additional-service-ca\") pod \"cluster-proxy-proxy-agent-7b74c45fdd-9tcbw\" (UID: \"83dbc4d1-49f4-4f07-97d2-80443a2c873f\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7b74c45fdd-9tcbw"
Aug 05 17:13:35.596296 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:13:35.596275    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/83dbc4d1-49f4-4f07-97d2-80443a2c873f-additional-service-ca\") pod \"cluster-proxy-proxy-agent-7b74c45fdd-9tcbw\" (UID: \"83dbc4d1-49f4-4f07-97d2-80443a2c873f\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7b74c45fdd-9tcbw"
Aug 05 17:13:35.599506 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:13:35.599481    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca\" (UniqueName: \"kubernetes.io/secret/83dbc4d1-49f4-4f07-97d2-80443a2c873f-ca\") pod \"cluster-proxy-proxy-agent-7b74c45fdd-9tcbw\" (UID: \"83dbc4d1-49f4-4f07-97d2-80443a2c873f\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7b74c45fdd-9tcbw"
Aug 05 17:13:35.599600 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:13:35.599505    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub\" (UniqueName: \"kubernetes.io/secret/83dbc4d1-49f4-4f07-97d2-80443a2c873f-hub\") pod \"cluster-proxy-proxy-agent-7b74c45fdd-9tcbw\" (UID: \"83dbc4d1-49f4-4f07-97d2-80443a2c873f\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7b74c45fdd-9tcbw"
Aug 05 17:13:35.599600 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:13:35.599524    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/83dbc4d1-49f4-4f07-97d2-80443a2c873f-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-7b74c45fdd-9tcbw\" (UID: \"83dbc4d1-49f4-4f07-97d2-80443a2c873f\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7b74c45fdd-9tcbw"
Aug 05 17:13:35.599712 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:13:35.599696    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/83dbc4d1-49f4-4f07-97d2-80443a2c873f-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-7b74c45fdd-9tcbw\" (UID: \"83dbc4d1-49f4-4f07-97d2-80443a2c873f\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7b74c45fdd-9tcbw"
Aug 05 17:13:35.605094 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:13:35.605073    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5z6zx\" (UniqueName: \"kubernetes.io/projected/83dbc4d1-49f4-4f07-97d2-80443a2c873f-kube-api-access-5z6zx\") pod \"cluster-proxy-proxy-agent-7b74c45fdd-9tcbw\" (UID: \"83dbc4d1-49f4-4f07-97d2-80443a2c873f\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7b74c45fdd-9tcbw"
Aug 05 17:13:35.655173 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:13:35.655148    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7b74c45fdd-9tcbw"
Aug 05 17:13:35.796938 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:13:35.796903    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/972f76e8-cfb8-4568-81e4-a90a0e5171d8-metrics-tls\") pod \"dns-default-gn486\" (UID: \"972f76e8-cfb8-4568-81e4-a90a0e5171d8\") " pod="openshift-dns/dns-default-gn486"
Aug 05 17:13:35.797065 ip-10-0-135-2 kubenswrapper[2571]: E0805 17:13:35.797040    2571 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Aug 05 17:13:35.797108 ip-10-0-135-2 kubenswrapper[2571]: E0805 17:13:35.797098    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/972f76e8-cfb8-4568-81e4-a90a0e5171d8-metrics-tls podName:972f76e8-cfb8-4568-81e4-a90a0e5171d8 nodeName:}" failed. No retries permitted until 2026-08-05 17:13:43.797083393 +0000 UTC m=+51.164449722 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/972f76e8-cfb8-4568-81e4-a90a0e5171d8-metrics-tls") pod "dns-default-gn486" (UID: "972f76e8-cfb8-4568-81e4-a90a0e5171d8") : secret "dns-default-metrics-tls" not found
Aug 05 17:13:35.806750 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:13:35.806723    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7b74c45fdd-9tcbw"]
Aug 05 17:13:35.809843 ip-10-0-135-2 kubenswrapper[2571]: W0805 17:13:35.809813    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod83dbc4d1_49f4_4f07_97d2_80443a2c873f.slice/crio-5e29ceb30a46559daade63c572873a5dc84c2562e314748fe2dea5f97430d7d4 WatchSource:0}: Error finding container 5e29ceb30a46559daade63c572873a5dc84c2562e314748fe2dea5f97430d7d4: Status 404 returned error can't find the container with id 5e29ceb30a46559daade63c572873a5dc84c2562e314748fe2dea5f97430d7d4
Aug 05 17:13:36.327542 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:13:36.327512    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7b74c45fdd-9tcbw" event={"ID":"83dbc4d1-49f4-4f07-97d2-80443a2c873f","Type":"ContainerStarted","Data":"5e29ceb30a46559daade63c572873a5dc84c2562e314748fe2dea5f97430d7d4"}
Aug 05 17:13:43.850796 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:13:43.850750    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/972f76e8-cfb8-4568-81e4-a90a0e5171d8-metrics-tls\") pod \"dns-default-gn486\" (UID: \"972f76e8-cfb8-4568-81e4-a90a0e5171d8\") " pod="openshift-dns/dns-default-gn486"
Aug 05 17:13:43.851347 ip-10-0-135-2 kubenswrapper[2571]: E0805 17:13:43.850929    2571 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Aug 05 17:13:43.851347 ip-10-0-135-2 kubenswrapper[2571]: E0805 17:13:43.851005    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/972f76e8-cfb8-4568-81e4-a90a0e5171d8-metrics-tls podName:972f76e8-cfb8-4568-81e4-a90a0e5171d8 nodeName:}" failed. No retries permitted until 2026-08-05 17:13:59.850986409 +0000 UTC m=+67.218352750 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/972f76e8-cfb8-4568-81e4-a90a0e5171d8-metrics-tls") pod "dns-default-gn486" (UID: "972f76e8-cfb8-4568-81e4-a90a0e5171d8") : secret "dns-default-metrics-tls" not found
Aug 05 17:13:45.351644 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:13:45.351608    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7b74c45fdd-9tcbw" event={"ID":"83dbc4d1-49f4-4f07-97d2-80443a2c873f","Type":"ContainerStarted","Data":"37629569b3e236ae5d3c984c209348a9d6919874bcf4f0332121ca5ebaabee27"}
Aug 05 17:13:45.351644 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:13:45.351651    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7b74c45fdd-9tcbw" event={"ID":"83dbc4d1-49f4-4f07-97d2-80443a2c873f","Type":"ContainerStarted","Data":"47aa922cbe50a1332038bd73aea31e8d00cf5d6682b6ec65477b376245aa9c62"}
Aug 05 17:13:45.352029 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:13:45.351664    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7b74c45fdd-9tcbw" event={"ID":"83dbc4d1-49f4-4f07-97d2-80443a2c873f","Type":"ContainerStarted","Data":"042e7ac4a976dde59e3ee4e8f3ae0f9fb231d54c336681a757df759bc3d528fe"}
Aug 05 17:13:54.300800 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:13:54.300760    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-tsjgl"
Aug 05 17:13:54.334487 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:13:54.334436    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7b74c45fdd-9tcbw" podStartSLOduration=10.44226667 podStartE2EDuration="19.334422602s" totalImagesPullingTime="8.892155932s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-05 17:13:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-05 17:13:45.377710036 +0000 UTC m=+52.745076378" watchObservedRunningTime="2026-08-05 17:13:54.334422602 +0000 UTC m=+61.701788945"
Aug 05 17:13:58.849246 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:13:58.849207    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f2ecc651-01ec-4fa1-a7c1-780b3a963996-original-pull-secret\") pod \"global-pull-secret-syncer-9jxp4\" (UID: \"f2ecc651-01ec-4fa1-a7c1-780b3a963996\") " pod="kube-system/global-pull-secret-syncer-9jxp4"
Aug 05 17:13:58.849734 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:13:58.849280    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/da6f032b-35a1-4c51-b115-defaf9b2a9a5-metrics-certs\") pod \"network-metrics-daemon-677x7\" (UID: \"da6f032b-35a1-4c51-b115-defaf9b2a9a5\") " pod="openshift-multus/network-metrics-daemon-677x7"
Aug 05 17:13:58.853822 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:13:58.853797    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Aug 05 17:13:58.853941 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:13:58.853798    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Aug 05 17:13:58.860045 ip-10-0-135-2 kubenswrapper[2571]: E0805 17:13:58.860027    2571 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Aug 05 17:13:58.860107 ip-10-0-135-2 kubenswrapper[2571]: E0805 17:13:58.860093    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/da6f032b-35a1-4c51-b115-defaf9b2a9a5-metrics-certs podName:da6f032b-35a1-4c51-b115-defaf9b2a9a5 nodeName:}" failed. No retries permitted until 2026-08-05 17:15:02.860076764 +0000 UTC m=+130.227443086 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/da6f032b-35a1-4c51-b115-defaf9b2a9a5-metrics-certs") pod "network-metrics-daemon-677x7" (UID: "da6f032b-35a1-4c51-b115-defaf9b2a9a5") : secret "metrics-daemon-secret" not found
Aug 05 17:13:58.862279 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:13:58.862251    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f2ecc651-01ec-4fa1-a7c1-780b3a963996-original-pull-secret\") pod \"global-pull-secret-syncer-9jxp4\" (UID: \"f2ecc651-01ec-4fa1-a7c1-780b3a963996\") " pod="kube-system/global-pull-secret-syncer-9jxp4"
Aug 05 17:13:58.950422 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:13:58.950394    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7xgvj\" (UniqueName: \"kubernetes.io/projected/a78df3c5-f92c-4dc6-a56c-a794929aa911-kube-api-access-7xgvj\") pod \"network-check-target-whps7\" (UID: \"a78df3c5-f92c-4dc6-a56c-a794929aa911\") " pod="openshift-network-diagnostics/network-check-target-whps7"
Aug 05 17:13:58.953966 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:13:58.953952    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Aug 05 17:13:58.964512 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:13:58.964496    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Aug 05 17:13:58.973757 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:13:58.973739    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7xgvj\" (UniqueName: \"kubernetes.io/projected/a78df3c5-f92c-4dc6-a56c-a794929aa911-kube-api-access-7xgvj\") pod \"network-check-target-whps7\" (UID: \"a78df3c5-f92c-4dc6-a56c-a794929aa911\") " pod="openshift-network-diagnostics/network-check-target-whps7"
Aug 05 17:13:59.043227 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:13:59.043207    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9jxp4"
Aug 05 17:13:59.062538 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:13:59.062520    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-g8bnx\""
Aug 05 17:13:59.068632 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:13:59.068614    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-whps7"
Aug 05 17:13:59.198412 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:13:59.198358    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-9jxp4"]
Aug 05 17:13:59.201800 ip-10-0-135-2 kubenswrapper[2571]: W0805 17:13:59.201770    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf2ecc651_01ec_4fa1_a7c1_780b3a963996.slice/crio-04202c6482e3da6c86d106ca88dc61e00156b7c7c8defe8e1de26f88c202ebcb WatchSource:0}: Error finding container 04202c6482e3da6c86d106ca88dc61e00156b7c7c8defe8e1de26f88c202ebcb: Status 404 returned error can't find the container with id 04202c6482e3da6c86d106ca88dc61e00156b7c7c8defe8e1de26f88c202ebcb
Aug 05 17:13:59.212292 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:13:59.212271    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-whps7"]
Aug 05 17:13:59.215184 ip-10-0-135-2 kubenswrapper[2571]: W0805 17:13:59.215164    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda78df3c5_f92c_4dc6_a56c_a794929aa911.slice/crio-6c6914382ee837b48744c24161411e44ab05e98af6bc094d27a6af42c8bb392a WatchSource:0}: Error finding container 6c6914382ee837b48744c24161411e44ab05e98af6bc094d27a6af42c8bb392a: Status 404 returned error can't find the container with id 6c6914382ee837b48744c24161411e44ab05e98af6bc094d27a6af42c8bb392a
Aug 05 17:13:59.386607 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:13:59.386529    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-9jxp4" event={"ID":"f2ecc651-01ec-4fa1-a7c1-780b3a963996","Type":"ContainerStarted","Data":"04202c6482e3da6c86d106ca88dc61e00156b7c7c8defe8e1de26f88c202ebcb"}
Aug 05 17:13:59.387662 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:13:59.387636    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-whps7" event={"ID":"a78df3c5-f92c-4dc6-a56c-a794929aa911","Type":"ContainerStarted","Data":"6c6914382ee837b48744c24161411e44ab05e98af6bc094d27a6af42c8bb392a"}
Aug 05 17:13:59.856863 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:13:59.856830    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/972f76e8-cfb8-4568-81e4-a90a0e5171d8-metrics-tls\") pod \"dns-default-gn486\" (UID: \"972f76e8-cfb8-4568-81e4-a90a0e5171d8\") " pod="openshift-dns/dns-default-gn486"
Aug 05 17:13:59.857343 ip-10-0-135-2 kubenswrapper[2571]: E0805 17:13:59.856957    2571 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Aug 05 17:13:59.857343 ip-10-0-135-2 kubenswrapper[2571]: E0805 17:13:59.857020    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/972f76e8-cfb8-4568-81e4-a90a0e5171d8-metrics-tls podName:972f76e8-cfb8-4568-81e4-a90a0e5171d8 nodeName:}" failed. No retries permitted until 2026-08-05 17:14:31.85700696 +0000 UTC m=+99.224373281 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/972f76e8-cfb8-4568-81e4-a90a0e5171d8-metrics-tls") pod "dns-default-gn486" (UID: "972f76e8-cfb8-4568-81e4-a90a0e5171d8") : secret "dns-default-metrics-tls" not found
Aug 05 17:14:04.404153 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:14:04.404059    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-9jxp4" event={"ID":"f2ecc651-01ec-4fa1-a7c1-780b3a963996","Type":"ContainerStarted","Data":"d738534b6dbb8db5e5ef514a40781833e34cf1aa5fc214f030702232790e8297"}
Aug 05 17:14:04.405720 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:14:04.405696    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-whps7" event={"ID":"a78df3c5-f92c-4dc6-a56c-a794929aa911","Type":"ContainerStarted","Data":"22c0e6db8b7c404f417b9a640002b8a4ec19dd5b1d2e2febe82c086589abaa70"}
Aug 05 17:14:04.405829 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:14:04.405757    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-whps7"
Aug 05 17:14:04.420938 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:14:04.420900    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-9jxp4" podStartSLOduration=65.592280345 podStartE2EDuration="1m10.420886501s" totalImagesPullingTime="4.828606156s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-08-05 17:12:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-05 17:14:04.420598784 +0000 UTC m=+71.787965128" watchObservedRunningTime="2026-08-05 17:14:04.420886501 +0000 UTC m=+71.788252844"
Aug 05 17:14:04.436511 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:14:04.436475    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-whps7" podStartSLOduration=65.62550849 podStartE2EDuration="1m10.4364631s" totalImagesPullingTime="4.81095461s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-05 17:12:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-05 17:14:04.436010888 +0000 UTC m=+71.803377230" watchObservedRunningTime="2026-08-05 17:14:04.4364631 +0000 UTC m=+71.803829442"
Aug 05 17:14:31.866122 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:14:31.866091    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/972f76e8-cfb8-4568-81e4-a90a0e5171d8-metrics-tls\") pod \"dns-default-gn486\" (UID: \"972f76e8-cfb8-4568-81e4-a90a0e5171d8\") " pod="openshift-dns/dns-default-gn486"
Aug 05 17:14:31.866473 ip-10-0-135-2 kubenswrapper[2571]: E0805 17:14:31.866207    2571 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Aug 05 17:14:31.866473 ip-10-0-135-2 kubenswrapper[2571]: E0805 17:14:31.866269    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/972f76e8-cfb8-4568-81e4-a90a0e5171d8-metrics-tls podName:972f76e8-cfb8-4568-81e4-a90a0e5171d8 nodeName:}" failed. No retries permitted until 2026-08-05 17:15:35.866255912 +0000 UTC m=+163.233622234 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/972f76e8-cfb8-4568-81e4-a90a0e5171d8-metrics-tls") pod "dns-default-gn486" (UID: "972f76e8-cfb8-4568-81e4-a90a0e5171d8") : secret "dns-default-metrics-tls" not found
Aug 05 17:14:35.410693 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:14:35.410661    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-whps7"
Aug 05 17:14:42.826418 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:14:42.826370    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-547968ddbb-vtdlf"]
Aug 05 17:14:42.831770 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:14:42.831755    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-547968ddbb-vtdlf"
Aug 05 17:14:42.834721 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:14:42.834700    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Aug 05 17:14:42.835798 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:14:42.835780    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Aug 05 17:14:42.835798 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:14:42.835784    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-g4tv2\""
Aug 05 17:14:42.836607 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:14:42.836586    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Aug 05 17:14:42.844045 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:14:42.844029    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Aug 05 17:14:42.853451 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:14:42.853431    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-547968ddbb-vtdlf"]
Aug 05 17:14:42.935178 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:14:42.935147    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/af418777-9235-45af-bdeb-a837dcb58a14-image-registry-private-configuration\") pod \"image-registry-547968ddbb-vtdlf\" (UID: \"af418777-9235-45af-bdeb-a837dcb58a14\") " pod="openshift-image-registry/image-registry-547968ddbb-vtdlf"
Aug 05 17:14:42.935178 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:14:42.935178    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/af418777-9235-45af-bdeb-a837dcb58a14-trusted-ca\") pod \"image-registry-547968ddbb-vtdlf\" (UID: \"af418777-9235-45af-bdeb-a837dcb58a14\") " pod="openshift-image-registry/image-registry-547968ddbb-vtdlf"
Aug 05 17:14:42.935354 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:14:42.935193    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/af418777-9235-45af-bdeb-a837dcb58a14-installation-pull-secrets\") pod \"image-registry-547968ddbb-vtdlf\" (UID: \"af418777-9235-45af-bdeb-a837dcb58a14\") " pod="openshift-image-registry/image-registry-547968ddbb-vtdlf"
Aug 05 17:14:42.935354 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:14:42.935212    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/af418777-9235-45af-bdeb-a837dcb58a14-registry-certificates\") pod \"image-registry-547968ddbb-vtdlf\" (UID: \"af418777-9235-45af-bdeb-a837dcb58a14\") " pod="openshift-image-registry/image-registry-547968ddbb-vtdlf"
Aug 05 17:14:42.935354 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:14:42.935231    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hqtn8\" (UniqueName: \"kubernetes.io/projected/af418777-9235-45af-bdeb-a837dcb58a14-kube-api-access-hqtn8\") pod \"image-registry-547968ddbb-vtdlf\" (UID: \"af418777-9235-45af-bdeb-a837dcb58a14\") " pod="openshift-image-registry/image-registry-547968ddbb-vtdlf"
Aug 05 17:14:42.935354 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:14:42.935315    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/af418777-9235-45af-bdeb-a837dcb58a14-registry-tls\") pod \"image-registry-547968ddbb-vtdlf\" (UID: \"af418777-9235-45af-bdeb-a837dcb58a14\") " pod="openshift-image-registry/image-registry-547968ddbb-vtdlf"
Aug 05 17:14:42.935354 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:14:42.935343    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/af418777-9235-45af-bdeb-a837dcb58a14-ca-trust-extracted\") pod \"image-registry-547968ddbb-vtdlf\" (UID: \"af418777-9235-45af-bdeb-a837dcb58a14\") " pod="openshift-image-registry/image-registry-547968ddbb-vtdlf"
Aug 05 17:14:42.935526 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:14:42.935361    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/af418777-9235-45af-bdeb-a837dcb58a14-bound-sa-token\") pod \"image-registry-547968ddbb-vtdlf\" (UID: \"af418777-9235-45af-bdeb-a837dcb58a14\") " pod="openshift-image-registry/image-registry-547968ddbb-vtdlf"
Aug 05 17:14:43.036107 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:14:43.036076    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/af418777-9235-45af-bdeb-a837dcb58a14-bound-sa-token\") pod \"image-registry-547968ddbb-vtdlf\" (UID: \"af418777-9235-45af-bdeb-a837dcb58a14\") " pod="openshift-image-registry/image-registry-547968ddbb-vtdlf"
Aug 05 17:14:43.036212 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:14:43.036155    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/af418777-9235-45af-bdeb-a837dcb58a14-image-registry-private-configuration\") pod \"image-registry-547968ddbb-vtdlf\" (UID: \"af418777-9235-45af-bdeb-a837dcb58a14\") " pod="openshift-image-registry/image-registry-547968ddbb-vtdlf"
Aug 05 17:14:43.036212 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:14:43.036183    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/af418777-9235-45af-bdeb-a837dcb58a14-trusted-ca\") pod \"image-registry-547968ddbb-vtdlf\" (UID: \"af418777-9235-45af-bdeb-a837dcb58a14\") " pod="openshift-image-registry/image-registry-547968ddbb-vtdlf"
Aug 05 17:14:43.036212 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:14:43.036205    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/af418777-9235-45af-bdeb-a837dcb58a14-installation-pull-secrets\") pod \"image-registry-547968ddbb-vtdlf\" (UID: \"af418777-9235-45af-bdeb-a837dcb58a14\") " pod="openshift-image-registry/image-registry-547968ddbb-vtdlf"
Aug 05 17:14:43.036351 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:14:43.036230    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/af418777-9235-45af-bdeb-a837dcb58a14-registry-certificates\") pod \"image-registry-547968ddbb-vtdlf\" (UID: \"af418777-9235-45af-bdeb-a837dcb58a14\") " pod="openshift-image-registry/image-registry-547968ddbb-vtdlf"
Aug 05 17:14:43.036351 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:14:43.036256    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hqtn8\" (UniqueName: \"kubernetes.io/projected/af418777-9235-45af-bdeb-a837dcb58a14-kube-api-access-hqtn8\") pod \"image-registry-547968ddbb-vtdlf\" (UID: \"af418777-9235-45af-bdeb-a837dcb58a14\") " pod="openshift-image-registry/image-registry-547968ddbb-vtdlf"
Aug 05 17:14:43.036351 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:14:43.036287    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/af418777-9235-45af-bdeb-a837dcb58a14-registry-tls\") pod \"image-registry-547968ddbb-vtdlf\" (UID: \"af418777-9235-45af-bdeb-a837dcb58a14\") " pod="openshift-image-registry/image-registry-547968ddbb-vtdlf"
Aug 05 17:14:43.036351 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:14:43.036312    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/af418777-9235-45af-bdeb-a837dcb58a14-ca-trust-extracted\") pod \"image-registry-547968ddbb-vtdlf\" (UID: \"af418777-9235-45af-bdeb-a837dcb58a14\") " pod="openshift-image-registry/image-registry-547968ddbb-vtdlf"
Aug 05 17:14:43.036553 ip-10-0-135-2 kubenswrapper[2571]: E0805 17:14:43.036454    2571 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Aug 05 17:14:43.036553 ip-10-0-135-2 kubenswrapper[2571]: E0805 17:14:43.036475    2571 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-547968ddbb-vtdlf: secret "image-registry-tls" not found
Aug 05 17:14:43.036651 ip-10-0-135-2 kubenswrapper[2571]: E0805 17:14:43.036557    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/af418777-9235-45af-bdeb-a837dcb58a14-registry-tls podName:af418777-9235-45af-bdeb-a837dcb58a14 nodeName:}" failed. No retries permitted until 2026-08-05 17:14:43.536533911 +0000 UTC m=+110.903900233 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/af418777-9235-45af-bdeb-a837dcb58a14-registry-tls") pod "image-registry-547968ddbb-vtdlf" (UID: "af418777-9235-45af-bdeb-a837dcb58a14") : secret "image-registry-tls" not found
Aug 05 17:14:43.036783 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:14:43.036760    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/af418777-9235-45af-bdeb-a837dcb58a14-ca-trust-extracted\") pod \"image-registry-547968ddbb-vtdlf\" (UID: \"af418777-9235-45af-bdeb-a837dcb58a14\") " pod="openshift-image-registry/image-registry-547968ddbb-vtdlf"
Aug 05 17:14:43.036919 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:14:43.036900    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/af418777-9235-45af-bdeb-a837dcb58a14-registry-certificates\") pod \"image-registry-547968ddbb-vtdlf\" (UID: \"af418777-9235-45af-bdeb-a837dcb58a14\") " pod="openshift-image-registry/image-registry-547968ddbb-vtdlf"
Aug 05 17:14:43.037268 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:14:43.037251    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/af418777-9235-45af-bdeb-a837dcb58a14-trusted-ca\") pod \"image-registry-547968ddbb-vtdlf\" (UID: \"af418777-9235-45af-bdeb-a837dcb58a14\") " pod="openshift-image-registry/image-registry-547968ddbb-vtdlf"
Aug 05 17:14:43.039098 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:14:43.039074    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/af418777-9235-45af-bdeb-a837dcb58a14-image-registry-private-configuration\") pod \"image-registry-547968ddbb-vtdlf\" (UID: \"af418777-9235-45af-bdeb-a837dcb58a14\") " pod="openshift-image-registry/image-registry-547968ddbb-vtdlf"
Aug 05 17:14:43.039098 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:14:43.039094    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/af418777-9235-45af-bdeb-a837dcb58a14-installation-pull-secrets\") pod \"image-registry-547968ddbb-vtdlf\" (UID: \"af418777-9235-45af-bdeb-a837dcb58a14\") " pod="openshift-image-registry/image-registry-547968ddbb-vtdlf"
Aug 05 17:14:43.050209 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:14:43.050185    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/af418777-9235-45af-bdeb-a837dcb58a14-bound-sa-token\") pod \"image-registry-547968ddbb-vtdlf\" (UID: \"af418777-9235-45af-bdeb-a837dcb58a14\") " pod="openshift-image-registry/image-registry-547968ddbb-vtdlf"
Aug 05 17:14:43.056129 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:14:43.056109    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hqtn8\" (UniqueName: \"kubernetes.io/projected/af418777-9235-45af-bdeb-a837dcb58a14-kube-api-access-hqtn8\") pod \"image-registry-547968ddbb-vtdlf\" (UID: \"af418777-9235-45af-bdeb-a837dcb58a14\") " pod="openshift-image-registry/image-registry-547968ddbb-vtdlf"
Aug 05 17:14:43.538221 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:14:43.538186    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/af418777-9235-45af-bdeb-a837dcb58a14-registry-tls\") pod \"image-registry-547968ddbb-vtdlf\" (UID: \"af418777-9235-45af-bdeb-a837dcb58a14\") " pod="openshift-image-registry/image-registry-547968ddbb-vtdlf"
Aug 05 17:14:43.538404 ip-10-0-135-2 kubenswrapper[2571]: E0805 17:14:43.538333    2571 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Aug 05 17:14:43.538404 ip-10-0-135-2 kubenswrapper[2571]: E0805 17:14:43.538350    2571 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-547968ddbb-vtdlf: secret "image-registry-tls" not found
Aug 05 17:14:43.538480 ip-10-0-135-2 kubenswrapper[2571]: E0805 17:14:43.538428    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/af418777-9235-45af-bdeb-a837dcb58a14-registry-tls podName:af418777-9235-45af-bdeb-a837dcb58a14 nodeName:}" failed. No retries permitted until 2026-08-05 17:14:44.538413209 +0000 UTC m=+111.905779530 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/af418777-9235-45af-bdeb-a837dcb58a14-registry-tls") pod "image-registry-547968ddbb-vtdlf" (UID: "af418777-9235-45af-bdeb-a837dcb58a14") : secret "image-registry-tls" not found
Aug 05 17:14:44.544984 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:14:44.544953    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/af418777-9235-45af-bdeb-a837dcb58a14-registry-tls\") pod \"image-registry-547968ddbb-vtdlf\" (UID: \"af418777-9235-45af-bdeb-a837dcb58a14\") " pod="openshift-image-registry/image-registry-547968ddbb-vtdlf"
Aug 05 17:14:44.545392 ip-10-0-135-2 kubenswrapper[2571]: E0805 17:14:44.545055    2571 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Aug 05 17:14:44.545392 ip-10-0-135-2 kubenswrapper[2571]: E0805 17:14:44.545065    2571 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-547968ddbb-vtdlf: secret "image-registry-tls" not found
Aug 05 17:14:44.545392 ip-10-0-135-2 kubenswrapper[2571]: E0805 17:14:44.545119    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/af418777-9235-45af-bdeb-a837dcb58a14-registry-tls podName:af418777-9235-45af-bdeb-a837dcb58a14 nodeName:}" failed. No retries permitted until 2026-08-05 17:14:46.545105762 +0000 UTC m=+113.912472084 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/af418777-9235-45af-bdeb-a837dcb58a14-registry-tls") pod "image-registry-547968ddbb-vtdlf" (UID: "af418777-9235-45af-bdeb-a837dcb58a14") : secret "image-registry-tls" not found
Aug 05 17:14:46.557583 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:14:46.557550    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/af418777-9235-45af-bdeb-a837dcb58a14-registry-tls\") pod \"image-registry-547968ddbb-vtdlf\" (UID: \"af418777-9235-45af-bdeb-a837dcb58a14\") " pod="openshift-image-registry/image-registry-547968ddbb-vtdlf"
Aug 05 17:14:46.557954 ip-10-0-135-2 kubenswrapper[2571]: E0805 17:14:46.557659    2571 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Aug 05 17:14:46.557954 ip-10-0-135-2 kubenswrapper[2571]: E0805 17:14:46.557671    2571 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-547968ddbb-vtdlf: secret "image-registry-tls" not found
Aug 05 17:14:46.557954 ip-10-0-135-2 kubenswrapper[2571]: E0805 17:14:46.557716    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/af418777-9235-45af-bdeb-a837dcb58a14-registry-tls podName:af418777-9235-45af-bdeb-a837dcb58a14 nodeName:}" failed. No retries permitted until 2026-08-05 17:14:50.557704057 +0000 UTC m=+117.925070378 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/af418777-9235-45af-bdeb-a837dcb58a14-registry-tls") pod "image-registry-547968ddbb-vtdlf" (UID: "af418777-9235-45af-bdeb-a837dcb58a14") : secret "image-registry-tls" not found
Aug 05 17:14:48.137387 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:14:48.137343    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-5jjlh"]
Aug 05 17:14:48.140204 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:14:48.140186    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-5jjlh"
Aug 05 17:14:48.143058 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:14:48.143030    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Aug 05 17:14:48.143202 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:14:48.143083    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Aug 05 17:14:48.143279 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:14:48.143232    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Aug 05 17:14:48.143338 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:14:48.143289    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Aug 05 17:14:48.144036 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:14:48.144018    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-7l8s7\""
Aug 05 17:14:48.157047 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:14:48.157025    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-5jjlh"]
Aug 05 17:14:48.269703 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:14:48.269674    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/fbccf6ae-01cd-4681-9f55-c702a5b85112-crio-socket\") pod \"insights-runtime-extractor-5jjlh\" (UID: \"fbccf6ae-01cd-4681-9f55-c702a5b85112\") " pod="openshift-insights/insights-runtime-extractor-5jjlh"
Aug 05 17:14:48.269827 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:14:48.269717    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/fbccf6ae-01cd-4681-9f55-c702a5b85112-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-5jjlh\" (UID: \"fbccf6ae-01cd-4681-9f55-c702a5b85112\") " pod="openshift-insights/insights-runtime-extractor-5jjlh"
Aug 05 17:14:48.269827 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:14:48.269746    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hg49k\" (UniqueName: \"kubernetes.io/projected/fbccf6ae-01cd-4681-9f55-c702a5b85112-kube-api-access-hg49k\") pod \"insights-runtime-extractor-5jjlh\" (UID: \"fbccf6ae-01cd-4681-9f55-c702a5b85112\") " pod="openshift-insights/insights-runtime-extractor-5jjlh"
Aug 05 17:14:48.269827 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:14:48.269806    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/fbccf6ae-01cd-4681-9f55-c702a5b85112-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-5jjlh\" (UID: \"fbccf6ae-01cd-4681-9f55-c702a5b85112\") " pod="openshift-insights/insights-runtime-extractor-5jjlh"
Aug 05 17:14:48.269927 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:14:48.269861    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/fbccf6ae-01cd-4681-9f55-c702a5b85112-data-volume\") pod \"insights-runtime-extractor-5jjlh\" (UID: \"fbccf6ae-01cd-4681-9f55-c702a5b85112\") " pod="openshift-insights/insights-runtime-extractor-5jjlh"
Aug 05 17:14:48.370953 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:14:48.370931    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/fbccf6ae-01cd-4681-9f55-c702a5b85112-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-5jjlh\" (UID: \"fbccf6ae-01cd-4681-9f55-c702a5b85112\") " pod="openshift-insights/insights-runtime-extractor-5jjlh"
Aug 05 17:14:48.371116 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:14:48.370960    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hg49k\" (UniqueName: \"kubernetes.io/projected/fbccf6ae-01cd-4681-9f55-c702a5b85112-kube-api-access-hg49k\") pod \"insights-runtime-extractor-5jjlh\" (UID: \"fbccf6ae-01cd-4681-9f55-c702a5b85112\") " pod="openshift-insights/insights-runtime-extractor-5jjlh"
Aug 05 17:14:48.371116 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:14:48.370983    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/fbccf6ae-01cd-4681-9f55-c702a5b85112-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-5jjlh\" (UID: \"fbccf6ae-01cd-4681-9f55-c702a5b85112\") " pod="openshift-insights/insights-runtime-extractor-5jjlh"
Aug 05 17:14:48.371116 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:14:48.371012    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/fbccf6ae-01cd-4681-9f55-c702a5b85112-data-volume\") pod \"insights-runtime-extractor-5jjlh\" (UID: \"fbccf6ae-01cd-4681-9f55-c702a5b85112\") " pod="openshift-insights/insights-runtime-extractor-5jjlh"
Aug 05 17:14:48.371116 ip-10-0-135-2 kubenswrapper[2571]: E0805 17:14:48.371048    2571 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Aug 05 17:14:48.371325 ip-10-0-135-2 kubenswrapper[2571]: E0805 17:14:48.371117    2571 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Aug 05 17:14:48.371325 ip-10-0-135-2 kubenswrapper[2571]: E0805 17:14:48.371125    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/fbccf6ae-01cd-4681-9f55-c702a5b85112-kube-rbac-proxy-cm podName:fbccf6ae-01cd-4681-9f55-c702a5b85112 nodeName:}" failed. No retries permitted until 2026-08-05 17:14:48.871107082 +0000 UTC m=+116.238473434 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/fbccf6ae-01cd-4681-9f55-c702a5b85112-kube-rbac-proxy-cm") pod "insights-runtime-extractor-5jjlh" (UID: "fbccf6ae-01cd-4681-9f55-c702a5b85112") : configmap "kube-rbac-proxy" not found
Aug 05 17:14:48.371325 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:14:48.371056    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/fbccf6ae-01cd-4681-9f55-c702a5b85112-crio-socket\") pod \"insights-runtime-extractor-5jjlh\" (UID: \"fbccf6ae-01cd-4681-9f55-c702a5b85112\") " pod="openshift-insights/insights-runtime-extractor-5jjlh"
Aug 05 17:14:48.371325 ip-10-0-135-2 kubenswrapper[2571]: E0805 17:14:48.371182    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fbccf6ae-01cd-4681-9f55-c702a5b85112-insights-runtime-extractor-tls podName:fbccf6ae-01cd-4681-9f55-c702a5b85112 nodeName:}" failed. No retries permitted until 2026-08-05 17:14:48.871165122 +0000 UTC m=+116.238531449 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/fbccf6ae-01cd-4681-9f55-c702a5b85112-insights-runtime-extractor-tls") pod "insights-runtime-extractor-5jjlh" (UID: "fbccf6ae-01cd-4681-9f55-c702a5b85112") : secret "insights-runtime-extractor-tls" not found
Aug 05 17:14:48.371325 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:14:48.371226    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/fbccf6ae-01cd-4681-9f55-c702a5b85112-crio-socket\") pod \"insights-runtime-extractor-5jjlh\" (UID: \"fbccf6ae-01cd-4681-9f55-c702a5b85112\") " pod="openshift-insights/insights-runtime-extractor-5jjlh"
Aug 05 17:14:48.371325 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:14:48.371321    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/fbccf6ae-01cd-4681-9f55-c702a5b85112-data-volume\") pod \"insights-runtime-extractor-5jjlh\" (UID: \"fbccf6ae-01cd-4681-9f55-c702a5b85112\") " pod="openshift-insights/insights-runtime-extractor-5jjlh"
Aug 05 17:14:48.385895 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:14:48.385866    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hg49k\" (UniqueName: \"kubernetes.io/projected/fbccf6ae-01cd-4681-9f55-c702a5b85112-kube-api-access-hg49k\") pod \"insights-runtime-extractor-5jjlh\" (UID: \"fbccf6ae-01cd-4681-9f55-c702a5b85112\") " pod="openshift-insights/insights-runtime-extractor-5jjlh"
Aug 05 17:14:48.874254 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:14:48.874222    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/fbccf6ae-01cd-4681-9f55-c702a5b85112-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-5jjlh\" (UID: \"fbccf6ae-01cd-4681-9f55-c702a5b85112\") " pod="openshift-insights/insights-runtime-extractor-5jjlh"
Aug 05 17:14:48.874424 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:14:48.874263    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/fbccf6ae-01cd-4681-9f55-c702a5b85112-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-5jjlh\" (UID: \"fbccf6ae-01cd-4681-9f55-c702a5b85112\") " pod="openshift-insights/insights-runtime-extractor-5jjlh"
Aug 05 17:14:48.874424 ip-10-0-135-2 kubenswrapper[2571]: E0805 17:14:48.874342    2571 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Aug 05 17:14:48.874424 ip-10-0-135-2 kubenswrapper[2571]: E0805 17:14:48.874358    2571 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Aug 05 17:14:48.874539 ip-10-0-135-2 kubenswrapper[2571]: E0805 17:14:48.874429    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fbccf6ae-01cd-4681-9f55-c702a5b85112-insights-runtime-extractor-tls podName:fbccf6ae-01cd-4681-9f55-c702a5b85112 nodeName:}" failed. No retries permitted until 2026-08-05 17:14:49.87441168 +0000 UTC m=+117.241778002 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/fbccf6ae-01cd-4681-9f55-c702a5b85112-insights-runtime-extractor-tls") pod "insights-runtime-extractor-5jjlh" (UID: "fbccf6ae-01cd-4681-9f55-c702a5b85112") : secret "insights-runtime-extractor-tls" not found
Aug 05 17:14:48.874539 ip-10-0-135-2 kubenswrapper[2571]: E0805 17:14:48.874469    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/fbccf6ae-01cd-4681-9f55-c702a5b85112-kube-rbac-proxy-cm podName:fbccf6ae-01cd-4681-9f55-c702a5b85112 nodeName:}" failed. No retries permitted until 2026-08-05 17:14:49.874459887 +0000 UTC m=+117.241826212 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/fbccf6ae-01cd-4681-9f55-c702a5b85112-kube-rbac-proxy-cm") pod "insights-runtime-extractor-5jjlh" (UID: "fbccf6ae-01cd-4681-9f55-c702a5b85112") : configmap "kube-rbac-proxy" not found
Aug 05 17:14:49.881894 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:14:49.881861    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/fbccf6ae-01cd-4681-9f55-c702a5b85112-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-5jjlh\" (UID: \"fbccf6ae-01cd-4681-9f55-c702a5b85112\") " pod="openshift-insights/insights-runtime-extractor-5jjlh"
Aug 05 17:14:49.882385 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:14:49.881904    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/fbccf6ae-01cd-4681-9f55-c702a5b85112-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-5jjlh\" (UID: \"fbccf6ae-01cd-4681-9f55-c702a5b85112\") " pod="openshift-insights/insights-runtime-extractor-5jjlh"
Aug 05 17:14:49.882385 ip-10-0-135-2 kubenswrapper[2571]: E0805 17:14:49.882028    2571 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Aug 05 17:14:49.882385 ip-10-0-135-2 kubenswrapper[2571]: E0805 17:14:49.882028    2571 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Aug 05 17:14:49.882385 ip-10-0-135-2 kubenswrapper[2571]: E0805 17:14:49.882099    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fbccf6ae-01cd-4681-9f55-c702a5b85112-insights-runtime-extractor-tls podName:fbccf6ae-01cd-4681-9f55-c702a5b85112 nodeName:}" failed. No retries permitted until 2026-08-05 17:14:51.882071623 +0000 UTC m=+119.249437946 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/fbccf6ae-01cd-4681-9f55-c702a5b85112-insights-runtime-extractor-tls") pod "insights-runtime-extractor-5jjlh" (UID: "fbccf6ae-01cd-4681-9f55-c702a5b85112") : secret "insights-runtime-extractor-tls" not found
Aug 05 17:14:49.882385 ip-10-0-135-2 kubenswrapper[2571]: E0805 17:14:49.882143    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/fbccf6ae-01cd-4681-9f55-c702a5b85112-kube-rbac-proxy-cm podName:fbccf6ae-01cd-4681-9f55-c702a5b85112 nodeName:}" failed. No retries permitted until 2026-08-05 17:14:51.882127733 +0000 UTC m=+119.249494059 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/fbccf6ae-01cd-4681-9f55-c702a5b85112-kube-rbac-proxy-cm") pod "insights-runtime-extractor-5jjlh" (UID: "fbccf6ae-01cd-4681-9f55-c702a5b85112") : configmap "kube-rbac-proxy" not found
Aug 05 17:14:50.587274 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:14:50.587230    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/af418777-9235-45af-bdeb-a837dcb58a14-registry-tls\") pod \"image-registry-547968ddbb-vtdlf\" (UID: \"af418777-9235-45af-bdeb-a837dcb58a14\") " pod="openshift-image-registry/image-registry-547968ddbb-vtdlf"
Aug 05 17:14:50.587455 ip-10-0-135-2 kubenswrapper[2571]: E0805 17:14:50.587353    2571 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Aug 05 17:14:50.587455 ip-10-0-135-2 kubenswrapper[2571]: E0805 17:14:50.587364    2571 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-547968ddbb-vtdlf: secret "image-registry-tls" not found
Aug 05 17:14:50.587455 ip-10-0-135-2 kubenswrapper[2571]: E0805 17:14:50.587432    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/af418777-9235-45af-bdeb-a837dcb58a14-registry-tls podName:af418777-9235-45af-bdeb-a837dcb58a14 nodeName:}" failed. No retries permitted until 2026-08-05 17:14:58.587417439 +0000 UTC m=+125.954783760 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/af418777-9235-45af-bdeb-a837dcb58a14-registry-tls") pod "image-registry-547968ddbb-vtdlf" (UID: "af418777-9235-45af-bdeb-a837dcb58a14") : secret "image-registry-tls" not found
Aug 05 17:14:50.884095 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:14:50.884016    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-qz2cd_c680deac-4ffe-40ff-b521-80beb96c5196/dns-node-resolver/0.log"
Aug 05 17:14:51.886728 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:14:51.886702    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-lrdmn_61599c74-5517-43f4-bed8-a232408b44c0/node-ca/0.log"
Aug 05 17:14:51.896922 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:14:51.896897    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/fbccf6ae-01cd-4681-9f55-c702a5b85112-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-5jjlh\" (UID: \"fbccf6ae-01cd-4681-9f55-c702a5b85112\") " pod="openshift-insights/insights-runtime-extractor-5jjlh"
Aug 05 17:14:51.897013 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:14:51.896935    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/fbccf6ae-01cd-4681-9f55-c702a5b85112-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-5jjlh\" (UID: \"fbccf6ae-01cd-4681-9f55-c702a5b85112\") " pod="openshift-insights/insights-runtime-extractor-5jjlh"
Aug 05 17:14:51.897052 ip-10-0-135-2 kubenswrapper[2571]: E0805 17:14:51.897014    2571 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Aug 05 17:14:51.897084 ip-10-0-135-2 kubenswrapper[2571]: E0805 17:14:51.897064    2571 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Aug 05 17:14:51.897084 ip-10-0-135-2 kubenswrapper[2571]: E0805 17:14:51.897081    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/fbccf6ae-01cd-4681-9f55-c702a5b85112-kube-rbac-proxy-cm podName:fbccf6ae-01cd-4681-9f55-c702a5b85112 nodeName:}" failed. No retries permitted until 2026-08-05 17:14:55.897066481 +0000 UTC m=+123.264432803 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/fbccf6ae-01cd-4681-9f55-c702a5b85112-kube-rbac-proxy-cm") pod "insights-runtime-extractor-5jjlh" (UID: "fbccf6ae-01cd-4681-9f55-c702a5b85112") : configmap "kube-rbac-proxy" not found
Aug 05 17:14:51.897154 ip-10-0-135-2 kubenswrapper[2571]: E0805 17:14:51.897101    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fbccf6ae-01cd-4681-9f55-c702a5b85112-insights-runtime-extractor-tls podName:fbccf6ae-01cd-4681-9f55-c702a5b85112 nodeName:}" failed. No retries permitted until 2026-08-05 17:14:55.897090246 +0000 UTC m=+123.264456567 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/fbccf6ae-01cd-4681-9f55-c702a5b85112-insights-runtime-extractor-tls") pod "insights-runtime-extractor-5jjlh" (UID: "fbccf6ae-01cd-4681-9f55-c702a5b85112") : secret "insights-runtime-extractor-tls" not found
Aug 05 17:14:52.679528 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:14:52.679495    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-phtkl_b59bced4-389d-494c-8376-d050e02710c4/serve-healthcheck-canary/0.log"
Aug 05 17:14:55.928938 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:14:55.928897    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/fbccf6ae-01cd-4681-9f55-c702a5b85112-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-5jjlh\" (UID: \"fbccf6ae-01cd-4681-9f55-c702a5b85112\") " pod="openshift-insights/insights-runtime-extractor-5jjlh"
Aug 05 17:14:55.928938 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:14:55.928941    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/fbccf6ae-01cd-4681-9f55-c702a5b85112-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-5jjlh\" (UID: \"fbccf6ae-01cd-4681-9f55-c702a5b85112\") " pod="openshift-insights/insights-runtime-extractor-5jjlh"
Aug 05 17:14:55.929530 ip-10-0-135-2 kubenswrapper[2571]: E0805 17:14:55.929020    2571 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Aug 05 17:14:55.929530 ip-10-0-135-2 kubenswrapper[2571]: E0805 17:14:55.929087    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/fbccf6ae-01cd-4681-9f55-c702a5b85112-kube-rbac-proxy-cm podName:fbccf6ae-01cd-4681-9f55-c702a5b85112 nodeName:}" failed. No retries permitted until 2026-08-05 17:15:03.929070126 +0000 UTC m=+131.296436446 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/fbccf6ae-01cd-4681-9f55-c702a5b85112-kube-rbac-proxy-cm") pod "insights-runtime-extractor-5jjlh" (UID: "fbccf6ae-01cd-4681-9f55-c702a5b85112") : configmap "kube-rbac-proxy" not found
Aug 05 17:14:55.929530 ip-10-0-135-2 kubenswrapper[2571]: E0805 17:14:55.929142    2571 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Aug 05 17:14:55.929530 ip-10-0-135-2 kubenswrapper[2571]: E0805 17:14:55.929227    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fbccf6ae-01cd-4681-9f55-c702a5b85112-insights-runtime-extractor-tls podName:fbccf6ae-01cd-4681-9f55-c702a5b85112 nodeName:}" failed. No retries permitted until 2026-08-05 17:15:03.929207068 +0000 UTC m=+131.296573406 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/fbccf6ae-01cd-4681-9f55-c702a5b85112-insights-runtime-extractor-tls") pod "insights-runtime-extractor-5jjlh" (UID: "fbccf6ae-01cd-4681-9f55-c702a5b85112") : secret "insights-runtime-extractor-tls" not found
Aug 05 17:14:58.646467 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:14:58.646425    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/af418777-9235-45af-bdeb-a837dcb58a14-registry-tls\") pod \"image-registry-547968ddbb-vtdlf\" (UID: \"af418777-9235-45af-bdeb-a837dcb58a14\") " pod="openshift-image-registry/image-registry-547968ddbb-vtdlf"
Aug 05 17:14:58.648933 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:14:58.648911    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/af418777-9235-45af-bdeb-a837dcb58a14-registry-tls\") pod \"image-registry-547968ddbb-vtdlf\" (UID: \"af418777-9235-45af-bdeb-a837dcb58a14\") " pod="openshift-image-registry/image-registry-547968ddbb-vtdlf"
Aug 05 17:14:58.742835 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:14:58.742806    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-g4tv2\""
Aug 05 17:14:58.750958 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:14:58.750937    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-547968ddbb-vtdlf"
Aug 05 17:14:58.878925 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:14:58.878897    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-547968ddbb-vtdlf"]
Aug 05 17:14:58.881559 ip-10-0-135-2 kubenswrapper[2571]: W0805 17:14:58.881537    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaf418777_9235_45af_bdeb_a837dcb58a14.slice/crio-f2ebe64b373d2382c9709f10a35aa76106956e9c05f6cdcc6d65c1b97aa48ce8 WatchSource:0}: Error finding container f2ebe64b373d2382c9709f10a35aa76106956e9c05f6cdcc6d65c1b97aa48ce8: Status 404 returned error can't find the container with id f2ebe64b373d2382c9709f10a35aa76106956e9c05f6cdcc6d65c1b97aa48ce8
Aug 05 17:14:59.568200 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:14:59.568162    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-547968ddbb-vtdlf" event={"ID":"af418777-9235-45af-bdeb-a837dcb58a14","Type":"ContainerStarted","Data":"a8ad44c34f2a7fcdd92bd986e1dfed4964f3c5ca82440605f3fed89678804524"}
Aug 05 17:14:59.568409 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:14:59.568217    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-547968ddbb-vtdlf" event={"ID":"af418777-9235-45af-bdeb-a837dcb58a14","Type":"ContainerStarted","Data":"f2ebe64b373d2382c9709f10a35aa76106956e9c05f6cdcc6d65c1b97aa48ce8"}
Aug 05 17:14:59.568409 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:14:59.568314    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-547968ddbb-vtdlf"
Aug 05 17:14:59.589223 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:14:59.589173    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-547968ddbb-vtdlf" podStartSLOduration=17.589156857 podStartE2EDuration="17.589156857s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-05 17:14:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-05 17:14:59.587928283 +0000 UTC m=+126.955294645" watchObservedRunningTime="2026-08-05 17:14:59.589156857 +0000 UTC m=+126.956523200"
Aug 05 17:15:02.875522 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:15:02.875474    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/da6f032b-35a1-4c51-b115-defaf9b2a9a5-metrics-certs\") pod \"network-metrics-daemon-677x7\" (UID: \"da6f032b-35a1-4c51-b115-defaf9b2a9a5\") " pod="openshift-multus/network-metrics-daemon-677x7"
Aug 05 17:15:02.878139 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:15:02.878107    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/da6f032b-35a1-4c51-b115-defaf9b2a9a5-metrics-certs\") pod \"network-metrics-daemon-677x7\" (UID: \"da6f032b-35a1-4c51-b115-defaf9b2a9a5\") " pod="openshift-multus/network-metrics-daemon-677x7"
Aug 05 17:15:02.954240 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:15:02.954206    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-z9xpb\""
Aug 05 17:15:02.961918 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:15:02.961887    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-677x7"
Aug 05 17:15:03.092023 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:15:03.091988    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-677x7"]
Aug 05 17:15:03.098507 ip-10-0-135-2 kubenswrapper[2571]: W0805 17:15:03.098478    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podda6f032b_35a1_4c51_b115_defaf9b2a9a5.slice/crio-7db30c007ed0cbea3dbc2f48371ba5147c5c7a57cbd536d4abed1665f6808a41 WatchSource:0}: Error finding container 7db30c007ed0cbea3dbc2f48371ba5147c5c7a57cbd536d4abed1665f6808a41: Status 404 returned error can't find the container with id 7db30c007ed0cbea3dbc2f48371ba5147c5c7a57cbd536d4abed1665f6808a41
Aug 05 17:15:03.580572 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:15:03.580539    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-677x7" event={"ID":"da6f032b-35a1-4c51-b115-defaf9b2a9a5","Type":"ContainerStarted","Data":"7db30c007ed0cbea3dbc2f48371ba5147c5c7a57cbd536d4abed1665f6808a41"}
Aug 05 17:15:03.983409 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:15:03.983361    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/fbccf6ae-01cd-4681-9f55-c702a5b85112-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-5jjlh\" (UID: \"fbccf6ae-01cd-4681-9f55-c702a5b85112\") " pod="openshift-insights/insights-runtime-extractor-5jjlh"
Aug 05 17:15:03.983754 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:15:03.983427    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/fbccf6ae-01cd-4681-9f55-c702a5b85112-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-5jjlh\" (UID: \"fbccf6ae-01cd-4681-9f55-c702a5b85112\") " pod="openshift-insights/insights-runtime-extractor-5jjlh"
Aug 05 17:15:03.983830 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:15:03.983805    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/fbccf6ae-01cd-4681-9f55-c702a5b85112-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-5jjlh\" (UID: \"fbccf6ae-01cd-4681-9f55-c702a5b85112\") " pod="openshift-insights/insights-runtime-extractor-5jjlh"
Aug 05 17:15:03.985548 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:15:03.985525    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/fbccf6ae-01cd-4681-9f55-c702a5b85112-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-5jjlh\" (UID: \"fbccf6ae-01cd-4681-9f55-c702a5b85112\") " pod="openshift-insights/insights-runtime-extractor-5jjlh"
Aug 05 17:15:04.052818 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:15:04.052726    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-7l8s7\""
Aug 05 17:15:04.060850 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:15:04.060833    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-5jjlh"
Aug 05 17:15:04.198988 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:15:04.198960    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-5jjlh"]
Aug 05 17:15:04.201314 ip-10-0-135-2 kubenswrapper[2571]: W0805 17:15:04.201285    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfbccf6ae_01cd_4681_9f55_c702a5b85112.slice/crio-5664c29c51a9e1ddf42924367f74a8346b89db1a0cd123cc8364741eca196a03 WatchSource:0}: Error finding container 5664c29c51a9e1ddf42924367f74a8346b89db1a0cd123cc8364741eca196a03: Status 404 returned error can't find the container with id 5664c29c51a9e1ddf42924367f74a8346b89db1a0cd123cc8364741eca196a03
Aug 05 17:15:04.586550 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:15:04.586458    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-677x7" event={"ID":"da6f032b-35a1-4c51-b115-defaf9b2a9a5","Type":"ContainerStarted","Data":"bd76925bddd27fcce08eb5a6da243ea43111b75bdf90f0469ce02bb31becbb95"}
Aug 05 17:15:04.586695 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:15:04.586552    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-677x7" event={"ID":"da6f032b-35a1-4c51-b115-defaf9b2a9a5","Type":"ContainerStarted","Data":"08d50d2f6c113e9f1b5c194dd434ca124f09332916b588679a0ab6e1289b6312"}
Aug 05 17:15:04.587794 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:15:04.587768    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-5jjlh" event={"ID":"fbccf6ae-01cd-4681-9f55-c702a5b85112","Type":"ContainerStarted","Data":"bcdcdb1c42b2c9acb72013c9c21b2e3a5439f1b8b514df3528f2d9e6a0c3332d"}
Aug 05 17:15:04.587914 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:15:04.587815    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-5jjlh" event={"ID":"fbccf6ae-01cd-4681-9f55-c702a5b85112","Type":"ContainerStarted","Data":"5664c29c51a9e1ddf42924367f74a8346b89db1a0cd123cc8364741eca196a03"}
Aug 05 17:15:04.606601 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:15:04.606562    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-677x7" podStartSLOduration=129.685249444 podStartE2EDuration="2m10.606548596s" totalImagesPullingTime="921.299152ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-05 17:12:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-05 17:15:04.605180352 +0000 UTC m=+131.972546694" watchObservedRunningTime="2026-08-05 17:15:04.606548596 +0000 UTC m=+131.973914940"
Aug 05 17:15:05.592145 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:15:05.592111    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-5jjlh" event={"ID":"fbccf6ae-01cd-4681-9f55-c702a5b85112","Type":"ContainerStarted","Data":"89746a9fa1fa4dd614328285b736f0a487d99869b728ee2986bf1b0ec3c577b4"}
Aug 05 17:15:07.600634 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:15:07.600599    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-5jjlh" event={"ID":"fbccf6ae-01cd-4681-9f55-c702a5b85112","Type":"ContainerStarted","Data":"c2ae8b4cdfeb3475487ecd6bc3863fd6b114e84c30f6af098b2061ff4e2861a0"}
Aug 05 17:15:07.633773 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:15:07.633723    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-5jjlh" podStartSLOduration=17.38597443 podStartE2EDuration="19.633707153s" totalImagesPullingTime="2.247732723s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-08-05 17:14:48 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-05 17:15:07.633531337 +0000 UTC m=+135.000897681" watchObservedRunningTime="2026-08-05 17:15:07.633707153 +0000 UTC m=+135.001073495"
Aug 05 17:15:09.454038 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:15:09.454002    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-console/networking-console-plugin-f77f48c6d-rd4jh"]
Aug 05 17:15:09.456884 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:15:09.456866    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-f77f48c6d-rd4jh"
Aug 05 17:15:09.460878 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:15:09.460853    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-c4r25\""
Aug 05 17:15:09.460878 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:15:09.460858    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\""
Aug 05 17:15:09.461025 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:15:09.460960    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\""
Aug 05 17:15:09.465134 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:15:09.465115    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-547968ddbb-vtdlf"]
Aug 05 17:15:09.467078 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:15:09.467052    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-f77f48c6d-rd4jh"]
Aug 05 17:15:09.623474 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:15:09.623448    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/a0be5476-71e0-4242-878f-a37806d86403-networking-console-plugin-cert\") pod \"networking-console-plugin-f77f48c6d-rd4jh\" (UID: \"a0be5476-71e0-4242-878f-a37806d86403\") " pod="openshift-network-console/networking-console-plugin-f77f48c6d-rd4jh"
Aug 05 17:15:09.623613 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:15:09.623503    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/a0be5476-71e0-4242-878f-a37806d86403-nginx-conf\") pod \"networking-console-plugin-f77f48c6d-rd4jh\" (UID: \"a0be5476-71e0-4242-878f-a37806d86403\") " pod="openshift-network-console/networking-console-plugin-f77f48c6d-rd4jh"
Aug 05 17:15:09.724749 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:15:09.724682    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/a0be5476-71e0-4242-878f-a37806d86403-nginx-conf\") pod \"networking-console-plugin-f77f48c6d-rd4jh\" (UID: \"a0be5476-71e0-4242-878f-a37806d86403\") " pod="openshift-network-console/networking-console-plugin-f77f48c6d-rd4jh"
Aug 05 17:15:09.724749 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:15:09.724732    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/a0be5476-71e0-4242-878f-a37806d86403-networking-console-plugin-cert\") pod \"networking-console-plugin-f77f48c6d-rd4jh\" (UID: \"a0be5476-71e0-4242-878f-a37806d86403\") " pod="openshift-network-console/networking-console-plugin-f77f48c6d-rd4jh"
Aug 05 17:15:09.725278 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:15:09.725256    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/a0be5476-71e0-4242-878f-a37806d86403-nginx-conf\") pod \"networking-console-plugin-f77f48c6d-rd4jh\" (UID: \"a0be5476-71e0-4242-878f-a37806d86403\") " pod="openshift-network-console/networking-console-plugin-f77f48c6d-rd4jh"
Aug 05 17:15:09.727051 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:15:09.727031    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/a0be5476-71e0-4242-878f-a37806d86403-networking-console-plugin-cert\") pod \"networking-console-plugin-f77f48c6d-rd4jh\" (UID: \"a0be5476-71e0-4242-878f-a37806d86403\") " pod="openshift-network-console/networking-console-plugin-f77f48c6d-rd4jh"
Aug 05 17:15:09.765063 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:15:09.765044    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-f77f48c6d-rd4jh"
Aug 05 17:15:09.902297 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:15:09.902268    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-f77f48c6d-rd4jh"]
Aug 05 17:15:09.905093 ip-10-0-135-2 kubenswrapper[2571]: W0805 17:15:09.905069    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda0be5476_71e0_4242_878f_a37806d86403.slice/crio-64c1e6107281b0294c360d365b141066cc42082c195c8eaf00af4d6b7e5393da WatchSource:0}: Error finding container 64c1e6107281b0294c360d365b141066cc42082c195c8eaf00af4d6b7e5393da: Status 404 returned error can't find the container with id 64c1e6107281b0294c360d365b141066cc42082c195c8eaf00af4d6b7e5393da
Aug 05 17:15:10.611440 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:15:10.611404    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-f77f48c6d-rd4jh" event={"ID":"a0be5476-71e0-4242-878f-a37806d86403","Type":"ContainerStarted","Data":"64c1e6107281b0294c360d365b141066cc42082c195c8eaf00af4d6b7e5393da"}
Aug 05 17:15:11.615273 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:15:11.615239    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-f77f48c6d-rd4jh" event={"ID":"a0be5476-71e0-4242-878f-a37806d86403","Type":"ContainerStarted","Data":"0ad4f861fed1838214b9d5feafb62824532b81191ed742a5155c6aaa3da269af"}
Aug 05 17:15:11.639392 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:15:11.639336    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-console/networking-console-plugin-f77f48c6d-rd4jh" podStartSLOduration=1.6082945789999998 podStartE2EDuration="2.639319282s" totalImagesPullingTime="1.031024703s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-05 17:15:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-05 17:15:11.638540002 +0000 UTC m=+139.005906347" watchObservedRunningTime="2026-08-05 17:15:11.639319282 +0000 UTC m=+139.006685625"
Aug 05 17:15:19.471139 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:15:19.471098    2571 patch_prober.go:28] interesting pod/image-registry-547968ddbb-vtdlf container/registry namespace/openshift-image-registry: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]}
Aug 05 17:15:19.471515 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:15:19.471161    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-image-registry/image-registry-547968ddbb-vtdlf" podUID="af418777-9235-45af-bdeb-a837dcb58a14" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Aug 05 17:15:26.901266 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:15:26.901232    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-6jh6w"]
Aug 05 17:15:26.905757 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:15:26.905736    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-6jh6w"
Aug 05 17:15:26.908228 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:15:26.908207    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Aug 05 17:15:26.908330 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:15:26.908214    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Aug 05 17:15:26.908533 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:15:26.908517    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-m5z48\""
Aug 05 17:15:26.908635 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:15:26.908550    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Aug 05 17:15:26.908635 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:15:26.908586    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Aug 05 17:15:26.909224 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:15:26.909209    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Aug 05 17:15:26.909295 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:15:26.909285    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Aug 05 17:15:27.042459 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:15:27.042428    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/5de6dda3-4f32-4fbe-be23-c9fd0f0b9f8d-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-6jh6w\" (UID: \"5de6dda3-4f32-4fbe-be23-c9fd0f0b9f8d\") " pod="openshift-monitoring/node-exporter-6jh6w"
Aug 05 17:15:27.042599 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:15:27.042471    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/5de6dda3-4f32-4fbe-be23-c9fd0f0b9f8d-node-exporter-textfile\") pod \"node-exporter-6jh6w\" (UID: \"5de6dda3-4f32-4fbe-be23-c9fd0f0b9f8d\") " pod="openshift-monitoring/node-exporter-6jh6w"
Aug 05 17:15:27.042599 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:15:27.042498    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5de6dda3-4f32-4fbe-be23-c9fd0f0b9f8d-sys\") pod \"node-exporter-6jh6w\" (UID: \"5de6dda3-4f32-4fbe-be23-c9fd0f0b9f8d\") " pod="openshift-monitoring/node-exporter-6jh6w"
Aug 05 17:15:27.042599 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:15:27.042544    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/5de6dda3-4f32-4fbe-be23-c9fd0f0b9f8d-node-exporter-wtmp\") pod \"node-exporter-6jh6w\" (UID: \"5de6dda3-4f32-4fbe-be23-c9fd0f0b9f8d\") " pod="openshift-monitoring/node-exporter-6jh6w"
Aug 05 17:15:27.042599 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:15:27.042574    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/5de6dda3-4f32-4fbe-be23-c9fd0f0b9f8d-node-exporter-tls\") pod \"node-exporter-6jh6w\" (UID: \"5de6dda3-4f32-4fbe-be23-c9fd0f0b9f8d\") " pod="openshift-monitoring/node-exporter-6jh6w"
Aug 05 17:15:27.042772 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:15:27.042655    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/5de6dda3-4f32-4fbe-be23-c9fd0f0b9f8d-node-exporter-accelerators-collector-config\") pod \"node-exporter-6jh6w\" (UID: \"5de6dda3-4f32-4fbe-be23-c9fd0f0b9f8d\") " pod="openshift-monitoring/node-exporter-6jh6w"
Aug 05 17:15:27.042772 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:15:27.042696    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/5de6dda3-4f32-4fbe-be23-c9fd0f0b9f8d-root\") pod \"node-exporter-6jh6w\" (UID: \"5de6dda3-4f32-4fbe-be23-c9fd0f0b9f8d\") " pod="openshift-monitoring/node-exporter-6jh6w"
Aug 05 17:15:27.042772 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:15:27.042716    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dhp8g\" (UniqueName: \"kubernetes.io/projected/5de6dda3-4f32-4fbe-be23-c9fd0f0b9f8d-kube-api-access-dhp8g\") pod \"node-exporter-6jh6w\" (UID: \"5de6dda3-4f32-4fbe-be23-c9fd0f0b9f8d\") " pod="openshift-monitoring/node-exporter-6jh6w"
Aug 05 17:15:27.042772 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:15:27.042744    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5de6dda3-4f32-4fbe-be23-c9fd0f0b9f8d-metrics-client-ca\") pod \"node-exporter-6jh6w\" (UID: \"5de6dda3-4f32-4fbe-be23-c9fd0f0b9f8d\") " pod="openshift-monitoring/node-exporter-6jh6w"
Aug 05 17:15:27.143478 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:15:27.143453    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/5de6dda3-4f32-4fbe-be23-c9fd0f0b9f8d-node-exporter-tls\") pod \"node-exporter-6jh6w\" (UID: \"5de6dda3-4f32-4fbe-be23-c9fd0f0b9f8d\") " pod="openshift-monitoring/node-exporter-6jh6w"
Aug 05 17:15:27.143603 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:15:27.143494    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/5de6dda3-4f32-4fbe-be23-c9fd0f0b9f8d-node-exporter-accelerators-collector-config\") pod \"node-exporter-6jh6w\" (UID: \"5de6dda3-4f32-4fbe-be23-c9fd0f0b9f8d\") " pod="openshift-monitoring/node-exporter-6jh6w"
Aug 05 17:15:27.143603 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:15:27.143525    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/5de6dda3-4f32-4fbe-be23-c9fd0f0b9f8d-root\") pod \"node-exporter-6jh6w\" (UID: \"5de6dda3-4f32-4fbe-be23-c9fd0f0b9f8d\") " pod="openshift-monitoring/node-exporter-6jh6w"
Aug 05 17:15:27.143603 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:15:27.143542    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dhp8g\" (UniqueName: \"kubernetes.io/projected/5de6dda3-4f32-4fbe-be23-c9fd0f0b9f8d-kube-api-access-dhp8g\") pod \"node-exporter-6jh6w\" (UID: \"5de6dda3-4f32-4fbe-be23-c9fd0f0b9f8d\") " pod="openshift-monitoring/node-exporter-6jh6w"
Aug 05 17:15:27.143603 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:15:27.143565    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5de6dda3-4f32-4fbe-be23-c9fd0f0b9f8d-metrics-client-ca\") pod \"node-exporter-6jh6w\" (UID: \"5de6dda3-4f32-4fbe-be23-c9fd0f0b9f8d\") " pod="openshift-monitoring/node-exporter-6jh6w"
Aug 05 17:15:27.143603 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:15:27.143585    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/5de6dda3-4f32-4fbe-be23-c9fd0f0b9f8d-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-6jh6w\" (UID: \"5de6dda3-4f32-4fbe-be23-c9fd0f0b9f8d\") " pod="openshift-monitoring/node-exporter-6jh6w"
Aug 05 17:15:27.143844 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:15:27.143607    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/5de6dda3-4f32-4fbe-be23-c9fd0f0b9f8d-node-exporter-textfile\") pod \"node-exporter-6jh6w\" (UID: \"5de6dda3-4f32-4fbe-be23-c9fd0f0b9f8d\") " pod="openshift-monitoring/node-exporter-6jh6w"
Aug 05 17:15:27.143844 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:15:27.143624    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5de6dda3-4f32-4fbe-be23-c9fd0f0b9f8d-sys\") pod \"node-exporter-6jh6w\" (UID: \"5de6dda3-4f32-4fbe-be23-c9fd0f0b9f8d\") " pod="openshift-monitoring/node-exporter-6jh6w"
Aug 05 17:15:27.143844 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:15:27.143636    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/5de6dda3-4f32-4fbe-be23-c9fd0f0b9f8d-root\") pod \"node-exporter-6jh6w\" (UID: \"5de6dda3-4f32-4fbe-be23-c9fd0f0b9f8d\") " pod="openshift-monitoring/node-exporter-6jh6w"
Aug 05 17:15:27.143844 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:15:27.143650    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/5de6dda3-4f32-4fbe-be23-c9fd0f0b9f8d-node-exporter-wtmp\") pod \"node-exporter-6jh6w\" (UID: \"5de6dda3-4f32-4fbe-be23-c9fd0f0b9f8d\") " pod="openshift-monitoring/node-exporter-6jh6w"
Aug 05 17:15:27.143844 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:15:27.143708    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5de6dda3-4f32-4fbe-be23-c9fd0f0b9f8d-sys\") pod \"node-exporter-6jh6w\" (UID: \"5de6dda3-4f32-4fbe-be23-c9fd0f0b9f8d\") " pod="openshift-monitoring/node-exporter-6jh6w"
Aug 05 17:15:27.143844 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:15:27.143827    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/5de6dda3-4f32-4fbe-be23-c9fd0f0b9f8d-node-exporter-wtmp\") pod \"node-exporter-6jh6w\" (UID: \"5de6dda3-4f32-4fbe-be23-c9fd0f0b9f8d\") " pod="openshift-monitoring/node-exporter-6jh6w"
Aug 05 17:15:27.144123 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:15:27.144024    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/5de6dda3-4f32-4fbe-be23-c9fd0f0b9f8d-node-exporter-textfile\") pod \"node-exporter-6jh6w\" (UID: \"5de6dda3-4f32-4fbe-be23-c9fd0f0b9f8d\") " pod="openshift-monitoring/node-exporter-6jh6w"
Aug 05 17:15:27.144270 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:15:27.144242    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/5de6dda3-4f32-4fbe-be23-c9fd0f0b9f8d-node-exporter-accelerators-collector-config\") pod \"node-exporter-6jh6w\" (UID: \"5de6dda3-4f32-4fbe-be23-c9fd0f0b9f8d\") " pod="openshift-monitoring/node-exporter-6jh6w"
Aug 05 17:15:27.144365 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:15:27.144308    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5de6dda3-4f32-4fbe-be23-c9fd0f0b9f8d-metrics-client-ca\") pod \"node-exporter-6jh6w\" (UID: \"5de6dda3-4f32-4fbe-be23-c9fd0f0b9f8d\") " pod="openshift-monitoring/node-exporter-6jh6w"
Aug 05 17:15:27.145772 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:15:27.145751    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/5de6dda3-4f32-4fbe-be23-c9fd0f0b9f8d-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-6jh6w\" (UID: \"5de6dda3-4f32-4fbe-be23-c9fd0f0b9f8d\") " pod="openshift-monitoring/node-exporter-6jh6w"
Aug 05 17:15:27.145937 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:15:27.145921    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/5de6dda3-4f32-4fbe-be23-c9fd0f0b9f8d-node-exporter-tls\") pod \"node-exporter-6jh6w\" (UID: \"5de6dda3-4f32-4fbe-be23-c9fd0f0b9f8d\") " pod="openshift-monitoring/node-exporter-6jh6w"
Aug 05 17:15:27.153917 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:15:27.153867    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dhp8g\" (UniqueName: \"kubernetes.io/projected/5de6dda3-4f32-4fbe-be23-c9fd0f0b9f8d-kube-api-access-dhp8g\") pod \"node-exporter-6jh6w\" (UID: \"5de6dda3-4f32-4fbe-be23-c9fd0f0b9f8d\") " pod="openshift-monitoring/node-exporter-6jh6w"
Aug 05 17:15:27.215056 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:15:27.215034    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-6jh6w"
Aug 05 17:15:27.226414 ip-10-0-135-2 kubenswrapper[2571]: W0805 17:15:27.226390    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5de6dda3_4f32_4fbe_be23_c9fd0f0b9f8d.slice/crio-5c269323f2bdddc8805b6f9d0a80abbfcc706ced01293c0528d86947a37be0e7 WatchSource:0}: Error finding container 5c269323f2bdddc8805b6f9d0a80abbfcc706ced01293c0528d86947a37be0e7: Status 404 returned error can't find the container with id 5c269323f2bdddc8805b6f9d0a80abbfcc706ced01293c0528d86947a37be0e7
Aug 05 17:15:27.664102 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:15:27.664065    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-6jh6w" event={"ID":"5de6dda3-4f32-4fbe-be23-c9fd0f0b9f8d","Type":"ContainerStarted","Data":"5c269323f2bdddc8805b6f9d0a80abbfcc706ced01293c0528d86947a37be0e7"}
Aug 05 17:15:28.185686 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:15:28.185653    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Aug 05 17:15:28.190049 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:15:28.190029    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Aug 05 17:15:28.199045 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:15:28.199025    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
Aug 05 17:15:28.199045 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:15:28.199036    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
Aug 05 17:15:28.199215 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:15:28.199099    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
Aug 05 17:15:28.199215 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:15:28.199143    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
Aug 05 17:15:28.199215 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:15:28.199173    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
Aug 05 17:15:28.199424 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:15:28.199407    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-8j7wv\""
Aug 05 17:15:28.200009 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:15:28.199990    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
Aug 05 17:15:28.200094 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:15:28.200010    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
Aug 05 17:15:28.200094 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:15:28.200046    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
Aug 05 17:15:28.200195 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:15:28.199989    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
Aug 05 17:15:28.217752 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:15:28.217729    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Aug 05 17:15:28.353803 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:15:28.353774    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/cd891c78-e151-4947-9b17-8e6661e14e3d-config-out\") pod \"alertmanager-main-0\" (UID: \"cd891c78-e151-4947-9b17-8e6661e14e3d\") " pod="openshift-monitoring/alertmanager-main-0"
Aug 05 17:15:28.353803 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:15:28.353808    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/cd891c78-e151-4947-9b17-8e6661e14e3d-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"cd891c78-e151-4947-9b17-8e6661e14e3d\") " pod="openshift-monitoring/alertmanager-main-0"
Aug 05 17:15:28.354004 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:15:28.353829    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/cd891c78-e151-4947-9b17-8e6661e14e3d-web-config\") pod \"alertmanager-main-0\" (UID: \"cd891c78-e151-4947-9b17-8e6661e14e3d\") " pod="openshift-monitoring/alertmanager-main-0"
Aug 05 17:15:28.354004 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:15:28.353875    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/cd891c78-e151-4947-9b17-8e6661e14e3d-config-volume\") pod \"alertmanager-main-0\" (UID: \"cd891c78-e151-4947-9b17-8e6661e14e3d\") " pod="openshift-monitoring/alertmanager-main-0"
Aug 05 17:15:28.354004 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:15:28.353913    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/cd891c78-e151-4947-9b17-8e6661e14e3d-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"cd891c78-e151-4947-9b17-8e6661e14e3d\") " pod="openshift-monitoring/alertmanager-main-0"
Aug 05 17:15:28.354004 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:15:28.353939    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/cd891c78-e151-4947-9b17-8e6661e14e3d-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"cd891c78-e151-4947-9b17-8e6661e14e3d\") " pod="openshift-monitoring/alertmanager-main-0"
Aug 05 17:15:28.354004 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:15:28.353983    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q5mnj\" (UniqueName: \"kubernetes.io/projected/cd891c78-e151-4947-9b17-8e6661e14e3d-kube-api-access-q5mnj\") pod \"alertmanager-main-0\" (UID: \"cd891c78-e151-4947-9b17-8e6661e14e3d\") " pod="openshift-monitoring/alertmanager-main-0"
Aug 05 17:15:28.354217 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:15:28.354028    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/cd891c78-e151-4947-9b17-8e6661e14e3d-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"cd891c78-e151-4947-9b17-8e6661e14e3d\") " pod="openshift-monitoring/alertmanager-main-0"
Aug 05 17:15:28.354217 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:15:28.354068    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/cd891c78-e151-4947-9b17-8e6661e14e3d-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"cd891c78-e151-4947-9b17-8e6661e14e3d\") " pod="openshift-monitoring/alertmanager-main-0"
Aug 05 17:15:28.354217 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:15:28.354122    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/cd891c78-e151-4947-9b17-8e6661e14e3d-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"cd891c78-e151-4947-9b17-8e6661e14e3d\") " pod="openshift-monitoring/alertmanager-main-0"
Aug 05 17:15:28.354217 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:15:28.354152    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/cd891c78-e151-4947-9b17-8e6661e14e3d-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"cd891c78-e151-4947-9b17-8e6661e14e3d\") " pod="openshift-monitoring/alertmanager-main-0"
Aug 05 17:15:28.354217 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:15:28.354199    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/cd891c78-e151-4947-9b17-8e6661e14e3d-tls-assets\") pod \"alertmanager-main-0\" (UID: \"cd891c78-e151-4947-9b17-8e6661e14e3d\") " pod="openshift-monitoring/alertmanager-main-0"
Aug 05 17:15:28.354402 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:15:28.354226    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cd891c78-e151-4947-9b17-8e6661e14e3d-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"cd891c78-e151-4947-9b17-8e6661e14e3d\") " pod="openshift-monitoring/alertmanager-main-0"
Aug 05 17:15:28.455237 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:15:28.455216    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cd891c78-e151-4947-9b17-8e6661e14e3d-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"cd891c78-e151-4947-9b17-8e6661e14e3d\") " pod="openshift-monitoring/alertmanager-main-0"
Aug 05 17:15:28.455345 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:15:28.455258    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/cd891c78-e151-4947-9b17-8e6661e14e3d-config-out\") pod \"alertmanager-main-0\" (UID: \"cd891c78-e151-4947-9b17-8e6661e14e3d\") " pod="openshift-monitoring/alertmanager-main-0"
Aug 05 17:15:28.455345 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:15:28.455279    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/cd891c78-e151-4947-9b17-8e6661e14e3d-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"cd891c78-e151-4947-9b17-8e6661e14e3d\") " pod="openshift-monitoring/alertmanager-main-0"
Aug 05 17:15:28.455345 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:15:28.455303    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/cd891c78-e151-4947-9b17-8e6661e14e3d-web-config\") pod \"alertmanager-main-0\" (UID: \"cd891c78-e151-4947-9b17-8e6661e14e3d\") " pod="openshift-monitoring/alertmanager-main-0"
Aug 05 17:15:28.455345 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:15:28.455323    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/cd891c78-e151-4947-9b17-8e6661e14e3d-config-volume\") pod \"alertmanager-main-0\" (UID: \"cd891c78-e151-4947-9b17-8e6661e14e3d\") " pod="openshift-monitoring/alertmanager-main-0"
Aug 05 17:15:28.455535 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:15:28.455350    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/cd891c78-e151-4947-9b17-8e6661e14e3d-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"cd891c78-e151-4947-9b17-8e6661e14e3d\") " pod="openshift-monitoring/alertmanager-main-0"
Aug 05 17:15:28.455535 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:15:28.455385    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/cd891c78-e151-4947-9b17-8e6661e14e3d-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"cd891c78-e151-4947-9b17-8e6661e14e3d\") " pod="openshift-monitoring/alertmanager-main-0"
Aug 05 17:15:28.455535 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:15:28.455410    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q5mnj\" (UniqueName: \"kubernetes.io/projected/cd891c78-e151-4947-9b17-8e6661e14e3d-kube-api-access-q5mnj\") pod \"alertmanager-main-0\" (UID: \"cd891c78-e151-4947-9b17-8e6661e14e3d\") " pod="openshift-monitoring/alertmanager-main-0"
Aug 05 17:15:28.455714 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:15:28.455676    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/cd891c78-e151-4947-9b17-8e6661e14e3d-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"cd891c78-e151-4947-9b17-8e6661e14e3d\") " pod="openshift-monitoring/alertmanager-main-0"
Aug 05 17:15:28.455822 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:15:28.455727    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/cd891c78-e151-4947-9b17-8e6661e14e3d-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"cd891c78-e151-4947-9b17-8e6661e14e3d\") " pod="openshift-monitoring/alertmanager-main-0"
Aug 05 17:15:28.455822 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:15:28.455768    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/cd891c78-e151-4947-9b17-8e6661e14e3d-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"cd891c78-e151-4947-9b17-8e6661e14e3d\") " pod="openshift-monitoring/alertmanager-main-0"
Aug 05 17:15:28.455822 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:15:28.455814    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/cd891c78-e151-4947-9b17-8e6661e14e3d-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"cd891c78-e151-4947-9b17-8e6661e14e3d\") " pod="openshift-monitoring/alertmanager-main-0"
Aug 05 17:15:28.455938 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:15:28.455844    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/cd891c78-e151-4947-9b17-8e6661e14e3d-tls-assets\") pod \"alertmanager-main-0\" (UID: \"cd891c78-e151-4947-9b17-8e6661e14e3d\") " pod="openshift-monitoring/alertmanager-main-0"
Aug 05 17:15:28.456039 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:15:28.456016    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/cd891c78-e151-4947-9b17-8e6661e14e3d-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"cd891c78-e151-4947-9b17-8e6661e14e3d\") " pod="openshift-monitoring/alertmanager-main-0"
Aug 05 17:15:28.456628 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:15:28.456275    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/cd891c78-e151-4947-9b17-8e6661e14e3d-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"cd891c78-e151-4947-9b17-8e6661e14e3d\") " pod="openshift-monitoring/alertmanager-main-0"
Aug 05 17:15:28.456628 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:15:28.456309    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cd891c78-e151-4947-9b17-8e6661e14e3d-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"cd891c78-e151-4947-9b17-8e6661e14e3d\") " pod="openshift-monitoring/alertmanager-main-0"
Aug 05 17:15:28.458162 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:15:28.458134    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/cd891c78-e151-4947-9b17-8e6661e14e3d-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"cd891c78-e151-4947-9b17-8e6661e14e3d\") " pod="openshift-monitoring/alertmanager-main-0"
Aug 05 17:15:28.458923 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:15:28.458901    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/cd891c78-e151-4947-9b17-8e6661e14e3d-config-volume\") pod \"alertmanager-main-0\" (UID: \"cd891c78-e151-4947-9b17-8e6661e14e3d\") " pod="openshift-monitoring/alertmanager-main-0"
Aug 05 17:15:28.459026 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:15:28.459005    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/cd891c78-e151-4947-9b17-8e6661e14e3d-tls-assets\") pod \"alertmanager-main-0\" (UID: \"cd891c78-e151-4947-9b17-8e6661e14e3d\") " pod="openshift-monitoring/alertmanager-main-0"
Aug 05 17:15:28.459103 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:15:28.459082    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/cd891c78-e151-4947-9b17-8e6661e14e3d-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"cd891c78-e151-4947-9b17-8e6661e14e3d\") " pod="openshift-monitoring/alertmanager-main-0"
Aug 05 17:15:28.459408 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:15:28.459365    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/cd891c78-e151-4947-9b17-8e6661e14e3d-web-config\") pod \"alertmanager-main-0\" (UID: \"cd891c78-e151-4947-9b17-8e6661e14e3d\") " pod="openshift-monitoring/alertmanager-main-0"
Aug 05 17:15:28.459815 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:15:28.459793    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/cd891c78-e151-4947-9b17-8e6661e14e3d-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"cd891c78-e151-4947-9b17-8e6661e14e3d\") " pod="openshift-monitoring/alertmanager-main-0"
Aug 05 17:15:28.460025 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:15:28.460005    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/cd891c78-e151-4947-9b17-8e6661e14e3d-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"cd891c78-e151-4947-9b17-8e6661e14e3d\") " pod="openshift-monitoring/alertmanager-main-0"
Aug 05 17:15:28.460184 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:15:28.460167    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/cd891c78-e151-4947-9b17-8e6661e14e3d-config-out\") pod \"alertmanager-main-0\" (UID: \"cd891c78-e151-4947-9b17-8e6661e14e3d\") " pod="openshift-monitoring/alertmanager-main-0"
Aug 05 17:15:28.460251 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:15:28.460174    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/cd891c78-e151-4947-9b17-8e6661e14e3d-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"cd891c78-e151-4947-9b17-8e6661e14e3d\") " pod="openshift-monitoring/alertmanager-main-0"
Aug 05 17:15:28.468272 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:15:28.468252    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q5mnj\" (UniqueName: \"kubernetes.io/projected/cd891c78-e151-4947-9b17-8e6661e14e3d-kube-api-access-q5mnj\") pod \"alertmanager-main-0\" (UID: \"cd891c78-e151-4947-9b17-8e6661e14e3d\") " pod="openshift-monitoring/alertmanager-main-0"
Aug 05 17:15:28.501142 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:15:28.501122    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Aug 05 17:15:28.648913 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:15:28.648871    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Aug 05 17:15:28.651619 ip-10-0-135-2 kubenswrapper[2571]: W0805 17:15:28.651597    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcd891c78_e151_4947_9b17_8e6661e14e3d.slice/crio-168197f2ab64ec98a2706756d33c0d8a5325ea53d0326cf978fa64c630e57f81 WatchSource:0}: Error finding container 168197f2ab64ec98a2706756d33c0d8a5325ea53d0326cf978fa64c630e57f81: Status 404 returned error can't find the container with id 168197f2ab64ec98a2706756d33c0d8a5325ea53d0326cf978fa64c630e57f81
Aug 05 17:15:28.669103 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:15:28.669075    2571 generic.go:356] "Generic (PLEG): container finished" podID="5de6dda3-4f32-4fbe-be23-c9fd0f0b9f8d" containerID="f537663fd95043bd8dbb65349e9c34747fe02198065dbc694045e9479dbadbd2" exitCode=0
Aug 05 17:15:28.669200 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:15:28.669160    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-6jh6w" event={"ID":"5de6dda3-4f32-4fbe-be23-c9fd0f0b9f8d","Type":"ContainerDied","Data":"f537663fd95043bd8dbb65349e9c34747fe02198065dbc694045e9479dbadbd2"}
Aug 05 17:15:28.670435 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:15:28.670411    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"cd891c78-e151-4947-9b17-8e6661e14e3d","Type":"ContainerStarted","Data":"168197f2ab64ec98a2706756d33c0d8a5325ea53d0326cf978fa64c630e57f81"}
Aug 05 17:15:29.146862 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:15:29.146831    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-756dfccb84-z56bd"]
Aug 05 17:15:29.150211 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:15:29.150188    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-756dfccb84-z56bd"
Aug 05 17:15:29.153587 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:15:29.153492    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\""
Aug 05 17:15:29.153587 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:15:29.153539    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-7rcbm\""
Aug 05 17:15:29.153759 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:15:29.153682    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\""
Aug 05 17:15:29.153814 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:15:29.153762    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\""
Aug 05 17:15:29.153814 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:15:29.153777    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\""
Aug 05 17:15:29.154218 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:15:29.154197    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\""
Aug 05 17:15:29.154339 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:15:29.154253    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-emcfrjsur1jg9\""
Aug 05 17:15:29.183362 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:15:29.183334    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-756dfccb84-z56bd"]
Aug 05 17:15:29.262100 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:15:29.262067    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/6c1987e0-88ac-427b-95d5-d728bff74e89-secret-thanos-querier-tls\") pod \"thanos-querier-756dfccb84-z56bd\" (UID: \"6c1987e0-88ac-427b-95d5-d728bff74e89\") " pod="openshift-monitoring/thanos-querier-756dfccb84-z56bd"
Aug 05 17:15:29.262538 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:15:29.262110    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/6c1987e0-88ac-427b-95d5-d728bff74e89-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-756dfccb84-z56bd\" (UID: \"6c1987e0-88ac-427b-95d5-d728bff74e89\") " pod="openshift-monitoring/thanos-querier-756dfccb84-z56bd"
Aug 05 17:15:29.262538 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:15:29.262147    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/6c1987e0-88ac-427b-95d5-d728bff74e89-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-756dfccb84-z56bd\" (UID: \"6c1987e0-88ac-427b-95d5-d728bff74e89\") " pod="openshift-monitoring/thanos-querier-756dfccb84-z56bd"
Aug 05 17:15:29.262538 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:15:29.262248    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/6c1987e0-88ac-427b-95d5-d728bff74e89-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-756dfccb84-z56bd\" (UID: \"6c1987e0-88ac-427b-95d5-d728bff74e89\") " pod="openshift-monitoring/thanos-querier-756dfccb84-z56bd"
Aug 05 17:15:29.262538 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:15:29.262297    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/6c1987e0-88ac-427b-95d5-d728bff74e89-secret-grpc-tls\") pod \"thanos-querier-756dfccb84-z56bd\" (UID: \"6c1987e0-88ac-427b-95d5-d728bff74e89\") " pod="openshift-monitoring/thanos-querier-756dfccb84-z56bd"
Aug 05 17:15:29.262538 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:15:29.262334    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fvbcd\" (UniqueName: \"kubernetes.io/projected/6c1987e0-88ac-427b-95d5-d728bff74e89-kube-api-access-fvbcd\") pod \"thanos-querier-756dfccb84-z56bd\" (UID: \"6c1987e0-88ac-427b-95d5-d728bff74e89\") " pod="openshift-monitoring/thanos-querier-756dfccb84-z56bd"
Aug 05 17:15:29.262538 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:15:29.262355    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/6c1987e0-88ac-427b-95d5-d728bff74e89-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-756dfccb84-z56bd\" (UID: \"6c1987e0-88ac-427b-95d5-d728bff74e89\") " pod="openshift-monitoring/thanos-querier-756dfccb84-z56bd"
Aug 05 17:15:29.262538 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:15:29.262430    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6c1987e0-88ac-427b-95d5-d728bff74e89-metrics-client-ca\") pod \"thanos-querier-756dfccb84-z56bd\" (UID: \"6c1987e0-88ac-427b-95d5-d728bff74e89\") " pod="openshift-monitoring/thanos-querier-756dfccb84-z56bd"
Aug 05 17:15:29.363447 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:15:29.363413    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/6c1987e0-88ac-427b-95d5-d728bff74e89-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-756dfccb84-z56bd\" (UID: \"6c1987e0-88ac-427b-95d5-d728bff74e89\") " pod="openshift-monitoring/thanos-querier-756dfccb84-z56bd"
Aug 05 17:15:29.363621 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:15:29.363592    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/6c1987e0-88ac-427b-95d5-d728bff74e89-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-756dfccb84-z56bd\" (UID: \"6c1987e0-88ac-427b-95d5-d728bff74e89\") " pod="openshift-monitoring/thanos-querier-756dfccb84-z56bd"
Aug 05 17:15:29.363693 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:15:29.363662    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/6c1987e0-88ac-427b-95d5-d728bff74e89-secret-grpc-tls\") pod \"thanos-querier-756dfccb84-z56bd\" (UID: \"6c1987e0-88ac-427b-95d5-d728bff74e89\") " pod="openshift-monitoring/thanos-querier-756dfccb84-z56bd"
Aug 05 17:15:29.363744 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:15:29.363699    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fvbcd\" (UniqueName: \"kubernetes.io/projected/6c1987e0-88ac-427b-95d5-d728bff74e89-kube-api-access-fvbcd\") pod \"thanos-querier-756dfccb84-z56bd\" (UID: \"6c1987e0-88ac-427b-95d5-d728bff74e89\") " pod="openshift-monitoring/thanos-querier-756dfccb84-z56bd"
Aug 05 17:15:29.363744 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:15:29.363732    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/6c1987e0-88ac-427b-95d5-d728bff74e89-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-756dfccb84-z56bd\" (UID: \"6c1987e0-88ac-427b-95d5-d728bff74e89\") " pod="openshift-monitoring/thanos-querier-756dfccb84-z56bd"
Aug 05 17:15:29.363849 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:15:29.363788    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6c1987e0-88ac-427b-95d5-d728bff74e89-metrics-client-ca\") pod \"thanos-querier-756dfccb84-z56bd\" (UID: \"6c1987e0-88ac-427b-95d5-d728bff74e89\") " pod="openshift-monitoring/thanos-querier-756dfccb84-z56bd"
Aug 05 17:15:29.363849 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:15:29.363837    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/6c1987e0-88ac-427b-95d5-d728bff74e89-secret-thanos-querier-tls\") pod \"thanos-querier-756dfccb84-z56bd\" (UID: \"6c1987e0-88ac-427b-95d5-d728bff74e89\") " pod="openshift-monitoring/thanos-querier-756dfccb84-z56bd"
Aug 05 17:15:29.363941 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:15:29.363867    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/6c1987e0-88ac-427b-95d5-d728bff74e89-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-756dfccb84-z56bd\" (UID: \"6c1987e0-88ac-427b-95d5-d728bff74e89\") " pod="openshift-monitoring/thanos-querier-756dfccb84-z56bd"
Aug 05 17:15:29.364680 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:15:29.364647    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6c1987e0-88ac-427b-95d5-d728bff74e89-metrics-client-ca\") pod \"thanos-querier-756dfccb84-z56bd\" (UID: \"6c1987e0-88ac-427b-95d5-d728bff74e89\") " pod="openshift-monitoring/thanos-querier-756dfccb84-z56bd"
Aug 05 17:15:29.366575 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:15:29.366533    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/6c1987e0-88ac-427b-95d5-d728bff74e89-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-756dfccb84-z56bd\" (UID: \"6c1987e0-88ac-427b-95d5-d728bff74e89\") " pod="openshift-monitoring/thanos-querier-756dfccb84-z56bd"
Aug 05 17:15:29.366739 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:15:29.366717    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/6c1987e0-88ac-427b-95d5-d728bff74e89-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-756dfccb84-z56bd\" (UID: \"6c1987e0-88ac-427b-95d5-d728bff74e89\") " pod="openshift-monitoring/thanos-querier-756dfccb84-z56bd"
Aug 05 17:15:29.366884 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:15:29.366866    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/6c1987e0-88ac-427b-95d5-d728bff74e89-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-756dfccb84-z56bd\" (UID: \"6c1987e0-88ac-427b-95d5-d728bff74e89\") " pod="openshift-monitoring/thanos-querier-756dfccb84-z56bd"
Aug 05 17:15:29.367165 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:15:29.367139    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/6c1987e0-88ac-427b-95d5-d728bff74e89-secret-thanos-querier-tls\") pod \"thanos-querier-756dfccb84-z56bd\" (UID: \"6c1987e0-88ac-427b-95d5-d728bff74e89\") " pod="openshift-monitoring/thanos-querier-756dfccb84-z56bd"
Aug 05 17:15:29.367350 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:15:29.367330    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/6c1987e0-88ac-427b-95d5-d728bff74e89-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-756dfccb84-z56bd\" (UID: \"6c1987e0-88ac-427b-95d5-d728bff74e89\") " pod="openshift-monitoring/thanos-querier-756dfccb84-z56bd"
Aug 05 17:15:29.367549 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:15:29.367526    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/6c1987e0-88ac-427b-95d5-d728bff74e89-secret-grpc-tls\") pod \"thanos-querier-756dfccb84-z56bd\" (UID: \"6c1987e0-88ac-427b-95d5-d728bff74e89\") " pod="openshift-monitoring/thanos-querier-756dfccb84-z56bd"
Aug 05 17:15:29.376958 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:15:29.376941    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fvbcd\" (UniqueName: \"kubernetes.io/projected/6c1987e0-88ac-427b-95d5-d728bff74e89-kube-api-access-fvbcd\") pod \"thanos-querier-756dfccb84-z56bd\" (UID: \"6c1987e0-88ac-427b-95d5-d728bff74e89\") " pod="openshift-monitoring/thanos-querier-756dfccb84-z56bd"
Aug 05 17:15:29.462022 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:15:29.461957    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-756dfccb84-z56bd"
Aug 05 17:15:29.470433 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:15:29.470410    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-547968ddbb-vtdlf"
Aug 05 17:15:29.678192 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:15:29.678156    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-6jh6w" event={"ID":"5de6dda3-4f32-4fbe-be23-c9fd0f0b9f8d","Type":"ContainerStarted","Data":"cece2d4ad80acffc0ea69f395b534df14a2c692367fc9f375a9564b3d5796348"}
Aug 05 17:15:29.678331 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:15:29.678216    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-6jh6w" event={"ID":"5de6dda3-4f32-4fbe-be23-c9fd0f0b9f8d","Type":"ContainerStarted","Data":"abb190d0fccbd79a04159849e9858f42f2276baea3a75a78184c566b99d17d56"}
Aug 05 17:15:29.721521 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:15:29.721479    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-6jh6w" podStartSLOduration=2.493430385 podStartE2EDuration="3.721464007s" totalImagesPullingTime="1.165019807s" totalInitContainerRuntime="63.013815ms" isStatefulPod=true podCreationTimestamp="2026-08-05 17:15:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-05 17:15:29.72089021 +0000 UTC m=+157.088256552" watchObservedRunningTime="2026-08-05 17:15:29.721464007 +0000 UTC m=+157.088830374"
Aug 05 17:15:29.772133 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:15:29.772011    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-756dfccb84-z56bd"]
Aug 05 17:15:29.827120 ip-10-0-135-2 kubenswrapper[2571]: W0805 17:15:29.827090    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6c1987e0_88ac_427b_95d5_d728bff74e89.slice/crio-397ef5f6c5783c3391bd4dfcadfd45dc68e3c6f1f8972bb537c6ca9208e1bb8c WatchSource:0}: Error finding container 397ef5f6c5783c3391bd4dfcadfd45dc68e3c6f1f8972bb537c6ca9208e1bb8c: Status 404 returned error can't find the container with id 397ef5f6c5783c3391bd4dfcadfd45dc68e3c6f1f8972bb537c6ca9208e1bb8c
Aug 05 17:15:30.683146 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:15:30.683106    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-756dfccb84-z56bd" event={"ID":"6c1987e0-88ac-427b-95d5-d728bff74e89","Type":"ContainerStarted","Data":"397ef5f6c5783c3391bd4dfcadfd45dc68e3c6f1f8972bb537c6ca9208e1bb8c"}
Aug 05 17:15:30.684935 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:15:30.684906    2571 generic.go:356] "Generic (PLEG): container finished" podID="cd891c78-e151-4947-9b17-8e6661e14e3d" containerID="54d1beff424951f03b3533bb884eb8ee8c5962958fd009ccc62c841e1e273a52" exitCode=0
Aug 05 17:15:30.685077 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:15:30.685001    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"cd891c78-e151-4947-9b17-8e6661e14e3d","Type":"ContainerDied","Data":"54d1beff424951f03b3533bb884eb8ee8c5962958fd009ccc62c841e1e273a52"}
Aug 05 17:15:31.023865 ip-10-0-135-2 kubenswrapper[2571]: E0805 17:15:31.023824    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-gn486" podUID="972f76e8-cfb8-4568-81e4-a90a0e5171d8"
Aug 05 17:15:31.688952 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:15:31.688900    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-6dffbbd67-7fzsj"]
Aug 05 17:15:31.689437 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:15:31.689047    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-gn486"
Aug 05 17:15:31.694568 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:15:31.694538    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-6dffbbd67-7fzsj"
Aug 05 17:15:31.697270 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:15:31.697248    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-mllhg\""
Aug 05 17:15:31.697419 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:15:31.697310    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\""
Aug 05 17:15:31.702705 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:15:31.702656    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-6dffbbd67-7fzsj"]
Aug 05 17:15:31.781440 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:15:31.781406    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8bf59\" (UniqueName: \"kubernetes.io/projected/a6075869-345c-4047-863a-ff43ba6ae645-kube-api-access-8bf59\") pod \"monitoring-plugin-6dffbbd67-7fzsj\" (UID: \"a6075869-345c-4047-863a-ff43ba6ae645\") " pod="openshift-monitoring/monitoring-plugin-6dffbbd67-7fzsj"
Aug 05 17:15:31.781593 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:15:31.781477    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/a6075869-345c-4047-863a-ff43ba6ae645-monitoring-plugin-cert\") pod \"monitoring-plugin-6dffbbd67-7fzsj\" (UID: \"a6075869-345c-4047-863a-ff43ba6ae645\") " pod="openshift-monitoring/monitoring-plugin-6dffbbd67-7fzsj"
Aug 05 17:15:31.882829 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:15:31.882784    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8bf59\" (UniqueName: \"kubernetes.io/projected/a6075869-345c-4047-863a-ff43ba6ae645-kube-api-access-8bf59\") pod \"monitoring-plugin-6dffbbd67-7fzsj\" (UID: \"a6075869-345c-4047-863a-ff43ba6ae645\") " pod="openshift-monitoring/monitoring-plugin-6dffbbd67-7fzsj"
Aug 05 17:15:31.882976 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:15:31.882852    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/a6075869-345c-4047-863a-ff43ba6ae645-monitoring-plugin-cert\") pod \"monitoring-plugin-6dffbbd67-7fzsj\" (UID: \"a6075869-345c-4047-863a-ff43ba6ae645\") " pod="openshift-monitoring/monitoring-plugin-6dffbbd67-7fzsj"
Aug 05 17:15:31.883053 ip-10-0-135-2 kubenswrapper[2571]: E0805 17:15:31.882987    2571 secret.go:189] Couldn't get secret openshift-monitoring/monitoring-plugin-cert: secret "monitoring-plugin-cert" not found
Aug 05 17:15:31.883106 ip-10-0-135-2 kubenswrapper[2571]: E0805 17:15:31.883066    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a6075869-345c-4047-863a-ff43ba6ae645-monitoring-plugin-cert podName:a6075869-345c-4047-863a-ff43ba6ae645 nodeName:}" failed. No retries permitted until 2026-08-05 17:15:32.383043687 +0000 UTC m=+159.750410012 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "monitoring-plugin-cert" (UniqueName: "kubernetes.io/secret/a6075869-345c-4047-863a-ff43ba6ae645-monitoring-plugin-cert") pod "monitoring-plugin-6dffbbd67-7fzsj" (UID: "a6075869-345c-4047-863a-ff43ba6ae645") : secret "monitoring-plugin-cert" not found
Aug 05 17:15:31.893039 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:15:31.893007    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8bf59\" (UniqueName: \"kubernetes.io/projected/a6075869-345c-4047-863a-ff43ba6ae645-kube-api-access-8bf59\") pod \"monitoring-plugin-6dffbbd67-7fzsj\" (UID: \"a6075869-345c-4047-863a-ff43ba6ae645\") " pod="openshift-monitoring/monitoring-plugin-6dffbbd67-7fzsj"
Aug 05 17:15:32.388155 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:15:32.388131    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/a6075869-345c-4047-863a-ff43ba6ae645-monitoring-plugin-cert\") pod \"monitoring-plugin-6dffbbd67-7fzsj\" (UID: \"a6075869-345c-4047-863a-ff43ba6ae645\") " pod="openshift-monitoring/monitoring-plugin-6dffbbd67-7fzsj"
Aug 05 17:15:32.388289 ip-10-0-135-2 kubenswrapper[2571]: E0805 17:15:32.388272    2571 secret.go:189] Couldn't get secret openshift-monitoring/monitoring-plugin-cert: secret "monitoring-plugin-cert" not found
Aug 05 17:15:32.388342 ip-10-0-135-2 kubenswrapper[2571]: E0805 17:15:32.388331    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a6075869-345c-4047-863a-ff43ba6ae645-monitoring-plugin-cert podName:a6075869-345c-4047-863a-ff43ba6ae645 nodeName:}" failed. No retries permitted until 2026-08-05 17:15:33.388314845 +0000 UTC m=+160.755681166 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "monitoring-plugin-cert" (UniqueName: "kubernetes.io/secret/a6075869-345c-4047-863a-ff43ba6ae645-monitoring-plugin-cert") pod "monitoring-plugin-6dffbbd67-7fzsj" (UID: "a6075869-345c-4047-863a-ff43ba6ae645") : secret "monitoring-plugin-cert" not found
Aug 05 17:15:32.695906 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:15:32.695880    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"cd891c78-e151-4947-9b17-8e6661e14e3d","Type":"ContainerStarted","Data":"12122ac4cb00528930f0b1dc451347eff33581b7b9778e10f74fe97e51bc84d0"}
Aug 05 17:15:32.696250 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:15:32.695965    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"cd891c78-e151-4947-9b17-8e6661e14e3d","Type":"ContainerStarted","Data":"5f2d32c522590c79d6328c038e4858a6aa15d85625718af9b73ce0d5151a9807"}
Aug 05 17:15:32.696250 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:15:32.696018    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"cd891c78-e151-4947-9b17-8e6661e14e3d","Type":"ContainerStarted","Data":"2c07e3a79698de281cdd0a56b1c7168d342b7937e17b04f5119352110d33a676"}
Aug 05 17:15:32.696250 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:15:32.696058    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"cd891c78-e151-4947-9b17-8e6661e14e3d","Type":"ContainerStarted","Data":"dfc1fe9f05269cf12ab75906467b12217226d984e37cb1f8f7ac761463138d0f"}
Aug 05 17:15:32.696250 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:15:32.696103    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"cd891c78-e151-4947-9b17-8e6661e14e3d","Type":"ContainerStarted","Data":"8f182ee1f6c240989276566eb44bcd62bfb446c0de87a0d49b4716766d161ea4"}
Aug 05 17:15:32.697948 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:15:32.697926    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-756dfccb84-z56bd" event={"ID":"6c1987e0-88ac-427b-95d5-d728bff74e89","Type":"ContainerStarted","Data":"5d89aee3029d39912ead5b125c44d05550c7d7c18a3238d0dff99ef94c934cef"}
Aug 05 17:15:32.698034 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:15:32.697985    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-756dfccb84-z56bd" event={"ID":"6c1987e0-88ac-427b-95d5-d728bff74e89","Type":"ContainerStarted","Data":"e9bd33eaac9aa1807e78599cae9042158f8f4dec5911ae07c7ad3806da99394e"}
Aug 05 17:15:32.698034 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:15:32.698012    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-756dfccb84-z56bd" event={"ID":"6c1987e0-88ac-427b-95d5-d728bff74e89","Type":"ContainerStarted","Data":"8a6d831a130bd307faf80e7bcd85d53bb23004625697ed8f91c7f4fec47ced19"}
Aug 05 17:15:33.395981 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:15:33.395955    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/a6075869-345c-4047-863a-ff43ba6ae645-monitoring-plugin-cert\") pod \"monitoring-plugin-6dffbbd67-7fzsj\" (UID: \"a6075869-345c-4047-863a-ff43ba6ae645\") " pod="openshift-monitoring/monitoring-plugin-6dffbbd67-7fzsj"
Aug 05 17:15:33.398031 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:15:33.398009    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/a6075869-345c-4047-863a-ff43ba6ae645-monitoring-plugin-cert\") pod \"monitoring-plugin-6dffbbd67-7fzsj\" (UID: \"a6075869-345c-4047-863a-ff43ba6ae645\") " pod="openshift-monitoring/monitoring-plugin-6dffbbd67-7fzsj"
Aug 05 17:15:33.506902 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:15:33.506840    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-6dffbbd67-7fzsj"
Aug 05 17:15:33.633757 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:15:33.633661    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-6dffbbd67-7fzsj"]
Aug 05 17:15:33.635983 ip-10-0-135-2 kubenswrapper[2571]: W0805 17:15:33.635956    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda6075869_345c_4047_863a_ff43ba6ae645.slice/crio-3089105548be7e0a68def0049db2547145fab5db1253aa2fad0c537ff367cdfb WatchSource:0}: Error finding container 3089105548be7e0a68def0049db2547145fab5db1253aa2fad0c537ff367cdfb: Status 404 returned error can't find the container with id 3089105548be7e0a68def0049db2547145fab5db1253aa2fad0c537ff367cdfb
Aug 05 17:15:33.708370 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:15:33.708339    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-756dfccb84-z56bd" event={"ID":"6c1987e0-88ac-427b-95d5-d728bff74e89","Type":"ContainerStarted","Data":"176fee7b35e3c5168d6da968a1b7a5b97d3e3146ef6ec142461f4b7b6e1e2e6c"}
Aug 05 17:15:33.708743 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:15:33.708469    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-756dfccb84-z56bd" event={"ID":"6c1987e0-88ac-427b-95d5-d728bff74e89","Type":"ContainerStarted","Data":"598a858c0e8646d70198d91400ee71cd97e0413908cd366abbecb1f9b78df786"}
Aug 05 17:15:33.708743 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:15:33.708525    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-756dfccb84-z56bd"
Aug 05 17:15:33.708743 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:15:33.708563    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-756dfccb84-z56bd" event={"ID":"6c1987e0-88ac-427b-95d5-d728bff74e89","Type":"ContainerStarted","Data":"dfca7c83e5d840c581045a811df1831a8a8452d0140a8566016b33199bd13ad3"}
Aug 05 17:15:33.709427 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:15:33.709405    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-6dffbbd67-7fzsj" event={"ID":"a6075869-345c-4047-863a-ff43ba6ae645","Type":"ContainerStarted","Data":"3089105548be7e0a68def0049db2547145fab5db1253aa2fad0c537ff367cdfb"}
Aug 05 17:15:33.712401 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:15:33.712359    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"cd891c78-e151-4947-9b17-8e6661e14e3d","Type":"ContainerStarted","Data":"07025a8abe4a96db80646075eb74a6869741bfd13f01b0ecd390cf030877af07"}
Aug 05 17:15:33.735101 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:15:33.735064    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-756dfccb84-z56bd" podStartSLOduration=1.508510438 podStartE2EDuration="4.735046949s" totalImagesPullingTime="3.226536511s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-05 17:15:29 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-05 17:15:33.734531245 +0000 UTC m=+161.101897587" watchObservedRunningTime="2026-08-05 17:15:33.735046949 +0000 UTC m=+161.102413291"
Aug 05 17:15:33.763262 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:15:33.763190    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=2.463466048 podStartE2EDuration="5.763176348s" totalImagesPullingTime="3.280765816s" totalInitContainerRuntime="18.944484ms" isStatefulPod=false podCreationTimestamp="2026-08-05 17:15:28 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-05 17:15:33.76233301 +0000 UTC m=+161.129699354" watchObservedRunningTime="2026-08-05 17:15:33.763176348 +0000 UTC m=+161.130542690"
Aug 05 17:15:34.484812 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:15:34.484772    2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-image-registry/image-registry-547968ddbb-vtdlf" podUID="af418777-9235-45af-bdeb-a837dcb58a14" containerName="registry" containerID="cri-o://a8ad44c34f2a7fcdd92bd986e1dfed4964f3c5ca82440605f3fed89678804524" gracePeriod=30
Aug 05 17:15:34.718869 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:15:34.718804    2571 generic.go:356] "Generic (PLEG): container finished" podID="af418777-9235-45af-bdeb-a837dcb58a14" containerID="a8ad44c34f2a7fcdd92bd986e1dfed4964f3c5ca82440605f3fed89678804524" exitCode=0
Aug 05 17:15:34.718869 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:15:34.718858    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-547968ddbb-vtdlf" event={"ID":"af418777-9235-45af-bdeb-a837dcb58a14","Type":"ContainerDied","Data":"a8ad44c34f2a7fcdd92bd986e1dfed4964f3c5ca82440605f3fed89678804524"}
Aug 05 17:15:34.978163 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:15:34.978142    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-547968ddbb-vtdlf"
Aug 05 17:15:35.009344 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:15:35.009319    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/af418777-9235-45af-bdeb-a837dcb58a14-bound-sa-token\") pod \"af418777-9235-45af-bdeb-a837dcb58a14\" (UID: \"af418777-9235-45af-bdeb-a837dcb58a14\") "
Aug 05 17:15:35.009486 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:15:35.009365    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/af418777-9235-45af-bdeb-a837dcb58a14-ca-trust-extracted\") pod \"af418777-9235-45af-bdeb-a837dcb58a14\" (UID: \"af418777-9235-45af-bdeb-a837dcb58a14\") "
Aug 05 17:15:35.009486 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:15:35.009447    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/af418777-9235-45af-bdeb-a837dcb58a14-installation-pull-secrets\") pod \"af418777-9235-45af-bdeb-a837dcb58a14\" (UID: \"af418777-9235-45af-bdeb-a837dcb58a14\") "
Aug 05 17:15:35.009571 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:15:35.009490    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/af418777-9235-45af-bdeb-a837dcb58a14-image-registry-private-configuration\") pod \"af418777-9235-45af-bdeb-a837dcb58a14\" (UID: \"af418777-9235-45af-bdeb-a837dcb58a14\") "
Aug 05 17:15:35.009571 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:15:35.009521    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/af418777-9235-45af-bdeb-a837dcb58a14-trusted-ca\") pod \"af418777-9235-45af-bdeb-a837dcb58a14\" (UID: \"af418777-9235-45af-bdeb-a837dcb58a14\") "
Aug 05 17:15:35.009571 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:15:35.009544    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/af418777-9235-45af-bdeb-a837dcb58a14-registry-certificates\") pod \"af418777-9235-45af-bdeb-a837dcb58a14\" (UID: \"af418777-9235-45af-bdeb-a837dcb58a14\") "
Aug 05 17:15:35.009571 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:15:35.009568    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/af418777-9235-45af-bdeb-a837dcb58a14-registry-tls\") pod \"af418777-9235-45af-bdeb-a837dcb58a14\" (UID: \"af418777-9235-45af-bdeb-a837dcb58a14\") "
Aug 05 17:15:35.009768 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:15:35.009744    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hqtn8\" (UniqueName: \"kubernetes.io/projected/af418777-9235-45af-bdeb-a837dcb58a14-kube-api-access-hqtn8\") pod \"af418777-9235-45af-bdeb-a837dcb58a14\" (UID: \"af418777-9235-45af-bdeb-a837dcb58a14\") "
Aug 05 17:15:35.010329 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:15:35.010283    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/af418777-9235-45af-bdeb-a837dcb58a14-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "af418777-9235-45af-bdeb-a837dcb58a14" (UID: "af418777-9235-45af-bdeb-a837dcb58a14"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 05 17:15:35.010454 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:15:35.010340    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/af418777-9235-45af-bdeb-a837dcb58a14-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "af418777-9235-45af-bdeb-a837dcb58a14" (UID: "af418777-9235-45af-bdeb-a837dcb58a14"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 05 17:15:35.012314 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:15:35.012283    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/af418777-9235-45af-bdeb-a837dcb58a14-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "af418777-9235-45af-bdeb-a837dcb58a14" (UID: "af418777-9235-45af-bdeb-a837dcb58a14"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 05 17:15:35.012469 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:15:35.012318    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/af418777-9235-45af-bdeb-a837dcb58a14-kube-api-access-hqtn8" (OuterVolumeSpecName: "kube-api-access-hqtn8") pod "af418777-9235-45af-bdeb-a837dcb58a14" (UID: "af418777-9235-45af-bdeb-a837dcb58a14"). InnerVolumeSpecName "kube-api-access-hqtn8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 05 17:15:35.012469 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:15:35.012396    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/af418777-9235-45af-bdeb-a837dcb58a14-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "af418777-9235-45af-bdeb-a837dcb58a14" (UID: "af418777-9235-45af-bdeb-a837dcb58a14"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 05 17:15:35.012469 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:15:35.012430    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/af418777-9235-45af-bdeb-a837dcb58a14-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "af418777-9235-45af-bdeb-a837dcb58a14" (UID: "af418777-9235-45af-bdeb-a837dcb58a14"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 05 17:15:35.012585 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:15:35.012515    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/af418777-9235-45af-bdeb-a837dcb58a14-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "af418777-9235-45af-bdeb-a837dcb58a14" (UID: "af418777-9235-45af-bdeb-a837dcb58a14"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 05 17:15:35.020172 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:15:35.020147    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/af418777-9235-45af-bdeb-a837dcb58a14-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "af418777-9235-45af-bdeb-a837dcb58a14" (UID: "af418777-9235-45af-bdeb-a837dcb58a14"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 05 17:15:35.110606 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:15:35.110558    2571 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/af418777-9235-45af-bdeb-a837dcb58a14-bound-sa-token\") on node \"ip-10-0-135-2.ec2.internal\" DevicePath \"\""
Aug 05 17:15:35.110606 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:15:35.110579    2571 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/af418777-9235-45af-bdeb-a837dcb58a14-ca-trust-extracted\") on node \"ip-10-0-135-2.ec2.internal\" DevicePath \"\""
Aug 05 17:15:35.110606 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:15:35.110589    2571 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/af418777-9235-45af-bdeb-a837dcb58a14-installation-pull-secrets\") on node \"ip-10-0-135-2.ec2.internal\" DevicePath \"\""
Aug 05 17:15:35.110606 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:15:35.110598    2571 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/af418777-9235-45af-bdeb-a837dcb58a14-image-registry-private-configuration\") on node \"ip-10-0-135-2.ec2.internal\" DevicePath \"\""
Aug 05 17:15:35.110606 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:15:35.110609    2571 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/af418777-9235-45af-bdeb-a837dcb58a14-trusted-ca\") on node \"ip-10-0-135-2.ec2.internal\" DevicePath \"\""
Aug 05 17:15:35.110806 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:15:35.110617    2571 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/af418777-9235-45af-bdeb-a837dcb58a14-registry-certificates\") on node \"ip-10-0-135-2.ec2.internal\" DevicePath \"\""
Aug 05 17:15:35.110806 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:15:35.110628    2571 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/af418777-9235-45af-bdeb-a837dcb58a14-registry-tls\") on node \"ip-10-0-135-2.ec2.internal\" DevicePath \"\""
Aug 05 17:15:35.110806 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:15:35.110636    2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hqtn8\" (UniqueName: \"kubernetes.io/projected/af418777-9235-45af-bdeb-a837dcb58a14-kube-api-access-hqtn8\") on node \"ip-10-0-135-2.ec2.internal\" DevicePath \"\""
Aug 05 17:15:35.723991 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:15:35.723960    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-6dffbbd67-7fzsj" event={"ID":"a6075869-345c-4047-863a-ff43ba6ae645","Type":"ContainerStarted","Data":"448a3cf76b8651176f08057dd074c498a3eb4a5ef8d54daeb6cd0b39de8452e7"}
Aug 05 17:15:35.724457 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:15:35.724099    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-6dffbbd67-7fzsj"
Aug 05 17:15:35.725429 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:15:35.725402    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-547968ddbb-vtdlf" event={"ID":"af418777-9235-45af-bdeb-a837dcb58a14","Type":"ContainerDied","Data":"f2ebe64b373d2382c9709f10a35aa76106956e9c05f6cdcc6d65c1b97aa48ce8"}
Aug 05 17:15:35.725565 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:15:35.725452    2571 scope.go:117] "RemoveContainer" containerID="a8ad44c34f2a7fcdd92bd986e1dfed4964f3c5ca82440605f3fed89678804524"
Aug 05 17:15:35.725565 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:15:35.725408    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-547968ddbb-vtdlf"
Aug 05 17:15:35.729650 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:15:35.729628    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-6dffbbd67-7fzsj"
Aug 05 17:15:35.745266 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:15:35.745227    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-6dffbbd67-7fzsj" podStartSLOduration=3.497940687 podStartE2EDuration="4.745213286s" totalImagesPullingTime="1.247272599s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-05 17:15:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-05 17:15:35.744710118 +0000 UTC m=+163.112076465" watchObservedRunningTime="2026-08-05 17:15:35.745213286 +0000 UTC m=+163.112579628"
Aug 05 17:15:35.778747 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:15:35.778722    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-547968ddbb-vtdlf"]
Aug 05 17:15:35.785241 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:15:35.785220    2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-547968ddbb-vtdlf"]
Aug 05 17:15:35.916833 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:15:35.916799    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/972f76e8-cfb8-4568-81e4-a90a0e5171d8-metrics-tls\") pod \"dns-default-gn486\" (UID: \"972f76e8-cfb8-4568-81e4-a90a0e5171d8\") " pod="openshift-dns/dns-default-gn486"
Aug 05 17:15:35.919094 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:15:35.919073    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/972f76e8-cfb8-4568-81e4-a90a0e5171d8-metrics-tls\") pod \"dns-default-gn486\" (UID: \"972f76e8-cfb8-4568-81e4-a90a0e5171d8\") " pod="openshift-dns/dns-default-gn486"
Aug 05 17:15:36.193483 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:15:36.193455    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-jxp5d\""
Aug 05 17:15:36.201159 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:15:36.201131    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-gn486"
Aug 05 17:15:36.331675 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:15:36.331644    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-gn486"]
Aug 05 17:15:36.336143 ip-10-0-135-2 kubenswrapper[2571]: W0805 17:15:36.336116    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod972f76e8_cfb8_4568_81e4_a90a0e5171d8.slice/crio-a3d057acca94e557063c72374569ae4bafe80914dbc408b3b387d5d2726f6393 WatchSource:0}: Error finding container a3d057acca94e557063c72374569ae4bafe80914dbc408b3b387d5d2726f6393: Status 404 returned error can't find the container with id a3d057acca94e557063c72374569ae4bafe80914dbc408b3b387d5d2726f6393
Aug 05 17:15:36.730200 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:15:36.730166    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-gn486" event={"ID":"972f76e8-cfb8-4568-81e4-a90a0e5171d8","Type":"ContainerStarted","Data":"a3d057acca94e557063c72374569ae4bafe80914dbc408b3b387d5d2726f6393"}
Aug 05 17:15:37.012102 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:15:37.012026    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-77c974ffcf-6zshd"]
Aug 05 17:15:37.012323 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:15:37.012310    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="af418777-9235-45af-bdeb-a837dcb58a14" containerName="registry"
Aug 05 17:15:37.012401 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:15:37.012328    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="af418777-9235-45af-bdeb-a837dcb58a14" containerName="registry"
Aug 05 17:15:37.012444 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:15:37.012400    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="af418777-9235-45af-bdeb-a837dcb58a14" containerName="registry"
Aug 05 17:15:37.015268 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:15:37.015245    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-77c974ffcf-6zshd"
Aug 05 17:15:37.017736 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:15:37.017708    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"default-dockercfg-zxv8q\""
Aug 05 17:15:37.017736 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:15:37.017732    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Aug 05 17:15:37.017934 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:15:37.017837    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Aug 05 17:15:37.028606 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:15:37.028584    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-77c974ffcf-6zshd"]
Aug 05 17:15:37.126352 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:15:37.126324    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gnmzd\" (UniqueName: \"kubernetes.io/projected/23b66e37-d633-4bed-a6dc-357d8bbb585e-kube-api-access-gnmzd\") pod \"downloads-77c974ffcf-6zshd\" (UID: \"23b66e37-d633-4bed-a6dc-357d8bbb585e\") " pod="openshift-console/downloads-77c974ffcf-6zshd"
Aug 05 17:15:37.136653 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:15:37.136623    2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="af418777-9235-45af-bdeb-a837dcb58a14" path="/var/lib/kubelet/pods/af418777-9235-45af-bdeb-a837dcb58a14/volumes"
Aug 05 17:15:37.226913 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:15:37.226871    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gnmzd\" (UniqueName: \"kubernetes.io/projected/23b66e37-d633-4bed-a6dc-357d8bbb585e-kube-api-access-gnmzd\") pod \"downloads-77c974ffcf-6zshd\" (UID: \"23b66e37-d633-4bed-a6dc-357d8bbb585e\") " pod="openshift-console/downloads-77c974ffcf-6zshd"
Aug 05 17:15:37.243882 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:15:37.243855    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gnmzd\" (UniqueName: \"kubernetes.io/projected/23b66e37-d633-4bed-a6dc-357d8bbb585e-kube-api-access-gnmzd\") pod \"downloads-77c974ffcf-6zshd\" (UID: \"23b66e37-d633-4bed-a6dc-357d8bbb585e\") " pod="openshift-console/downloads-77c974ffcf-6zshd"
Aug 05 17:15:37.326799 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:15:37.326726    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-77c974ffcf-6zshd"
Aug 05 17:15:37.486298 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:15:37.486240    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-77c974ffcf-6zshd"]
Aug 05 17:15:37.931677 ip-10-0-135-2 kubenswrapper[2571]: W0805 17:15:37.931645    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod23b66e37_d633_4bed_a6dc_357d8bbb585e.slice/crio-3e8e530394171a67104a3ab9da9dc13ccb14b6b1e1e7c76a137aa3c8e205d1d3 WatchSource:0}: Error finding container 3e8e530394171a67104a3ab9da9dc13ccb14b6b1e1e7c76a137aa3c8e205d1d3: Status 404 returned error can't find the container with id 3e8e530394171a67104a3ab9da9dc13ccb14b6b1e1e7c76a137aa3c8e205d1d3
Aug 05 17:15:38.740217 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:15:38.740181    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-77c974ffcf-6zshd" event={"ID":"23b66e37-d633-4bed-a6dc-357d8bbb585e","Type":"ContainerStarted","Data":"3e8e530394171a67104a3ab9da9dc13ccb14b6b1e1e7c76a137aa3c8e205d1d3"}
Aug 05 17:15:38.742649 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:15:38.742625    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-gn486" event={"ID":"972f76e8-cfb8-4568-81e4-a90a0e5171d8","Type":"ContainerStarted","Data":"0f36e75afb2b1494c2447d5e7ef3a55964d7e78c3b2baec177b1c555812c501d"}
Aug 05 17:15:38.742780 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:15:38.742672    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-gn486" event={"ID":"972f76e8-cfb8-4568-81e4-a90a0e5171d8","Type":"ContainerStarted","Data":"5fcef55ae2afaa4498cb4ab397c7004badc09cc9f485ed3860095db77be4096a"}
Aug 05 17:15:38.742832 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:15:38.742787    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-gn486"
Aug 05 17:15:38.768830 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:15:38.768779    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-gn486" podStartSLOduration=130.128777249 podStartE2EDuration="2m11.768762614s" totalImagesPullingTime="1.639985365s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-05 17:13:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-05 17:15:38.768756924 +0000 UTC m=+166.136123268" watchObservedRunningTime="2026-08-05 17:15:38.768762614 +0000 UTC m=+166.136128957"
Aug 05 17:15:39.726197 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:15:39.726165    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-756dfccb84-z56bd"
Aug 05 17:15:47.163662 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:15:47.163587    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-6bb977d6dd-227dh"]
Aug 05 17:15:47.167115 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:15:47.167096    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6bb977d6dd-227dh"
Aug 05 17:15:47.171008 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:15:47.170984    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Aug 05 17:15:47.171265 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:15:47.171209    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Aug 05 17:15:47.171370 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:15:47.171280    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Aug 05 17:15:47.171370 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:15:47.171298    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Aug 05 17:15:47.171742 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:15:47.171725    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-5mvd4\""
Aug 05 17:15:47.174892 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:15:47.174870    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Aug 05 17:15:47.181506 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:15:47.181482    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6bb977d6dd-227dh"]
Aug 05 17:15:47.210840 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:15:47.210815    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/2ff1f472-19b1-433f-ba0a-91def41dcaca-oauth-serving-cert\") pod \"console-6bb977d6dd-227dh\" (UID: \"2ff1f472-19b1-433f-ba0a-91def41dcaca\") " pod="openshift-console/console-6bb977d6dd-227dh"
Aug 05 17:15:47.210930 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:15:47.210859    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/2ff1f472-19b1-433f-ba0a-91def41dcaca-console-oauth-config\") pod \"console-6bb977d6dd-227dh\" (UID: \"2ff1f472-19b1-433f-ba0a-91def41dcaca\") " pod="openshift-console/console-6bb977d6dd-227dh"
Aug 05 17:15:47.210930 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:15:47.210886    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/2ff1f472-19b1-433f-ba0a-91def41dcaca-console-serving-cert\") pod \"console-6bb977d6dd-227dh\" (UID: \"2ff1f472-19b1-433f-ba0a-91def41dcaca\") " pod="openshift-console/console-6bb977d6dd-227dh"
Aug 05 17:15:47.211032 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:15:47.210934    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/2ff1f472-19b1-433f-ba0a-91def41dcaca-service-ca\") pod \"console-6bb977d6dd-227dh\" (UID: \"2ff1f472-19b1-433f-ba0a-91def41dcaca\") " pod="openshift-console/console-6bb977d6dd-227dh"
Aug 05 17:15:47.211032 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:15:47.210971    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/2ff1f472-19b1-433f-ba0a-91def41dcaca-console-config\") pod \"console-6bb977d6dd-227dh\" (UID: \"2ff1f472-19b1-433f-ba0a-91def41dcaca\") " pod="openshift-console/console-6bb977d6dd-227dh"
Aug 05 17:15:47.211122 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:15:47.211079    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k2fmm\" (UniqueName: \"kubernetes.io/projected/2ff1f472-19b1-433f-ba0a-91def41dcaca-kube-api-access-k2fmm\") pod \"console-6bb977d6dd-227dh\" (UID: \"2ff1f472-19b1-433f-ba0a-91def41dcaca\") " pod="openshift-console/console-6bb977d6dd-227dh"
Aug 05 17:15:47.312521 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:15:47.312495    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/2ff1f472-19b1-433f-ba0a-91def41dcaca-service-ca\") pod \"console-6bb977d6dd-227dh\" (UID: \"2ff1f472-19b1-433f-ba0a-91def41dcaca\") " pod="openshift-console/console-6bb977d6dd-227dh"
Aug 05 17:15:47.312641 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:15:47.312530    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/2ff1f472-19b1-433f-ba0a-91def41dcaca-console-config\") pod \"console-6bb977d6dd-227dh\" (UID: \"2ff1f472-19b1-433f-ba0a-91def41dcaca\") " pod="openshift-console/console-6bb977d6dd-227dh"
Aug 05 17:15:47.312641 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:15:47.312551    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k2fmm\" (UniqueName: \"kubernetes.io/projected/2ff1f472-19b1-433f-ba0a-91def41dcaca-kube-api-access-k2fmm\") pod \"console-6bb977d6dd-227dh\" (UID: \"2ff1f472-19b1-433f-ba0a-91def41dcaca\") " pod="openshift-console/console-6bb977d6dd-227dh"
Aug 05 17:15:47.312641 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:15:47.312601    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/2ff1f472-19b1-433f-ba0a-91def41dcaca-oauth-serving-cert\") pod \"console-6bb977d6dd-227dh\" (UID: \"2ff1f472-19b1-433f-ba0a-91def41dcaca\") " pod="openshift-console/console-6bb977d6dd-227dh"
Aug 05 17:15:47.312802 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:15:47.312646    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/2ff1f472-19b1-433f-ba0a-91def41dcaca-console-oauth-config\") pod \"console-6bb977d6dd-227dh\" (UID: \"2ff1f472-19b1-433f-ba0a-91def41dcaca\") " pod="openshift-console/console-6bb977d6dd-227dh"
Aug 05 17:15:47.312802 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:15:47.312665    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/2ff1f472-19b1-433f-ba0a-91def41dcaca-console-serving-cert\") pod \"console-6bb977d6dd-227dh\" (UID: \"2ff1f472-19b1-433f-ba0a-91def41dcaca\") " pod="openshift-console/console-6bb977d6dd-227dh"
Aug 05 17:15:47.313304 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:15:47.313271    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/2ff1f472-19b1-433f-ba0a-91def41dcaca-service-ca\") pod \"console-6bb977d6dd-227dh\" (UID: \"2ff1f472-19b1-433f-ba0a-91def41dcaca\") " pod="openshift-console/console-6bb977d6dd-227dh"
Aug 05 17:15:47.313445 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:15:47.313411    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/2ff1f472-19b1-433f-ba0a-91def41dcaca-console-config\") pod \"console-6bb977d6dd-227dh\" (UID: \"2ff1f472-19b1-433f-ba0a-91def41dcaca\") " pod="openshift-console/console-6bb977d6dd-227dh"
Aug 05 17:15:47.313518 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:15:47.313464    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/2ff1f472-19b1-433f-ba0a-91def41dcaca-oauth-serving-cert\") pod \"console-6bb977d6dd-227dh\" (UID: \"2ff1f472-19b1-433f-ba0a-91def41dcaca\") " pod="openshift-console/console-6bb977d6dd-227dh"
Aug 05 17:15:47.315354 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:15:47.315330    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/2ff1f472-19b1-433f-ba0a-91def41dcaca-console-oauth-config\") pod \"console-6bb977d6dd-227dh\" (UID: \"2ff1f472-19b1-433f-ba0a-91def41dcaca\") " pod="openshift-console/console-6bb977d6dd-227dh"
Aug 05 17:15:47.315505 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:15:47.315489    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/2ff1f472-19b1-433f-ba0a-91def41dcaca-console-serving-cert\") pod \"console-6bb977d6dd-227dh\" (UID: \"2ff1f472-19b1-433f-ba0a-91def41dcaca\") " pod="openshift-console/console-6bb977d6dd-227dh"
Aug 05 17:15:47.321356 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:15:47.321332    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k2fmm\" (UniqueName: \"kubernetes.io/projected/2ff1f472-19b1-433f-ba0a-91def41dcaca-kube-api-access-k2fmm\") pod \"console-6bb977d6dd-227dh\" (UID: \"2ff1f472-19b1-433f-ba0a-91def41dcaca\") " pod="openshift-console/console-6bb977d6dd-227dh"
Aug 05 17:15:47.478386 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:15:47.478336    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6bb977d6dd-227dh"
Aug 05 17:15:48.750197 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:15:48.750165    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-gn486"
Aug 05 17:15:54.051621 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:15:54.051597    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6bb977d6dd-227dh"]
Aug 05 17:15:54.055415 ip-10-0-135-2 kubenswrapper[2571]: W0805 17:15:54.055385    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2ff1f472_19b1_433f_ba0a_91def41dcaca.slice/crio-e2ffec397ff596cb2ab74988d0168dbb7db0af9e95b03409790420cbeb9e1912 WatchSource:0}: Error finding container e2ffec397ff596cb2ab74988d0168dbb7db0af9e95b03409790420cbeb9e1912: Status 404 returned error can't find the container with id e2ffec397ff596cb2ab74988d0168dbb7db0af9e95b03409790420cbeb9e1912
Aug 05 17:15:54.813418 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:15:54.813356    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-77c974ffcf-6zshd" event={"ID":"23b66e37-d633-4bed-a6dc-357d8bbb585e","Type":"ContainerStarted","Data":"6451391e9171ca2c70c78a52b6090d8c43acbf113f603a4d68136c2a78d4c844"}
Aug 05 17:15:54.813921 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:15:54.813890    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-77c974ffcf-6zshd"
Aug 05 17:15:54.815986 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:15:54.815957    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6bb977d6dd-227dh" event={"ID":"2ff1f472-19b1-433f-ba0a-91def41dcaca","Type":"ContainerStarted","Data":"e2ffec397ff596cb2ab74988d0168dbb7db0af9e95b03409790420cbeb9e1912"}
Aug 05 17:15:54.824259 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:15:54.824236    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-77c974ffcf-6zshd"
Aug 05 17:15:54.836821 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:15:54.836692    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/downloads-77c974ffcf-6zshd" podStartSLOduration=2.74492988 podStartE2EDuration="18.836674974s" totalImagesPullingTime="16.091745094s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-05 17:15:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-05 17:15:54.835037396 +0000 UTC m=+182.202403741" watchObservedRunningTime="2026-08-05 17:15:54.836674974 +0000 UTC m=+182.204041318"
Aug 05 17:15:55.747693 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:15:55.747660    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-5d799cddfd-fhl99"]
Aug 05 17:15:55.754478 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:15:55.754420    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5d799cddfd-fhl99"
Aug 05 17:15:55.777743 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:15:55.777709    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5d799cddfd-fhl99"]
Aug 05 17:15:55.778146 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:15:55.778121    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\""
Aug 05 17:15:55.885948 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:15:55.885722    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/736198e4-bb0b-45a3-8500-050c572dded4-service-ca\") pod \"console-5d799cddfd-fhl99\" (UID: \"736198e4-bb0b-45a3-8500-050c572dded4\") " pod="openshift-console/console-5d799cddfd-fhl99"
Aug 05 17:15:55.885948 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:15:55.885806    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/736198e4-bb0b-45a3-8500-050c572dded4-oauth-serving-cert\") pod \"console-5d799cddfd-fhl99\" (UID: \"736198e4-bb0b-45a3-8500-050c572dded4\") " pod="openshift-console/console-5d799cddfd-fhl99"
Aug 05 17:15:55.885948 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:15:55.885863    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/736198e4-bb0b-45a3-8500-050c572dded4-trusted-ca-bundle\") pod \"console-5d799cddfd-fhl99\" (UID: \"736198e4-bb0b-45a3-8500-050c572dded4\") " pod="openshift-console/console-5d799cddfd-fhl99"
Aug 05 17:15:55.885948 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:15:55.885888    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/736198e4-bb0b-45a3-8500-050c572dded4-console-config\") pod \"console-5d799cddfd-fhl99\" (UID: \"736198e4-bb0b-45a3-8500-050c572dded4\") " pod="openshift-console/console-5d799cddfd-fhl99"
Aug 05 17:15:55.886772 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:15:55.886035    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dv72k\" (UniqueName: \"kubernetes.io/projected/736198e4-bb0b-45a3-8500-050c572dded4-kube-api-access-dv72k\") pod \"console-5d799cddfd-fhl99\" (UID: \"736198e4-bb0b-45a3-8500-050c572dded4\") " pod="openshift-console/console-5d799cddfd-fhl99"
Aug 05 17:15:55.886772 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:15:55.886076    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/736198e4-bb0b-45a3-8500-050c572dded4-console-serving-cert\") pod \"console-5d799cddfd-fhl99\" (UID: \"736198e4-bb0b-45a3-8500-050c572dded4\") " pod="openshift-console/console-5d799cddfd-fhl99"
Aug 05 17:15:55.886772 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:15:55.886112    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/736198e4-bb0b-45a3-8500-050c572dded4-console-oauth-config\") pod \"console-5d799cddfd-fhl99\" (UID: \"736198e4-bb0b-45a3-8500-050c572dded4\") " pod="openshift-console/console-5d799cddfd-fhl99"
Aug 05 17:15:55.987515 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:15:55.987480    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/736198e4-bb0b-45a3-8500-050c572dded4-oauth-serving-cert\") pod \"console-5d799cddfd-fhl99\" (UID: \"736198e4-bb0b-45a3-8500-050c572dded4\") " pod="openshift-console/console-5d799cddfd-fhl99"
Aug 05 17:15:55.987697 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:15:55.987541    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/736198e4-bb0b-45a3-8500-050c572dded4-trusted-ca-bundle\") pod \"console-5d799cddfd-fhl99\" (UID: \"736198e4-bb0b-45a3-8500-050c572dded4\") " pod="openshift-console/console-5d799cddfd-fhl99"
Aug 05 17:15:55.987697 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:15:55.987570    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/736198e4-bb0b-45a3-8500-050c572dded4-console-config\") pod \"console-5d799cddfd-fhl99\" (UID: \"736198e4-bb0b-45a3-8500-050c572dded4\") " pod="openshift-console/console-5d799cddfd-fhl99"
Aug 05 17:15:55.987697 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:15:55.987606    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dv72k\" (UniqueName: \"kubernetes.io/projected/736198e4-bb0b-45a3-8500-050c572dded4-kube-api-access-dv72k\") pod \"console-5d799cddfd-fhl99\" (UID: \"736198e4-bb0b-45a3-8500-050c572dded4\") " pod="openshift-console/console-5d799cddfd-fhl99"
Aug 05 17:15:55.987697 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:15:55.987635    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/736198e4-bb0b-45a3-8500-050c572dded4-console-serving-cert\") pod \"console-5d799cddfd-fhl99\" (UID: \"736198e4-bb0b-45a3-8500-050c572dded4\") " pod="openshift-console/console-5d799cddfd-fhl99"
Aug 05 17:15:55.987697 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:15:55.987665    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/736198e4-bb0b-45a3-8500-050c572dded4-console-oauth-config\") pod \"console-5d799cddfd-fhl99\" (UID: \"736198e4-bb0b-45a3-8500-050c572dded4\") " pod="openshift-console/console-5d799cddfd-fhl99"
Aug 05 17:15:55.988155 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:15:55.988056    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/736198e4-bb0b-45a3-8500-050c572dded4-service-ca\") pod \"console-5d799cddfd-fhl99\" (UID: \"736198e4-bb0b-45a3-8500-050c572dded4\") " pod="openshift-console/console-5d799cddfd-fhl99"
Aug 05 17:15:55.988314 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:15:55.988286    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/736198e4-bb0b-45a3-8500-050c572dded4-oauth-serving-cert\") pod \"console-5d799cddfd-fhl99\" (UID: \"736198e4-bb0b-45a3-8500-050c572dded4\") " pod="openshift-console/console-5d799cddfd-fhl99"
Aug 05 17:15:55.988893 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:15:55.988642    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/736198e4-bb0b-45a3-8500-050c572dded4-console-config\") pod \"console-5d799cddfd-fhl99\" (UID: \"736198e4-bb0b-45a3-8500-050c572dded4\") " pod="openshift-console/console-5d799cddfd-fhl99"
Aug 05 17:15:55.988893 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:15:55.988750    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/736198e4-bb0b-45a3-8500-050c572dded4-trusted-ca-bundle\") pod \"console-5d799cddfd-fhl99\" (UID: \"736198e4-bb0b-45a3-8500-050c572dded4\") " pod="openshift-console/console-5d799cddfd-fhl99"
Aug 05 17:15:55.989351 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:15:55.989187    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/736198e4-bb0b-45a3-8500-050c572dded4-service-ca\") pod \"console-5d799cddfd-fhl99\" (UID: \"736198e4-bb0b-45a3-8500-050c572dded4\") " pod="openshift-console/console-5d799cddfd-fhl99"
Aug 05 17:15:55.991638 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:15:55.991609    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/736198e4-bb0b-45a3-8500-050c572dded4-console-serving-cert\") pod \"console-5d799cddfd-fhl99\" (UID: \"736198e4-bb0b-45a3-8500-050c572dded4\") " pod="openshift-console/console-5d799cddfd-fhl99"
Aug 05 17:15:55.991959 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:15:55.991939    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/736198e4-bb0b-45a3-8500-050c572dded4-console-oauth-config\") pod \"console-5d799cddfd-fhl99\" (UID: \"736198e4-bb0b-45a3-8500-050c572dded4\") " pod="openshift-console/console-5d799cddfd-fhl99"
Aug 05 17:15:55.996593 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:15:55.996570    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dv72k\" (UniqueName: \"kubernetes.io/projected/736198e4-bb0b-45a3-8500-050c572dded4-kube-api-access-dv72k\") pod \"console-5d799cddfd-fhl99\" (UID: \"736198e4-bb0b-45a3-8500-050c572dded4\") " pod="openshift-console/console-5d799cddfd-fhl99"
Aug 05 17:15:56.077623 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:15:56.077541    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5d799cddfd-fhl99"
Aug 05 17:15:57.558051 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:15:57.558019    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5d799cddfd-fhl99"]
Aug 05 17:15:57.562236 ip-10-0-135-2 kubenswrapper[2571]: W0805 17:15:57.562208    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod736198e4_bb0b_45a3_8500_050c572dded4.slice/crio-b5ef9bff55dd19d8703b4dfce4f7071a0e985c2562630047d468e21c8f6ec386 WatchSource:0}: Error finding container b5ef9bff55dd19d8703b4dfce4f7071a0e985c2562630047d468e21c8f6ec386: Status 404 returned error can't find the container with id b5ef9bff55dd19d8703b4dfce4f7071a0e985c2562630047d468e21c8f6ec386
Aug 05 17:15:57.831445 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:15:57.831413    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5d799cddfd-fhl99" event={"ID":"736198e4-bb0b-45a3-8500-050c572dded4","Type":"ContainerStarted","Data":"b5ef9bff55dd19d8703b4dfce4f7071a0e985c2562630047d468e21c8f6ec386"}
Aug 05 17:15:58.837083 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:15:58.837045    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6bb977d6dd-227dh" event={"ID":"2ff1f472-19b1-433f-ba0a-91def41dcaca","Type":"ContainerStarted","Data":"7b0b49ad0a12256746ec9e06d38fbe284a1228cf286fb2902ed42c682f651782"}
Aug 05 17:15:58.838942 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:15:58.838910    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5d799cddfd-fhl99" event={"ID":"736198e4-bb0b-45a3-8500-050c572dded4","Type":"ContainerStarted","Data":"8e67af0f3aaa01205ac5d6474b9a86a34a6bdd6a931039a07832e0715be94e2b"}
Aug 05 17:15:58.856926 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:15:58.856880    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-6bb977d6dd-227dh" podStartSLOduration=8.071167195 podStartE2EDuration="11.856863802s" totalImagesPullingTime="3.785696607s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-05 17:15:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-05 17:15:58.854996693 +0000 UTC m=+186.222363055" watchObservedRunningTime="2026-08-05 17:15:58.856863802 +0000 UTC m=+186.224230148"
Aug 05 17:15:58.874751 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:15:58.874704    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-5d799cddfd-fhl99" podStartSLOduration=3.215540902 podStartE2EDuration="3.874689235s" totalImagesPullingTime="659.148333ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-05 17:15:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-05 17:15:58.873465766 +0000 UTC m=+186.240832110" watchObservedRunningTime="2026-08-05 17:15:58.874689235 +0000 UTC m=+186.242055645"
Aug 05 17:16:06.078461 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:16:06.078424    2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-5d799cddfd-fhl99"
Aug 05 17:16:06.078931 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:16:06.078486    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-5d799cddfd-fhl99"
Aug 05 17:16:06.083099 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:16:06.083076    2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-5d799cddfd-fhl99"
Aug 05 17:16:06.874968 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:16:06.874940    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-5d799cddfd-fhl99"
Aug 05 17:16:06.931097 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:16:06.931067    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6bb977d6dd-227dh"]
Aug 05 17:16:07.479009 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:16:07.478978    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-6bb977d6dd-227dh"
Aug 05 17:16:15.656766 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:16:15.656723    2571 prober.go:120] "Probe failed" probeType="Liveness" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7b74c45fdd-9tcbw" podUID="83dbc4d1-49f4-4f07-97d2-80443a2c873f" containerName="service-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 500"
Aug 05 17:16:25.656646 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:16:25.656606    2571 prober.go:120] "Probe failed" probeType="Liveness" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7b74c45fdd-9tcbw" podUID="83dbc4d1-49f4-4f07-97d2-80443a2c873f" containerName="service-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 500"
Aug 05 17:16:31.949576 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:16:31.949530    2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-6bb977d6dd-227dh" podUID="2ff1f472-19b1-433f-ba0a-91def41dcaca" containerName="console" containerID="cri-o://7b0b49ad0a12256746ec9e06d38fbe284a1228cf286fb2902ed42c682f651782" gracePeriod=15
Aug 05 17:16:32.223839 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:16:32.223813    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6bb977d6dd-227dh"
Aug 05 17:16:32.282361 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:16:32.282331    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/2ff1f472-19b1-433f-ba0a-91def41dcaca-console-serving-cert\") pod \"2ff1f472-19b1-433f-ba0a-91def41dcaca\" (UID: \"2ff1f472-19b1-433f-ba0a-91def41dcaca\") "
Aug 05 17:16:32.282361 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:16:32.282370    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-k2fmm\" (UniqueName: \"kubernetes.io/projected/2ff1f472-19b1-433f-ba0a-91def41dcaca-kube-api-access-k2fmm\") pod \"2ff1f472-19b1-433f-ba0a-91def41dcaca\" (UID: \"2ff1f472-19b1-433f-ba0a-91def41dcaca\") "
Aug 05 17:16:32.282589 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:16:32.282409    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/2ff1f472-19b1-433f-ba0a-91def41dcaca-console-oauth-config\") pod \"2ff1f472-19b1-433f-ba0a-91def41dcaca\" (UID: \"2ff1f472-19b1-433f-ba0a-91def41dcaca\") "
Aug 05 17:16:32.282589 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:16:32.282430    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/2ff1f472-19b1-433f-ba0a-91def41dcaca-oauth-serving-cert\") pod \"2ff1f472-19b1-433f-ba0a-91def41dcaca\" (UID: \"2ff1f472-19b1-433f-ba0a-91def41dcaca\") "
Aug 05 17:16:32.282589 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:16:32.282475    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/2ff1f472-19b1-433f-ba0a-91def41dcaca-console-config\") pod \"2ff1f472-19b1-433f-ba0a-91def41dcaca\" (UID: \"2ff1f472-19b1-433f-ba0a-91def41dcaca\") "
Aug 05 17:16:32.282589 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:16:32.282526    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/2ff1f472-19b1-433f-ba0a-91def41dcaca-service-ca\") pod \"2ff1f472-19b1-433f-ba0a-91def41dcaca\" (UID: \"2ff1f472-19b1-433f-ba0a-91def41dcaca\") "
Aug 05 17:16:32.282862 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:16:32.282835    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2ff1f472-19b1-433f-ba0a-91def41dcaca-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "2ff1f472-19b1-433f-ba0a-91def41dcaca" (UID: "2ff1f472-19b1-433f-ba0a-91def41dcaca"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 05 17:16:32.282989 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:16:32.282956    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2ff1f472-19b1-433f-ba0a-91def41dcaca-console-config" (OuterVolumeSpecName: "console-config") pod "2ff1f472-19b1-433f-ba0a-91def41dcaca" (UID: "2ff1f472-19b1-433f-ba0a-91def41dcaca"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 05 17:16:32.283056 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:16:32.282992    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2ff1f472-19b1-433f-ba0a-91def41dcaca-service-ca" (OuterVolumeSpecName: "service-ca") pod "2ff1f472-19b1-433f-ba0a-91def41dcaca" (UID: "2ff1f472-19b1-433f-ba0a-91def41dcaca"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 05 17:16:32.284629 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:16:32.284592    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2ff1f472-19b1-433f-ba0a-91def41dcaca-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "2ff1f472-19b1-433f-ba0a-91def41dcaca" (UID: "2ff1f472-19b1-433f-ba0a-91def41dcaca"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 05 17:16:32.284757 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:16:32.284632    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2ff1f472-19b1-433f-ba0a-91def41dcaca-kube-api-access-k2fmm" (OuterVolumeSpecName: "kube-api-access-k2fmm") pod "2ff1f472-19b1-433f-ba0a-91def41dcaca" (UID: "2ff1f472-19b1-433f-ba0a-91def41dcaca"). InnerVolumeSpecName "kube-api-access-k2fmm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 05 17:16:32.284881 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:16:32.284862    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2ff1f472-19b1-433f-ba0a-91def41dcaca-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "2ff1f472-19b1-433f-ba0a-91def41dcaca" (UID: "2ff1f472-19b1-433f-ba0a-91def41dcaca"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 05 17:16:32.383523 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:16:32.383439    2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k2fmm\" (UniqueName: \"kubernetes.io/projected/2ff1f472-19b1-433f-ba0a-91def41dcaca-kube-api-access-k2fmm\") on node \"ip-10-0-135-2.ec2.internal\" DevicePath \"\""
Aug 05 17:16:32.383523 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:16:32.383466    2571 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/2ff1f472-19b1-433f-ba0a-91def41dcaca-console-oauth-config\") on node \"ip-10-0-135-2.ec2.internal\" DevicePath \"\""
Aug 05 17:16:32.383523 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:16:32.383478    2571 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/2ff1f472-19b1-433f-ba0a-91def41dcaca-oauth-serving-cert\") on node \"ip-10-0-135-2.ec2.internal\" DevicePath \"\""
Aug 05 17:16:32.383523 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:16:32.383486    2571 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/2ff1f472-19b1-433f-ba0a-91def41dcaca-console-config\") on node \"ip-10-0-135-2.ec2.internal\" DevicePath \"\""
Aug 05 17:16:32.383523 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:16:32.383494    2571 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/2ff1f472-19b1-433f-ba0a-91def41dcaca-service-ca\") on node \"ip-10-0-135-2.ec2.internal\" DevicePath \"\""
Aug 05 17:16:32.383523 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:16:32.383502    2571 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/2ff1f472-19b1-433f-ba0a-91def41dcaca-console-serving-cert\") on node \"ip-10-0-135-2.ec2.internal\" DevicePath \"\""
Aug 05 17:16:32.970486 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:16:32.970454    2571 generic.go:356] "Generic (PLEG): container finished" podID="2ff1f472-19b1-433f-ba0a-91def41dcaca" containerID="7b0b49ad0a12256746ec9e06d38fbe284a1228cf286fb2902ed42c682f651782" exitCode=0
Aug 05 17:16:32.970891 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:16:32.970507    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6bb977d6dd-227dh"
Aug 05 17:16:32.970891 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:16:32.970511    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6bb977d6dd-227dh" event={"ID":"2ff1f472-19b1-433f-ba0a-91def41dcaca","Type":"ContainerDied","Data":"7b0b49ad0a12256746ec9e06d38fbe284a1228cf286fb2902ed42c682f651782"}
Aug 05 17:16:32.970891 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:16:32.970632    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6bb977d6dd-227dh" event={"ID":"2ff1f472-19b1-433f-ba0a-91def41dcaca","Type":"ContainerDied","Data":"e2ffec397ff596cb2ab74988d0168dbb7db0af9e95b03409790420cbeb9e1912"}
Aug 05 17:16:32.970891 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:16:32.970651    2571 scope.go:117] "RemoveContainer" containerID="7b0b49ad0a12256746ec9e06d38fbe284a1228cf286fb2902ed42c682f651782"
Aug 05 17:16:32.979264 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:16:32.979247    2571 scope.go:117] "RemoveContainer" containerID="7b0b49ad0a12256746ec9e06d38fbe284a1228cf286fb2902ed42c682f651782"
Aug 05 17:16:32.979525 ip-10-0-135-2 kubenswrapper[2571]: E0805 17:16:32.979508    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7b0b49ad0a12256746ec9e06d38fbe284a1228cf286fb2902ed42c682f651782\": container with ID starting with 7b0b49ad0a12256746ec9e06d38fbe284a1228cf286fb2902ed42c682f651782 not found: ID does not exist" containerID="7b0b49ad0a12256746ec9e06d38fbe284a1228cf286fb2902ed42c682f651782"
Aug 05 17:16:32.979575 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:16:32.979533    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7b0b49ad0a12256746ec9e06d38fbe284a1228cf286fb2902ed42c682f651782"} err="failed to get container status \"7b0b49ad0a12256746ec9e06d38fbe284a1228cf286fb2902ed42c682f651782\": rpc error: code = NotFound desc = could not find container \"7b0b49ad0a12256746ec9e06d38fbe284a1228cf286fb2902ed42c682f651782\": container with ID starting with 7b0b49ad0a12256746ec9e06d38fbe284a1228cf286fb2902ed42c682f651782 not found: ID does not exist"
Aug 05 17:16:32.991734 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:16:32.991715    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6bb977d6dd-227dh"]
Aug 05 17:16:32.996821 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:16:32.996801    2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-6bb977d6dd-227dh"]
Aug 05 17:16:33.135794 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:16:33.135765    2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2ff1f472-19b1-433f-ba0a-91def41dcaca" path="/var/lib/kubelet/pods/2ff1f472-19b1-433f-ba0a-91def41dcaca/volumes"
Aug 05 17:16:35.656422 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:16:35.656365    2571 prober.go:120] "Probe failed" probeType="Liveness" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7b74c45fdd-9tcbw" podUID="83dbc4d1-49f4-4f07-97d2-80443a2c873f" containerName="service-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 500"
Aug 05 17:16:35.656869 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:16:35.656461    2571 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7b74c45fdd-9tcbw"
Aug 05 17:16:35.656919 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:16:35.656906    2571 kuberuntime_manager.go:1192] "Message for Container of pod" containerName="service-proxy" containerStatusID={"Type":"cri-o","ID":"37629569b3e236ae5d3c984c209348a9d6919874bcf4f0332121ca5ebaabee27"} pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7b74c45fdd-9tcbw" containerMessage="Container service-proxy failed liveness probe, will be restarted"
Aug 05 17:16:35.656969 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:16:35.656937    2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7b74c45fdd-9tcbw" podUID="83dbc4d1-49f4-4f07-97d2-80443a2c873f" containerName="service-proxy" containerID="cri-o://37629569b3e236ae5d3c984c209348a9d6919874bcf4f0332121ca5ebaabee27" gracePeriod=30
Aug 05 17:16:35.983642 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:16:35.983611    2571 generic.go:356] "Generic (PLEG): container finished" podID="83dbc4d1-49f4-4f07-97d2-80443a2c873f" containerID="37629569b3e236ae5d3c984c209348a9d6919874bcf4f0332121ca5ebaabee27" exitCode=2
Aug 05 17:16:35.983781 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:16:35.983683    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7b74c45fdd-9tcbw" event={"ID":"83dbc4d1-49f4-4f07-97d2-80443a2c873f","Type":"ContainerDied","Data":"37629569b3e236ae5d3c984c209348a9d6919874bcf4f0332121ca5ebaabee27"}
Aug 05 17:16:35.983781 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:16:35.983722    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7b74c45fdd-9tcbw" event={"ID":"83dbc4d1-49f4-4f07-97d2-80443a2c873f","Type":"ContainerStarted","Data":"f6e2831585d8f22c30e09843fc5d2ce672cbf3821acfcb9f18c74bc305df7a28"}
Aug 05 17:16:47.627413 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:16:47.627367    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Aug 05 17:16:47.628054 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:16:47.627636    2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="cd891c78-e151-4947-9b17-8e6661e14e3d" containerName="alertmanager" containerID="cri-o://8f182ee1f6c240989276566eb44bcd62bfb446c0de87a0d49b4716766d161ea4" gracePeriod=120
Aug 05 17:16:47.628054 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:16:47.627715    2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="cd891c78-e151-4947-9b17-8e6661e14e3d" containerName="kube-rbac-proxy-metric" containerID="cri-o://12122ac4cb00528930f0b1dc451347eff33581b7b9778e10f74fe97e51bc84d0" gracePeriod=120
Aug 05 17:16:47.628054 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:16:47.627741    2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="cd891c78-e151-4947-9b17-8e6661e14e3d" containerName="kube-rbac-proxy" containerID="cri-o://5f2d32c522590c79d6328c038e4858a6aa15d85625718af9b73ce0d5151a9807" gracePeriod=120
Aug 05 17:16:47.628054 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:16:47.627782    2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="cd891c78-e151-4947-9b17-8e6661e14e3d" containerName="prom-label-proxy" containerID="cri-o://07025a8abe4a96db80646075eb74a6869741bfd13f01b0ecd390cf030877af07" gracePeriod=120
Aug 05 17:16:47.628054 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:16:47.627782    2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="cd891c78-e151-4947-9b17-8e6661e14e3d" containerName="config-reloader" containerID="cri-o://dfc1fe9f05269cf12ab75906467b12217226d984e37cb1f8f7ac761463138d0f" gracePeriod=120
Aug 05 17:16:47.628054 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:16:47.627740    2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="cd891c78-e151-4947-9b17-8e6661e14e3d" containerName="kube-rbac-proxy-web" containerID="cri-o://2c07e3a79698de281cdd0a56b1c7168d342b7937e17b04f5119352110d33a676" gracePeriod=120
Aug 05 17:16:48.031588 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:16:48.031550    2571 generic.go:356] "Generic (PLEG): container finished" podID="cd891c78-e151-4947-9b17-8e6661e14e3d" containerID="07025a8abe4a96db80646075eb74a6869741bfd13f01b0ecd390cf030877af07" exitCode=0
Aug 05 17:16:48.031588 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:16:48.031583    2571 generic.go:356] "Generic (PLEG): container finished" podID="cd891c78-e151-4947-9b17-8e6661e14e3d" containerID="12122ac4cb00528930f0b1dc451347eff33581b7b9778e10f74fe97e51bc84d0" exitCode=0
Aug 05 17:16:48.031588 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:16:48.031590    2571 generic.go:356] "Generic (PLEG): container finished" podID="cd891c78-e151-4947-9b17-8e6661e14e3d" containerID="5f2d32c522590c79d6328c038e4858a6aa15d85625718af9b73ce0d5151a9807" exitCode=0
Aug 05 17:16:48.031588 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:16:48.031595    2571 generic.go:356] "Generic (PLEG): container finished" podID="cd891c78-e151-4947-9b17-8e6661e14e3d" containerID="dfc1fe9f05269cf12ab75906467b12217226d984e37cb1f8f7ac761463138d0f" exitCode=0
Aug 05 17:16:48.031588 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:16:48.031600    2571 generic.go:356] "Generic (PLEG): container finished" podID="cd891c78-e151-4947-9b17-8e6661e14e3d" containerID="8f182ee1f6c240989276566eb44bcd62bfb446c0de87a0d49b4716766d161ea4" exitCode=0
Aug 05 17:16:48.031877 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:16:48.031624    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"cd891c78-e151-4947-9b17-8e6661e14e3d","Type":"ContainerDied","Data":"07025a8abe4a96db80646075eb74a6869741bfd13f01b0ecd390cf030877af07"}
Aug 05 17:16:48.031877 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:16:48.031691    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"cd891c78-e151-4947-9b17-8e6661e14e3d","Type":"ContainerDied","Data":"12122ac4cb00528930f0b1dc451347eff33581b7b9778e10f74fe97e51bc84d0"}
Aug 05 17:16:48.031877 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:16:48.031719    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"cd891c78-e151-4947-9b17-8e6661e14e3d","Type":"ContainerDied","Data":"5f2d32c522590c79d6328c038e4858a6aa15d85625718af9b73ce0d5151a9807"}
Aug 05 17:16:48.031877 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:16:48.031744    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"cd891c78-e151-4947-9b17-8e6661e14e3d","Type":"ContainerDied","Data":"dfc1fe9f05269cf12ab75906467b12217226d984e37cb1f8f7ac761463138d0f"}
Aug 05 17:16:48.031877 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:16:48.031768    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"cd891c78-e151-4947-9b17-8e6661e14e3d","Type":"ContainerDied","Data":"8f182ee1f6c240989276566eb44bcd62bfb446c0de87a0d49b4716766d161ea4"}
Aug 05 17:16:48.867308 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:16:48.867285    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Aug 05 17:16:48.906785 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:16:48.906757    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/cd891c78-e151-4947-9b17-8e6661e14e3d-config-out\") pod \"cd891c78-e151-4947-9b17-8e6661e14e3d\" (UID: \"cd891c78-e151-4947-9b17-8e6661e14e3d\") "
Aug 05 17:16:48.906908 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:16:48.906793    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/cd891c78-e151-4947-9b17-8e6661e14e3d-secret-alertmanager-main-tls\") pod \"cd891c78-e151-4947-9b17-8e6661e14e3d\" (UID: \"cd891c78-e151-4947-9b17-8e6661e14e3d\") "
Aug 05 17:16:48.906908 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:16:48.906816    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/cd891c78-e151-4947-9b17-8e6661e14e3d-cluster-tls-config\") pod \"cd891c78-e151-4947-9b17-8e6661e14e3d\" (UID: \"cd891c78-e151-4947-9b17-8e6661e14e3d\") "
Aug 05 17:16:48.906908 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:16:48.906845    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/cd891c78-e151-4947-9b17-8e6661e14e3d-config-volume\") pod \"cd891c78-e151-4947-9b17-8e6661e14e3d\" (UID: \"cd891c78-e151-4947-9b17-8e6661e14e3d\") "
Aug 05 17:16:48.907074 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:16:48.906981    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/cd891c78-e151-4947-9b17-8e6661e14e3d-secret-alertmanager-kube-rbac-proxy-web\") pod \"cd891c78-e151-4947-9b17-8e6661e14e3d\" (UID: \"cd891c78-e151-4947-9b17-8e6661e14e3d\") "
Aug 05 17:16:48.907074 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:16:48.907024    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/cd891c78-e151-4947-9b17-8e6661e14e3d-secret-alertmanager-kube-rbac-proxy\") pod \"cd891c78-e151-4947-9b17-8e6661e14e3d\" (UID: \"cd891c78-e151-4947-9b17-8e6661e14e3d\") "
Aug 05 17:16:48.907074 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:16:48.907060    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/cd891c78-e151-4947-9b17-8e6661e14e3d-metrics-client-ca\") pod \"cd891c78-e151-4947-9b17-8e6661e14e3d\" (UID: \"cd891c78-e151-4947-9b17-8e6661e14e3d\") "
Aug 05 17:16:48.907231 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:16:48.907098    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/cd891c78-e151-4947-9b17-8e6661e14e3d-alertmanager-main-db\") pod \"cd891c78-e151-4947-9b17-8e6661e14e3d\" (UID: \"cd891c78-e151-4947-9b17-8e6661e14e3d\") "
Aug 05 17:16:48.907231 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:16:48.907137    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/cd891c78-e151-4947-9b17-8e6661e14e3d-secret-alertmanager-kube-rbac-proxy-metric\") pod \"cd891c78-e151-4947-9b17-8e6661e14e3d\" (UID: \"cd891c78-e151-4947-9b17-8e6661e14e3d\") "
Aug 05 17:16:48.907231 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:16:48.907170    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cd891c78-e151-4947-9b17-8e6661e14e3d-alertmanager-trusted-ca-bundle\") pod \"cd891c78-e151-4947-9b17-8e6661e14e3d\" (UID: \"cd891c78-e151-4947-9b17-8e6661e14e3d\") "
Aug 05 17:16:48.907231 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:16:48.907207    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-q5mnj\" (UniqueName: \"kubernetes.io/projected/cd891c78-e151-4947-9b17-8e6661e14e3d-kube-api-access-q5mnj\") pod \"cd891c78-e151-4947-9b17-8e6661e14e3d\" (UID: \"cd891c78-e151-4947-9b17-8e6661e14e3d\") "
Aug 05 17:16:48.907934 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:16:48.907236    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/cd891c78-e151-4947-9b17-8e6661e14e3d-web-config\") pod \"cd891c78-e151-4947-9b17-8e6661e14e3d\" (UID: \"cd891c78-e151-4947-9b17-8e6661e14e3d\") "
Aug 05 17:16:48.907934 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:16:48.907259    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/cd891c78-e151-4947-9b17-8e6661e14e3d-tls-assets\") pod \"cd891c78-e151-4947-9b17-8e6661e14e3d\" (UID: \"cd891c78-e151-4947-9b17-8e6661e14e3d\") "
Aug 05 17:16:48.908474 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:16:48.908446    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cd891c78-e151-4947-9b17-8e6661e14e3d-metrics-client-ca" (OuterVolumeSpecName: "metrics-client-ca") pod "cd891c78-e151-4947-9b17-8e6661e14e3d" (UID: "cd891c78-e151-4947-9b17-8e6661e14e3d"). InnerVolumeSpecName "metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 05 17:16:48.908617 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:16:48.908595    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cd891c78-e151-4947-9b17-8e6661e14e3d-alertmanager-main-db" (OuterVolumeSpecName: "alertmanager-main-db") pod "cd891c78-e151-4947-9b17-8e6661e14e3d" (UID: "cd891c78-e151-4947-9b17-8e6661e14e3d"). InnerVolumeSpecName "alertmanager-main-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 05 17:16:48.910095 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:16:48.909906    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cd891c78-e151-4947-9b17-8e6661e14e3d-alertmanager-trusted-ca-bundle" (OuterVolumeSpecName: "alertmanager-trusted-ca-bundle") pod "cd891c78-e151-4947-9b17-8e6661e14e3d" (UID: "cd891c78-e151-4947-9b17-8e6661e14e3d"). InnerVolumeSpecName "alertmanager-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 05 17:16:48.914505 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:16:48.911409    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cd891c78-e151-4947-9b17-8e6661e14e3d-secret-alertmanager-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-web") pod "cd891c78-e151-4947-9b17-8e6661e14e3d" (UID: "cd891c78-e151-4947-9b17-8e6661e14e3d"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 05 17:16:48.914505 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:16:48.912344    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cd891c78-e151-4947-9b17-8e6661e14e3d-secret-alertmanager-main-tls" (OuterVolumeSpecName: "secret-alertmanager-main-tls") pod "cd891c78-e151-4947-9b17-8e6661e14e3d" (UID: "cd891c78-e151-4947-9b17-8e6661e14e3d"). InnerVolumeSpecName "secret-alertmanager-main-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 05 17:16:48.914505 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:16:48.912420    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cd891c78-e151-4947-9b17-8e6661e14e3d-config-volume" (OuterVolumeSpecName: "config-volume") pod "cd891c78-e151-4947-9b17-8e6661e14e3d" (UID: "cd891c78-e151-4947-9b17-8e6661e14e3d"). InnerVolumeSpecName "config-volume". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 05 17:16:48.914505 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:16:48.912494    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cd891c78-e151-4947-9b17-8e6661e14e3d-secret-alertmanager-kube-rbac-proxy-metric" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-metric") pod "cd891c78-e151-4947-9b17-8e6661e14e3d" (UID: "cd891c78-e151-4947-9b17-8e6661e14e3d"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-metric". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 05 17:16:48.914505 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:16:48.912574    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cd891c78-e151-4947-9b17-8e6661e14e3d-kube-api-access-q5mnj" (OuterVolumeSpecName: "kube-api-access-q5mnj") pod "cd891c78-e151-4947-9b17-8e6661e14e3d" (UID: "cd891c78-e151-4947-9b17-8e6661e14e3d"). InnerVolumeSpecName "kube-api-access-q5mnj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 05 17:16:48.914505 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:16:48.912629    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cd891c78-e151-4947-9b17-8e6661e14e3d-config-out" (OuterVolumeSpecName: "config-out") pod "cd891c78-e151-4947-9b17-8e6661e14e3d" (UID: "cd891c78-e151-4947-9b17-8e6661e14e3d"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 05 17:16:48.921637 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:16:48.920891    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cd891c78-e151-4947-9b17-8e6661e14e3d-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "cd891c78-e151-4947-9b17-8e6661e14e3d" (UID: "cd891c78-e151-4947-9b17-8e6661e14e3d"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 05 17:16:48.922065 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:16:48.922029    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cd891c78-e151-4947-9b17-8e6661e14e3d-secret-alertmanager-kube-rbac-proxy" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy") pod "cd891c78-e151-4947-9b17-8e6661e14e3d" (UID: "cd891c78-e151-4947-9b17-8e6661e14e3d"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 05 17:16:48.926063 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:16:48.926014    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cd891c78-e151-4947-9b17-8e6661e14e3d-cluster-tls-config" (OuterVolumeSpecName: "cluster-tls-config") pod "cd891c78-e151-4947-9b17-8e6661e14e3d" (UID: "cd891c78-e151-4947-9b17-8e6661e14e3d"). InnerVolumeSpecName "cluster-tls-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 05 17:16:48.927520 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:16:48.927490    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cd891c78-e151-4947-9b17-8e6661e14e3d-web-config" (OuterVolumeSpecName: "web-config") pod "cd891c78-e151-4947-9b17-8e6661e14e3d" (UID: "cd891c78-e151-4947-9b17-8e6661e14e3d"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 05 17:16:49.008054 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:16:49.008021    2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q5mnj\" (UniqueName: \"kubernetes.io/projected/cd891c78-e151-4947-9b17-8e6661e14e3d-kube-api-access-q5mnj\") on node \"ip-10-0-135-2.ec2.internal\" DevicePath \"\""
Aug 05 17:16:49.008054 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:16:49.008054    2571 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/cd891c78-e151-4947-9b17-8e6661e14e3d-web-config\") on node \"ip-10-0-135-2.ec2.internal\" DevicePath \"\""
Aug 05 17:16:49.008222 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:16:49.008065    2571 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/cd891c78-e151-4947-9b17-8e6661e14e3d-tls-assets\") on node \"ip-10-0-135-2.ec2.internal\" DevicePath \"\""
Aug 05 17:16:49.008222 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:16:49.008073    2571 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/cd891c78-e151-4947-9b17-8e6661e14e3d-config-out\") on node \"ip-10-0-135-2.ec2.internal\" DevicePath \"\""
Aug 05 17:16:49.008222 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:16:49.008081    2571 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/cd891c78-e151-4947-9b17-8e6661e14e3d-secret-alertmanager-main-tls\") on node \"ip-10-0-135-2.ec2.internal\" DevicePath \"\""
Aug 05 17:16:49.008222 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:16:49.008089    2571 reconciler_common.go:299] "Volume detached for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/cd891c78-e151-4947-9b17-8e6661e14e3d-cluster-tls-config\") on node \"ip-10-0-135-2.ec2.internal\" DevicePath \"\""
Aug 05 17:16:49.008222 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:16:49.008096    2571 reconciler_common.go:299] "Volume detached for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/cd891c78-e151-4947-9b17-8e6661e14e3d-config-volume\") on node \"ip-10-0-135-2.ec2.internal\" DevicePath \"\""
Aug 05 17:16:49.008222 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:16:49.008106    2571 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/cd891c78-e151-4947-9b17-8e6661e14e3d-secret-alertmanager-kube-rbac-proxy-web\") on node \"ip-10-0-135-2.ec2.internal\" DevicePath \"\""
Aug 05 17:16:49.008222 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:16:49.008115    2571 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/cd891c78-e151-4947-9b17-8e6661e14e3d-secret-alertmanager-kube-rbac-proxy\") on node \"ip-10-0-135-2.ec2.internal\" DevicePath \"\""
Aug 05 17:16:49.008222 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:16:49.008124    2571 reconciler_common.go:299] "Volume detached for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/cd891c78-e151-4947-9b17-8e6661e14e3d-metrics-client-ca\") on node \"ip-10-0-135-2.ec2.internal\" DevicePath \"\""
Aug 05 17:16:49.008222 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:16:49.008132    2571 reconciler_common.go:299] "Volume detached for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/cd891c78-e151-4947-9b17-8e6661e14e3d-alertmanager-main-db\") on node \"ip-10-0-135-2.ec2.internal\" DevicePath \"\""
Aug 05 17:16:49.008222 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:16:49.008140    2571 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/cd891c78-e151-4947-9b17-8e6661e14e3d-secret-alertmanager-kube-rbac-proxy-metric\") on node \"ip-10-0-135-2.ec2.internal\" DevicePath \"\""
Aug 05 17:16:49.008222 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:16:49.008150    2571 reconciler_common.go:299] "Volume detached for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cd891c78-e151-4947-9b17-8e6661e14e3d-alertmanager-trusted-ca-bundle\") on node \"ip-10-0-135-2.ec2.internal\" DevicePath \"\""
Aug 05 17:16:49.038110 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:16:49.038076    2571 generic.go:356] "Generic (PLEG): container finished" podID="cd891c78-e151-4947-9b17-8e6661e14e3d" containerID="2c07e3a79698de281cdd0a56b1c7168d342b7937e17b04f5119352110d33a676" exitCode=0
Aug 05 17:16:49.038225 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:16:49.038156    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Aug 05 17:16:49.038268 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:16:49.038153    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"cd891c78-e151-4947-9b17-8e6661e14e3d","Type":"ContainerDied","Data":"2c07e3a79698de281cdd0a56b1c7168d342b7937e17b04f5119352110d33a676"}
Aug 05 17:16:49.038318 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:16:49.038304    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"cd891c78-e151-4947-9b17-8e6661e14e3d","Type":"ContainerDied","Data":"168197f2ab64ec98a2706756d33c0d8a5325ea53d0326cf978fa64c630e57f81"}
Aug 05 17:16:49.038352 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:16:49.038343    2571 scope.go:117] "RemoveContainer" containerID="07025a8abe4a96db80646075eb74a6869741bfd13f01b0ecd390cf030877af07"
Aug 05 17:16:49.048349 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:16:49.048331    2571 scope.go:117] "RemoveContainer" containerID="12122ac4cb00528930f0b1dc451347eff33581b7b9778e10f74fe97e51bc84d0"
Aug 05 17:16:49.054883 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:16:49.054867    2571 scope.go:117] "RemoveContainer" containerID="5f2d32c522590c79d6328c038e4858a6aa15d85625718af9b73ce0d5151a9807"
Aug 05 17:16:49.061065 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:16:49.061048    2571 scope.go:117] "RemoveContainer" containerID="2c07e3a79698de281cdd0a56b1c7168d342b7937e17b04f5119352110d33a676"
Aug 05 17:16:49.063637 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:16:49.063619    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Aug 05 17:16:49.067776 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:16:49.067757    2571 scope.go:117] "RemoveContainer" containerID="dfc1fe9f05269cf12ab75906467b12217226d984e37cb1f8f7ac761463138d0f"
Aug 05 17:16:49.069187 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:16:49.069144    2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Aug 05 17:16:49.074599 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:16:49.074585    2571 scope.go:117] "RemoveContainer" containerID="8f182ee1f6c240989276566eb44bcd62bfb446c0de87a0d49b4716766d161ea4"
Aug 05 17:16:49.080892 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:16:49.080876    2571 scope.go:117] "RemoveContainer" containerID="54d1beff424951f03b3533bb884eb8ee8c5962958fd009ccc62c841e1e273a52"
Aug 05 17:16:49.087202 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:16:49.087187    2571 scope.go:117] "RemoveContainer" containerID="07025a8abe4a96db80646075eb74a6869741bfd13f01b0ecd390cf030877af07"
Aug 05 17:16:49.087456 ip-10-0-135-2 kubenswrapper[2571]: E0805 17:16:49.087439    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"07025a8abe4a96db80646075eb74a6869741bfd13f01b0ecd390cf030877af07\": container with ID starting with 07025a8abe4a96db80646075eb74a6869741bfd13f01b0ecd390cf030877af07 not found: ID does not exist" containerID="07025a8abe4a96db80646075eb74a6869741bfd13f01b0ecd390cf030877af07"
Aug 05 17:16:49.087517 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:16:49.087463    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"07025a8abe4a96db80646075eb74a6869741bfd13f01b0ecd390cf030877af07"} err="failed to get container status \"07025a8abe4a96db80646075eb74a6869741bfd13f01b0ecd390cf030877af07\": rpc error: code = NotFound desc = could not find container \"07025a8abe4a96db80646075eb74a6869741bfd13f01b0ecd390cf030877af07\": container with ID starting with 07025a8abe4a96db80646075eb74a6869741bfd13f01b0ecd390cf030877af07 not found: ID does not exist"
Aug 05 17:16:49.087517 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:16:49.087481    2571 scope.go:117] "RemoveContainer" containerID="12122ac4cb00528930f0b1dc451347eff33581b7b9778e10f74fe97e51bc84d0"
Aug 05 17:16:49.087712 ip-10-0-135-2 kubenswrapper[2571]: E0805 17:16:49.087698    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"12122ac4cb00528930f0b1dc451347eff33581b7b9778e10f74fe97e51bc84d0\": container with ID starting with 12122ac4cb00528930f0b1dc451347eff33581b7b9778e10f74fe97e51bc84d0 not found: ID does not exist" containerID="12122ac4cb00528930f0b1dc451347eff33581b7b9778e10f74fe97e51bc84d0"
Aug 05 17:16:49.087755 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:16:49.087715    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"12122ac4cb00528930f0b1dc451347eff33581b7b9778e10f74fe97e51bc84d0"} err="failed to get container status \"12122ac4cb00528930f0b1dc451347eff33581b7b9778e10f74fe97e51bc84d0\": rpc error: code = NotFound desc = could not find container \"12122ac4cb00528930f0b1dc451347eff33581b7b9778e10f74fe97e51bc84d0\": container with ID starting with 12122ac4cb00528930f0b1dc451347eff33581b7b9778e10f74fe97e51bc84d0 not found: ID does not exist"
Aug 05 17:16:49.087755 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:16:49.087728    2571 scope.go:117] "RemoveContainer" containerID="5f2d32c522590c79d6328c038e4858a6aa15d85625718af9b73ce0d5151a9807"
Aug 05 17:16:49.087971 ip-10-0-135-2 kubenswrapper[2571]: E0805 17:16:49.087955    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5f2d32c522590c79d6328c038e4858a6aa15d85625718af9b73ce0d5151a9807\": container with ID starting with 5f2d32c522590c79d6328c038e4858a6aa15d85625718af9b73ce0d5151a9807 not found: ID does not exist" containerID="5f2d32c522590c79d6328c038e4858a6aa15d85625718af9b73ce0d5151a9807"
Aug 05 17:16:49.088011 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:16:49.087976    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5f2d32c522590c79d6328c038e4858a6aa15d85625718af9b73ce0d5151a9807"} err="failed to get container status \"5f2d32c522590c79d6328c038e4858a6aa15d85625718af9b73ce0d5151a9807\": rpc error: code = NotFound desc = could not find container \"5f2d32c522590c79d6328c038e4858a6aa15d85625718af9b73ce0d5151a9807\": container with ID starting with 5f2d32c522590c79d6328c038e4858a6aa15d85625718af9b73ce0d5151a9807 not found: ID does not exist"
Aug 05 17:16:49.088011 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:16:49.087991    2571 scope.go:117] "RemoveContainer" containerID="2c07e3a79698de281cdd0a56b1c7168d342b7937e17b04f5119352110d33a676"
Aug 05 17:16:49.088187 ip-10-0-135-2 kubenswrapper[2571]: E0805 17:16:49.088174    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2c07e3a79698de281cdd0a56b1c7168d342b7937e17b04f5119352110d33a676\": container with ID starting with 2c07e3a79698de281cdd0a56b1c7168d342b7937e17b04f5119352110d33a676 not found: ID does not exist" containerID="2c07e3a79698de281cdd0a56b1c7168d342b7937e17b04f5119352110d33a676"
Aug 05 17:16:49.088225 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:16:49.088191    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2c07e3a79698de281cdd0a56b1c7168d342b7937e17b04f5119352110d33a676"} err="failed to get container status \"2c07e3a79698de281cdd0a56b1c7168d342b7937e17b04f5119352110d33a676\": rpc error: code = NotFound desc = could not find container \"2c07e3a79698de281cdd0a56b1c7168d342b7937e17b04f5119352110d33a676\": container with ID starting with 2c07e3a79698de281cdd0a56b1c7168d342b7937e17b04f5119352110d33a676 not found: ID does not exist"
Aug 05 17:16:49.088225 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:16:49.088203    2571 scope.go:117] "RemoveContainer" containerID="dfc1fe9f05269cf12ab75906467b12217226d984e37cb1f8f7ac761463138d0f"
Aug 05 17:16:49.088476 ip-10-0-135-2 kubenswrapper[2571]: E0805 17:16:49.088461    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dfc1fe9f05269cf12ab75906467b12217226d984e37cb1f8f7ac761463138d0f\": container with ID starting with dfc1fe9f05269cf12ab75906467b12217226d984e37cb1f8f7ac761463138d0f not found: ID does not exist" containerID="dfc1fe9f05269cf12ab75906467b12217226d984e37cb1f8f7ac761463138d0f"
Aug 05 17:16:49.088519 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:16:49.088480    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dfc1fe9f05269cf12ab75906467b12217226d984e37cb1f8f7ac761463138d0f"} err="failed to get container status \"dfc1fe9f05269cf12ab75906467b12217226d984e37cb1f8f7ac761463138d0f\": rpc error: code = NotFound desc = could not find container \"dfc1fe9f05269cf12ab75906467b12217226d984e37cb1f8f7ac761463138d0f\": container with ID starting with dfc1fe9f05269cf12ab75906467b12217226d984e37cb1f8f7ac761463138d0f not found: ID does not exist"
Aug 05 17:16:49.088519 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:16:49.088491    2571 scope.go:117] "RemoveContainer" containerID="8f182ee1f6c240989276566eb44bcd62bfb446c0de87a0d49b4716766d161ea4"
Aug 05 17:16:49.088709 ip-10-0-135-2 kubenswrapper[2571]: E0805 17:16:49.088694    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8f182ee1f6c240989276566eb44bcd62bfb446c0de87a0d49b4716766d161ea4\": container with ID starting with 8f182ee1f6c240989276566eb44bcd62bfb446c0de87a0d49b4716766d161ea4 not found: ID does not exist" containerID="8f182ee1f6c240989276566eb44bcd62bfb446c0de87a0d49b4716766d161ea4"
Aug 05 17:16:49.088758 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:16:49.088712    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8f182ee1f6c240989276566eb44bcd62bfb446c0de87a0d49b4716766d161ea4"} err="failed to get container status \"8f182ee1f6c240989276566eb44bcd62bfb446c0de87a0d49b4716766d161ea4\": rpc error: code = NotFound desc = could not find container \"8f182ee1f6c240989276566eb44bcd62bfb446c0de87a0d49b4716766d161ea4\": container with ID starting with 8f182ee1f6c240989276566eb44bcd62bfb446c0de87a0d49b4716766d161ea4 not found: ID does not exist"
Aug 05 17:16:49.088758 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:16:49.088724    2571 scope.go:117] "RemoveContainer" containerID="54d1beff424951f03b3533bb884eb8ee8c5962958fd009ccc62c841e1e273a52"
Aug 05 17:16:49.088912 ip-10-0-135-2 kubenswrapper[2571]: E0805 17:16:49.088897    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"54d1beff424951f03b3533bb884eb8ee8c5962958fd009ccc62c841e1e273a52\": container with ID starting with 54d1beff424951f03b3533bb884eb8ee8c5962958fd009ccc62c841e1e273a52 not found: ID does not exist" containerID="54d1beff424951f03b3533bb884eb8ee8c5962958fd009ccc62c841e1e273a52"
Aug 05 17:16:49.088952 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:16:49.088915    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"54d1beff424951f03b3533bb884eb8ee8c5962958fd009ccc62c841e1e273a52"} err="failed to get container status \"54d1beff424951f03b3533bb884eb8ee8c5962958fd009ccc62c841e1e273a52\": rpc error: code = NotFound desc = could not find container \"54d1beff424951f03b3533bb884eb8ee8c5962958fd009ccc62c841e1e273a52\": container with ID starting with 54d1beff424951f03b3533bb884eb8ee8c5962958fd009ccc62c841e1e273a52 not found: ID does not exist"
Aug 05 17:16:49.116903 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:16:49.116879    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Aug 05 17:16:49.117136 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:16:49.117125    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cd891c78-e151-4947-9b17-8e6661e14e3d" containerName="kube-rbac-proxy-web"
Aug 05 17:16:49.117173 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:16:49.117139    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="cd891c78-e151-4947-9b17-8e6661e14e3d" containerName="kube-rbac-proxy-web"
Aug 05 17:16:49.117173 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:16:49.117152    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cd891c78-e151-4947-9b17-8e6661e14e3d" containerName="init-config-reloader"
Aug 05 17:16:49.117173 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:16:49.117158    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="cd891c78-e151-4947-9b17-8e6661e14e3d" containerName="init-config-reloader"
Aug 05 17:16:49.117173 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:16:49.117166    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2ff1f472-19b1-433f-ba0a-91def41dcaca" containerName="console"
Aug 05 17:16:49.117173 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:16:49.117171    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="2ff1f472-19b1-433f-ba0a-91def41dcaca" containerName="console"
Aug 05 17:16:49.117316 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:16:49.117181    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cd891c78-e151-4947-9b17-8e6661e14e3d" containerName="alertmanager"
Aug 05 17:16:49.117316 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:16:49.117186    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="cd891c78-e151-4947-9b17-8e6661e14e3d" containerName="alertmanager"
Aug 05 17:16:49.117316 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:16:49.117192    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cd891c78-e151-4947-9b17-8e6661e14e3d" containerName="kube-rbac-proxy"
Aug 05 17:16:49.117316 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:16:49.117197    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="cd891c78-e151-4947-9b17-8e6661e14e3d" containerName="kube-rbac-proxy"
Aug 05 17:16:49.117316 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:16:49.117203    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cd891c78-e151-4947-9b17-8e6661e14e3d" containerName="prom-label-proxy"
Aug 05 17:16:49.117316 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:16:49.117209    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="cd891c78-e151-4947-9b17-8e6661e14e3d" containerName="prom-label-proxy"
Aug 05 17:16:49.117316 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:16:49.117217    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cd891c78-e151-4947-9b17-8e6661e14e3d" containerName="config-reloader"
Aug 05 17:16:49.117316 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:16:49.117222    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="cd891c78-e151-4947-9b17-8e6661e14e3d" containerName="config-reloader"
Aug 05 17:16:49.117316 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:16:49.117228    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cd891c78-e151-4947-9b17-8e6661e14e3d" containerName="kube-rbac-proxy-metric"
Aug 05 17:16:49.117316 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:16:49.117233    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="cd891c78-e151-4947-9b17-8e6661e14e3d" containerName="kube-rbac-proxy-metric"
Aug 05 17:16:49.117316 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:16:49.117273    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="cd891c78-e151-4947-9b17-8e6661e14e3d" containerName="prom-label-proxy"
Aug 05 17:16:49.117316 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:16:49.117280    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="cd891c78-e151-4947-9b17-8e6661e14e3d" containerName="alertmanager"
Aug 05 17:16:49.117316 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:16:49.117286    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="cd891c78-e151-4947-9b17-8e6661e14e3d" containerName="kube-rbac-proxy"
Aug 05 17:16:49.117316 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:16:49.117292    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="cd891c78-e151-4947-9b17-8e6661e14e3d" containerName="config-reloader"
Aug 05 17:16:49.117316 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:16:49.117297    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="2ff1f472-19b1-433f-ba0a-91def41dcaca" containerName="console"
Aug 05 17:16:49.117316 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:16:49.117303    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="cd891c78-e151-4947-9b17-8e6661e14e3d" containerName="kube-rbac-proxy-web"
Aug 05 17:16:49.117316 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:16:49.117308    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="cd891c78-e151-4947-9b17-8e6661e14e3d" containerName="kube-rbac-proxy-metric"
Aug 05 17:16:49.121843 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:16:49.121828    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Aug 05 17:16:49.125196 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:16:49.125180    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
Aug 05 17:16:49.125269 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:16:49.125247    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
Aug 05 17:16:49.126662 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:16:49.126640    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
Aug 05 17:16:49.126750 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:16:49.126677    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
Aug 05 17:16:49.126750 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:16:49.126688    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
Aug 05 17:16:49.126750 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:16:49.126698    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
Aug 05 17:16:49.126750 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:16:49.126688    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-8j7wv\""
Aug 05 17:16:49.126750 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:16:49.126733    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
Aug 05 17:16:49.127039 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:16:49.126692    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
Aug 05 17:16:49.133503 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:16:49.133487    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
Aug 05 17:16:49.135386 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:16:49.135357    2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cd891c78-e151-4947-9b17-8e6661e14e3d" path="/var/lib/kubelet/pods/cd891c78-e151-4947-9b17-8e6661e14e3d/volumes"
Aug 05 17:16:49.142427 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:16:49.142407    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Aug 05 17:16:49.209886 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:16:49.209849    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/10e66bf1-f170-4f1b-a4ab-28ff387a260b-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"10e66bf1-f170-4f1b-a4ab-28ff387a260b\") " pod="openshift-monitoring/alertmanager-main-0"
Aug 05 17:16:49.209886 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:16:49.209890    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/10e66bf1-f170-4f1b-a4ab-28ff387a260b-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"10e66bf1-f170-4f1b-a4ab-28ff387a260b\") " pod="openshift-monitoring/alertmanager-main-0"
Aug 05 17:16:49.210107 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:16:49.209907    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/10e66bf1-f170-4f1b-a4ab-28ff387a260b-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"10e66bf1-f170-4f1b-a4ab-28ff387a260b\") " pod="openshift-monitoring/alertmanager-main-0"
Aug 05 17:16:49.210107 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:16:49.209960    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/10e66bf1-f170-4f1b-a4ab-28ff387a260b-web-config\") pod \"alertmanager-main-0\" (UID: \"10e66bf1-f170-4f1b-a4ab-28ff387a260b\") " pod="openshift-monitoring/alertmanager-main-0"
Aug 05 17:16:49.210107 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:16:49.209996    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/10e66bf1-f170-4f1b-a4ab-28ff387a260b-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"10e66bf1-f170-4f1b-a4ab-28ff387a260b\") " pod="openshift-monitoring/alertmanager-main-0"
Aug 05 17:16:49.210107 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:16:49.210031    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/10e66bf1-f170-4f1b-a4ab-28ff387a260b-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"10e66bf1-f170-4f1b-a4ab-28ff387a260b\") " pod="openshift-monitoring/alertmanager-main-0"
Aug 05 17:16:49.210107 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:16:49.210098    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/10e66bf1-f170-4f1b-a4ab-28ff387a260b-config-volume\") pod \"alertmanager-main-0\" (UID: \"10e66bf1-f170-4f1b-a4ab-28ff387a260b\") " pod="openshift-monitoring/alertmanager-main-0"
Aug 05 17:16:49.210357 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:16:49.210120    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/10e66bf1-f170-4f1b-a4ab-28ff387a260b-config-out\") pod \"alertmanager-main-0\" (UID: \"10e66bf1-f170-4f1b-a4ab-28ff387a260b\") " pod="openshift-monitoring/alertmanager-main-0"
Aug 05 17:16:49.210357 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:16:49.210141    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/10e66bf1-f170-4f1b-a4ab-28ff387a260b-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"10e66bf1-f170-4f1b-a4ab-28ff387a260b\") " pod="openshift-monitoring/alertmanager-main-0"
Aug 05 17:16:49.210357 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:16:49.210168    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/10e66bf1-f170-4f1b-a4ab-28ff387a260b-tls-assets\") pod \"alertmanager-main-0\" (UID: \"10e66bf1-f170-4f1b-a4ab-28ff387a260b\") " pod="openshift-monitoring/alertmanager-main-0"
Aug 05 17:16:49.210357 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:16:49.210194    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-96kbb\" (UniqueName: \"kubernetes.io/projected/10e66bf1-f170-4f1b-a4ab-28ff387a260b-kube-api-access-96kbb\") pod \"alertmanager-main-0\" (UID: \"10e66bf1-f170-4f1b-a4ab-28ff387a260b\") " pod="openshift-monitoring/alertmanager-main-0"
Aug 05 17:16:49.210357 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:16:49.210220    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/10e66bf1-f170-4f1b-a4ab-28ff387a260b-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"10e66bf1-f170-4f1b-a4ab-28ff387a260b\") " pod="openshift-monitoring/alertmanager-main-0"
Aug 05 17:16:49.210357 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:16:49.210255    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/10e66bf1-f170-4f1b-a4ab-28ff387a260b-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"10e66bf1-f170-4f1b-a4ab-28ff387a260b\") " pod="openshift-monitoring/alertmanager-main-0"
Aug 05 17:16:49.311014 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:16:49.310933    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/10e66bf1-f170-4f1b-a4ab-28ff387a260b-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"10e66bf1-f170-4f1b-a4ab-28ff387a260b\") " pod="openshift-monitoring/alertmanager-main-0"
Aug 05 17:16:49.311014 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:16:49.310981    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/10e66bf1-f170-4f1b-a4ab-28ff387a260b-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"10e66bf1-f170-4f1b-a4ab-28ff387a260b\") " pod="openshift-monitoring/alertmanager-main-0"
Aug 05 17:16:49.311184 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:16:49.311105    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/10e66bf1-f170-4f1b-a4ab-28ff387a260b-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"10e66bf1-f170-4f1b-a4ab-28ff387a260b\") " pod="openshift-monitoring/alertmanager-main-0"
Aug 05 17:16:49.311184 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:16:49.311147    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/10e66bf1-f170-4f1b-a4ab-28ff387a260b-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"10e66bf1-f170-4f1b-a4ab-28ff387a260b\") " pod="openshift-monitoring/alertmanager-main-0"
Aug 05 17:16:49.311184 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:16:49.311174    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/10e66bf1-f170-4f1b-a4ab-28ff387a260b-web-config\") pod \"alertmanager-main-0\" (UID: \"10e66bf1-f170-4f1b-a4ab-28ff387a260b\") " pod="openshift-monitoring/alertmanager-main-0"
Aug 05 17:16:49.311333 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:16:49.311200    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/10e66bf1-f170-4f1b-a4ab-28ff387a260b-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"10e66bf1-f170-4f1b-a4ab-28ff387a260b\") " pod="openshift-monitoring/alertmanager-main-0"
Aug 05 17:16:49.311333 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:16:49.311233    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/10e66bf1-f170-4f1b-a4ab-28ff387a260b-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"10e66bf1-f170-4f1b-a4ab-28ff387a260b\") " pod="openshift-monitoring/alertmanager-main-0"
Aug 05 17:16:49.311333 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:16:49.311286    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/10e66bf1-f170-4f1b-a4ab-28ff387a260b-config-volume\") pod \"alertmanager-main-0\" (UID: \"10e66bf1-f170-4f1b-a4ab-28ff387a260b\") " pod="openshift-monitoring/alertmanager-main-0"
Aug 05 17:16:49.311333 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:16:49.311313    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/10e66bf1-f170-4f1b-a4ab-28ff387a260b-config-out\") pod \"alertmanager-main-0\" (UID: \"10e66bf1-f170-4f1b-a4ab-28ff387a260b\") " pod="openshift-monitoring/alertmanager-main-0"
Aug 05 17:16:49.311333 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:16:49.311322    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/10e66bf1-f170-4f1b-a4ab-28ff387a260b-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"10e66bf1-f170-4f1b-a4ab-28ff387a260b\") " pod="openshift-monitoring/alertmanager-main-0"
Aug 05 17:16:49.311591 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:16:49.311336    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/10e66bf1-f170-4f1b-a4ab-28ff387a260b-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"10e66bf1-f170-4f1b-a4ab-28ff387a260b\") " pod="openshift-monitoring/alertmanager-main-0"
Aug 05 17:16:49.311591 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:16:49.311393    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/10e66bf1-f170-4f1b-a4ab-28ff387a260b-tls-assets\") pod \"alertmanager-main-0\" (UID: \"10e66bf1-f170-4f1b-a4ab-28ff387a260b\") " pod="openshift-monitoring/alertmanager-main-0"
Aug 05 17:16:49.311591 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:16:49.311452    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-96kbb\" (UniqueName: \"kubernetes.io/projected/10e66bf1-f170-4f1b-a4ab-28ff387a260b-kube-api-access-96kbb\") pod \"alertmanager-main-0\" (UID: \"10e66bf1-f170-4f1b-a4ab-28ff387a260b\") " pod="openshift-monitoring/alertmanager-main-0"
Aug 05 17:16:49.311591 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:16:49.311481    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/10e66bf1-f170-4f1b-a4ab-28ff387a260b-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"10e66bf1-f170-4f1b-a4ab-28ff387a260b\") " pod="openshift-monitoring/alertmanager-main-0"
Aug 05 17:16:49.312220 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:16:49.311880    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/10e66bf1-f170-4f1b-a4ab-28ff387a260b-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"10e66bf1-f170-4f1b-a4ab-28ff387a260b\") " pod="openshift-monitoring/alertmanager-main-0"
Aug 05 17:16:49.312220 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:16:49.312084    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/10e66bf1-f170-4f1b-a4ab-28ff387a260b-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"10e66bf1-f170-4f1b-a4ab-28ff387a260b\") " pod="openshift-monitoring/alertmanager-main-0"
Aug 05 17:16:49.314086 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:16:49.314059    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/10e66bf1-f170-4f1b-a4ab-28ff387a260b-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"10e66bf1-f170-4f1b-a4ab-28ff387a260b\") " pod="openshift-monitoring/alertmanager-main-0"
Aug 05 17:16:49.314186 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:16:49.314089    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/10e66bf1-f170-4f1b-a4ab-28ff387a260b-web-config\") pod \"alertmanager-main-0\" (UID: \"10e66bf1-f170-4f1b-a4ab-28ff387a260b\") " pod="openshift-monitoring/alertmanager-main-0"
Aug 05 17:16:49.314186 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:16:49.314135    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/10e66bf1-f170-4f1b-a4ab-28ff387a260b-config-volume\") pod \"alertmanager-main-0\" (UID: \"10e66bf1-f170-4f1b-a4ab-28ff387a260b\") " pod="openshift-monitoring/alertmanager-main-0"
Aug 05 17:16:49.314289 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:16:49.314267    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/10e66bf1-f170-4f1b-a4ab-28ff387a260b-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"10e66bf1-f170-4f1b-a4ab-28ff387a260b\") " pod="openshift-monitoring/alertmanager-main-0"
Aug 05 17:16:49.314481 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:16:49.314454    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/10e66bf1-f170-4f1b-a4ab-28ff387a260b-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"10e66bf1-f170-4f1b-a4ab-28ff387a260b\") " pod="openshift-monitoring/alertmanager-main-0"
Aug 05 17:16:49.314564 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:16:49.314534    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/10e66bf1-f170-4f1b-a4ab-28ff387a260b-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"10e66bf1-f170-4f1b-a4ab-28ff387a260b\") " pod="openshift-monitoring/alertmanager-main-0"
Aug 05 17:16:49.314682 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:16:49.314661    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/10e66bf1-f170-4f1b-a4ab-28ff387a260b-tls-assets\") pod \"alertmanager-main-0\" (UID: \"10e66bf1-f170-4f1b-a4ab-28ff387a260b\") " pod="openshift-monitoring/alertmanager-main-0"
Aug 05 17:16:49.315003 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:16:49.314980    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/10e66bf1-f170-4f1b-a4ab-28ff387a260b-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"10e66bf1-f170-4f1b-a4ab-28ff387a260b\") " pod="openshift-monitoring/alertmanager-main-0"
Aug 05 17:16:49.315921 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:16:49.315905    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/10e66bf1-f170-4f1b-a4ab-28ff387a260b-config-out\") pod \"alertmanager-main-0\" (UID: \"10e66bf1-f170-4f1b-a4ab-28ff387a260b\") " pod="openshift-monitoring/alertmanager-main-0"
Aug 05 17:16:49.320720 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:16:49.320705    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-96kbb\" (UniqueName: \"kubernetes.io/projected/10e66bf1-f170-4f1b-a4ab-28ff387a260b-kube-api-access-96kbb\") pod \"alertmanager-main-0\" (UID: \"10e66bf1-f170-4f1b-a4ab-28ff387a260b\") " pod="openshift-monitoring/alertmanager-main-0"
Aug 05 17:16:49.430969 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:16:49.430946    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Aug 05 17:16:49.573983 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:16:49.573905    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Aug 05 17:16:49.577048 ip-10-0-135-2 kubenswrapper[2571]: W0805 17:16:49.577025    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod10e66bf1_f170_4f1b_a4ab_28ff387a260b.slice/crio-c18cde30ba5e3365b882a04e5d53f039233d99be40e65bed1d15c38ac5b239f0 WatchSource:0}: Error finding container c18cde30ba5e3365b882a04e5d53f039233d99be40e65bed1d15c38ac5b239f0: Status 404 returned error can't find the container with id c18cde30ba5e3365b882a04e5d53f039233d99be40e65bed1d15c38ac5b239f0
Aug 05 17:16:50.043977 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:16:50.043944    2571 generic.go:356] "Generic (PLEG): container finished" podID="10e66bf1-f170-4f1b-a4ab-28ff387a260b" containerID="029fe3b0d1b162182e610aa8e7915fda23f3553a0515d6efae55afe70968207a" exitCode=0
Aug 05 17:16:50.044438 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:16:50.043998    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"10e66bf1-f170-4f1b-a4ab-28ff387a260b","Type":"ContainerDied","Data":"029fe3b0d1b162182e610aa8e7915fda23f3553a0515d6efae55afe70968207a"}
Aug 05 17:16:50.044438 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:16:50.044066    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"10e66bf1-f170-4f1b-a4ab-28ff387a260b","Type":"ContainerStarted","Data":"c18cde30ba5e3365b882a04e5d53f039233d99be40e65bed1d15c38ac5b239f0"}
Aug 05 17:16:51.051754 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:16:51.051717    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"10e66bf1-f170-4f1b-a4ab-28ff387a260b","Type":"ContainerStarted","Data":"abfbd39ddb71f4ea9323e576a4a857ce914e97ca16030660e84baf42b06c55c3"}
Aug 05 17:16:51.052212 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:16:51.051833    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"10e66bf1-f170-4f1b-a4ab-28ff387a260b","Type":"ContainerStarted","Data":"f163c0cea943c90ff8c4b84a4ca65f9534e5ca9fcf6186729f9cafce01359b31"}
Aug 05 17:16:51.052212 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:16:51.051884    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"10e66bf1-f170-4f1b-a4ab-28ff387a260b","Type":"ContainerStarted","Data":"b47574c6a23eaa9e3dbba18236223fde09edcb82e3a390ce13f7d3f136e0183d"}
Aug 05 17:16:51.052212 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:16:51.051924    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"10e66bf1-f170-4f1b-a4ab-28ff387a260b","Type":"ContainerStarted","Data":"7bb434dc230f4e0d0eff54a996b3bd01ac1c569e2b573d8bc5dbbc512fd1229f"}
Aug 05 17:16:51.052212 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:16:51.051961    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"10e66bf1-f170-4f1b-a4ab-28ff387a260b","Type":"ContainerStarted","Data":"d17c536a0c62e03a153afe45f237e2067d4a65ac49c448ae0984f41c46d7bbf7"}
Aug 05 17:16:51.052212 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:16:51.051998    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"10e66bf1-f170-4f1b-a4ab-28ff387a260b","Type":"ContainerStarted","Data":"dc84b8f8a20a4e6ea86ca08d5e095210989ccc9dabad7c8dec3b994f1d2b7818"}
Aug 05 17:16:51.089548 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:16:51.089504    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=2.073696806 podStartE2EDuration="2.089489295s" totalImagesPullingTime="0s" totalInitContainerRuntime="15.792489ms" isStatefulPod=false podCreationTimestamp="2026-08-05 17:16:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-05 17:16:51.08797284 +0000 UTC m=+238.455339182" watchObservedRunningTime="2026-08-05 17:16:51.089489295 +0000 UTC m=+238.456855637"
Aug 05 17:16:51.803120 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:16:51.803067    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-9df4c78b4-8mg98"]
Aug 05 17:16:51.806330 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:16:51.806314    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-9df4c78b4-8mg98"
Aug 05 17:16:51.811099 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:16:51.811072    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\""
Aug 05 17:16:51.811251 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:16:51.811234    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\""
Aug 05 17:16:51.811532 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:16:51.811513    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\""
Aug 05 17:16:51.811739 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:16:51.811712    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-bsn4n\""
Aug 05 17:16:51.811739 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:16:51.811730    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\""
Aug 05 17:16:51.811888 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:16:51.811712    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\""
Aug 05 17:16:51.823033 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:16:51.822996    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-8i12ta5c71j38\""
Aug 05 17:16:51.829338 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:16:51.829319    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-9df4c78b4-8mg98"]
Aug 05 17:16:51.933182 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:16:51.933152    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f72d410a-a6cf-4f12-8d00-917f1550c160-metrics-client-ca\") pod \"telemeter-client-9df4c78b4-8mg98\" (UID: \"f72d410a-a6cf-4f12-8d00-917f1550c160\") " pod="openshift-monitoring/telemeter-client-9df4c78b4-8mg98"
Aug 05 17:16:51.933349 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:16:51.933189    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f72d410a-a6cf-4f12-8d00-917f1550c160-serving-certs-ca-bundle\") pod \"telemeter-client-9df4c78b4-8mg98\" (UID: \"f72d410a-a6cf-4f12-8d00-917f1550c160\") " pod="openshift-monitoring/telemeter-client-9df4c78b4-8mg98"
Aug 05 17:16:51.933349 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:16:51.933208    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f72d410a-a6cf-4f12-8d00-917f1550c160-telemeter-trusted-ca-bundle\") pod \"telemeter-client-9df4c78b4-8mg98\" (UID: \"f72d410a-a6cf-4f12-8d00-917f1550c160\") " pod="openshift-monitoring/telemeter-client-9df4c78b4-8mg98"
Aug 05 17:16:51.933349 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:16:51.933264    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mkxmc\" (UniqueName: \"kubernetes.io/projected/f72d410a-a6cf-4f12-8d00-917f1550c160-kube-api-access-mkxmc\") pod \"telemeter-client-9df4c78b4-8mg98\" (UID: \"f72d410a-a6cf-4f12-8d00-917f1550c160\") " pod="openshift-monitoring/telemeter-client-9df4c78b4-8mg98"
Aug 05 17:16:51.933349 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:16:51.933281    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/f72d410a-a6cf-4f12-8d00-917f1550c160-secret-telemeter-client\") pod \"telemeter-client-9df4c78b4-8mg98\" (UID: \"f72d410a-a6cf-4f12-8d00-917f1550c160\") " pod="openshift-monitoring/telemeter-client-9df4c78b4-8mg98"
Aug 05 17:16:51.933349 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:16:51.933308    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/f72d410a-a6cf-4f12-8d00-917f1550c160-telemeter-client-tls\") pod \"telemeter-client-9df4c78b4-8mg98\" (UID: \"f72d410a-a6cf-4f12-8d00-917f1550c160\") " pod="openshift-monitoring/telemeter-client-9df4c78b4-8mg98"
Aug 05 17:16:51.933349 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:16:51.933326    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f72d410a-a6cf-4f12-8d00-917f1550c160-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-9df4c78b4-8mg98\" (UID: \"f72d410a-a6cf-4f12-8d00-917f1550c160\") " pod="openshift-monitoring/telemeter-client-9df4c78b4-8mg98"
Aug 05 17:16:51.933747 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:16:51.933355    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/f72d410a-a6cf-4f12-8d00-917f1550c160-federate-client-tls\") pod \"telemeter-client-9df4c78b4-8mg98\" (UID: \"f72d410a-a6cf-4f12-8d00-917f1550c160\") " pod="openshift-monitoring/telemeter-client-9df4c78b4-8mg98"
Aug 05 17:16:52.034615 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:16:52.034569    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mkxmc\" (UniqueName: \"kubernetes.io/projected/f72d410a-a6cf-4f12-8d00-917f1550c160-kube-api-access-mkxmc\") pod \"telemeter-client-9df4c78b4-8mg98\" (UID: \"f72d410a-a6cf-4f12-8d00-917f1550c160\") " pod="openshift-monitoring/telemeter-client-9df4c78b4-8mg98"
Aug 05 17:16:52.034752 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:16:52.034622    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/f72d410a-a6cf-4f12-8d00-917f1550c160-secret-telemeter-client\") pod \"telemeter-client-9df4c78b4-8mg98\" (UID: \"f72d410a-a6cf-4f12-8d00-917f1550c160\") " pod="openshift-monitoring/telemeter-client-9df4c78b4-8mg98"
Aug 05 17:16:52.034752 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:16:52.034652    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/f72d410a-a6cf-4f12-8d00-917f1550c160-telemeter-client-tls\") pod \"telemeter-client-9df4c78b4-8mg98\" (UID: \"f72d410a-a6cf-4f12-8d00-917f1550c160\") " pod="openshift-monitoring/telemeter-client-9df4c78b4-8mg98"
Aug 05 17:16:52.034752 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:16:52.034680    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f72d410a-a6cf-4f12-8d00-917f1550c160-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-9df4c78b4-8mg98\" (UID: \"f72d410a-a6cf-4f12-8d00-917f1550c160\") " pod="openshift-monitoring/telemeter-client-9df4c78b4-8mg98"
Aug 05 17:16:52.034973 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:16:52.034945    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/f72d410a-a6cf-4f12-8d00-917f1550c160-federate-client-tls\") pod \"telemeter-client-9df4c78b4-8mg98\" (UID: \"f72d410a-a6cf-4f12-8d00-917f1550c160\") " pod="openshift-monitoring/telemeter-client-9df4c78b4-8mg98"
Aug 05 17:16:52.035054 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:16:52.035004    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f72d410a-a6cf-4f12-8d00-917f1550c160-metrics-client-ca\") pod \"telemeter-client-9df4c78b4-8mg98\" (UID: \"f72d410a-a6cf-4f12-8d00-917f1550c160\") " pod="openshift-monitoring/telemeter-client-9df4c78b4-8mg98"
Aug 05 17:16:52.035054 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:16:52.035045    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f72d410a-a6cf-4f12-8d00-917f1550c160-serving-certs-ca-bundle\") pod \"telemeter-client-9df4c78b4-8mg98\" (UID: \"f72d410a-a6cf-4f12-8d00-917f1550c160\") " pod="openshift-monitoring/telemeter-client-9df4c78b4-8mg98"
Aug 05 17:16:52.035229 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:16:52.035202    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f72d410a-a6cf-4f12-8d00-917f1550c160-telemeter-trusted-ca-bundle\") pod \"telemeter-client-9df4c78b4-8mg98\" (UID: \"f72d410a-a6cf-4f12-8d00-917f1550c160\") " pod="openshift-monitoring/telemeter-client-9df4c78b4-8mg98"
Aug 05 17:16:52.035769 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:16:52.035740    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f72d410a-a6cf-4f12-8d00-917f1550c160-metrics-client-ca\") pod \"telemeter-client-9df4c78b4-8mg98\" (UID: \"f72d410a-a6cf-4f12-8d00-917f1550c160\") " pod="openshift-monitoring/telemeter-client-9df4c78b4-8mg98"
Aug 05 17:16:52.035859 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:16:52.035831    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f72d410a-a6cf-4f12-8d00-917f1550c160-serving-certs-ca-bundle\") pod \"telemeter-client-9df4c78b4-8mg98\" (UID: \"f72d410a-a6cf-4f12-8d00-917f1550c160\") " pod="openshift-monitoring/telemeter-client-9df4c78b4-8mg98"
Aug 05 17:16:52.035997 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:16:52.035979    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f72d410a-a6cf-4f12-8d00-917f1550c160-telemeter-trusted-ca-bundle\") pod \"telemeter-client-9df4c78b4-8mg98\" (UID: \"f72d410a-a6cf-4f12-8d00-917f1550c160\") " pod="openshift-monitoring/telemeter-client-9df4c78b4-8mg98"
Aug 05 17:16:52.037298 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:16:52.037271    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f72d410a-a6cf-4f12-8d00-917f1550c160-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-9df4c78b4-8mg98\" (UID: \"f72d410a-a6cf-4f12-8d00-917f1550c160\") " pod="openshift-monitoring/telemeter-client-9df4c78b4-8mg98"
Aug 05 17:16:52.037402 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:16:52.037307    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/f72d410a-a6cf-4f12-8d00-917f1550c160-secret-telemeter-client\") pod \"telemeter-client-9df4c78b4-8mg98\" (UID: \"f72d410a-a6cf-4f12-8d00-917f1550c160\") " pod="openshift-monitoring/telemeter-client-9df4c78b4-8mg98"
Aug 05 17:16:52.037649 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:16:52.037628    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/f72d410a-a6cf-4f12-8d00-917f1550c160-federate-client-tls\") pod \"telemeter-client-9df4c78b4-8mg98\" (UID: \"f72d410a-a6cf-4f12-8d00-917f1550c160\") " pod="openshift-monitoring/telemeter-client-9df4c78b4-8mg98"
Aug 05 17:16:52.037688 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:16:52.037631    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/f72d410a-a6cf-4f12-8d00-917f1550c160-telemeter-client-tls\") pod \"telemeter-client-9df4c78b4-8mg98\" (UID: \"f72d410a-a6cf-4f12-8d00-917f1550c160\") " pod="openshift-monitoring/telemeter-client-9df4c78b4-8mg98"
Aug 05 17:16:52.044047 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:16:52.044028    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mkxmc\" (UniqueName: \"kubernetes.io/projected/f72d410a-a6cf-4f12-8d00-917f1550c160-kube-api-access-mkxmc\") pod \"telemeter-client-9df4c78b4-8mg98\" (UID: \"f72d410a-a6cf-4f12-8d00-917f1550c160\") " pod="openshift-monitoring/telemeter-client-9df4c78b4-8mg98"
Aug 05 17:16:52.116539 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:16:52.116485    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-9df4c78b4-8mg98"
Aug 05 17:16:52.254018 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:16:52.253989    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-9df4c78b4-8mg98"]
Aug 05 17:16:52.257654 ip-10-0-135-2 kubenswrapper[2571]: W0805 17:16:52.257630    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf72d410a_a6cf_4f12_8d00_917f1550c160.slice/crio-baf1b805b25acbcad17db70dd61d3e64d400ceba9fef8492de5a8a7e66028a4b WatchSource:0}: Error finding container baf1b805b25acbcad17db70dd61d3e64d400ceba9fef8492de5a8a7e66028a4b: Status 404 returned error can't find the container with id baf1b805b25acbcad17db70dd61d3e64d400ceba9fef8492de5a8a7e66028a4b
Aug 05 17:16:53.062075 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:16:53.062044    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-9df4c78b4-8mg98" event={"ID":"f72d410a-a6cf-4f12-8d00-917f1550c160","Type":"ContainerStarted","Data":"baf1b805b25acbcad17db70dd61d3e64d400ceba9fef8492de5a8a7e66028a4b"}
Aug 05 17:16:54.067474 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:16:54.067370    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-9df4c78b4-8mg98" event={"ID":"f72d410a-a6cf-4f12-8d00-917f1550c160","Type":"ContainerStarted","Data":"1f454c8986355509e2bffdb966acfd6f04757feb23fb707d4d839f1005e61604"}
Aug 05 17:16:54.067474 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:16:54.067454    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-9df4c78b4-8mg98" event={"ID":"f72d410a-a6cf-4f12-8d00-917f1550c160","Type":"ContainerStarted","Data":"fe0e2fbbed804065c85ccb1497a5048f9fc2da809fabc2dc6180ed188522cbb0"}
Aug 05 17:16:54.067946 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:16:54.067479    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-9df4c78b4-8mg98" event={"ID":"f72d410a-a6cf-4f12-8d00-917f1550c160","Type":"ContainerStarted","Data":"46d1edfe98fe51bc1a1ad83403b32c74775b3e85f276c3fa343b76a1de36efe1"}
Aug 05 17:16:54.270494 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:16:54.270440    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-9df4c78b4-8mg98" podStartSLOduration=1.759462209 podStartE2EDuration="3.270424458s" totalImagesPullingTime="1.510962249s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-05 17:16:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-05 17:16:54.103920849 +0000 UTC m=+241.471287192" watchObservedRunningTime="2026-08-05 17:16:54.270424458 +0000 UTC m=+241.637790798"
Aug 05 17:16:54.271413 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:16:54.271396    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-9685bddcf-bflgp"]
Aug 05 17:16:54.275712 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:16:54.275697    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-9685bddcf-bflgp"
Aug 05 17:16:54.294367 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:16:54.294340    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-9685bddcf-bflgp"]
Aug 05 17:16:54.351837 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:16:54.351769    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-25l95\" (UniqueName: \"kubernetes.io/projected/13fe9531-744f-414a-84d9-c37c2f3c2109-kube-api-access-25l95\") pod \"console-9685bddcf-bflgp\" (UID: \"13fe9531-744f-414a-84d9-c37c2f3c2109\") " pod="openshift-console/console-9685bddcf-bflgp"
Aug 05 17:16:54.351837 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:16:54.351807    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/13fe9531-744f-414a-84d9-c37c2f3c2109-oauth-serving-cert\") pod \"console-9685bddcf-bflgp\" (UID: \"13fe9531-744f-414a-84d9-c37c2f3c2109\") " pod="openshift-console/console-9685bddcf-bflgp"
Aug 05 17:16:54.351988 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:16:54.351849    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/13fe9531-744f-414a-84d9-c37c2f3c2109-console-config\") pod \"console-9685bddcf-bflgp\" (UID: \"13fe9531-744f-414a-84d9-c37c2f3c2109\") " pod="openshift-console/console-9685bddcf-bflgp"
Aug 05 17:16:54.351988 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:16:54.351887    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/13fe9531-744f-414a-84d9-c37c2f3c2109-console-oauth-config\") pod \"console-9685bddcf-bflgp\" (UID: \"13fe9531-744f-414a-84d9-c37c2f3c2109\") " pod="openshift-console/console-9685bddcf-bflgp"
Aug 05 17:16:54.351988 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:16:54.351920    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/13fe9531-744f-414a-84d9-c37c2f3c2109-trusted-ca-bundle\") pod \"console-9685bddcf-bflgp\" (UID: \"13fe9531-744f-414a-84d9-c37c2f3c2109\") " pod="openshift-console/console-9685bddcf-bflgp"
Aug 05 17:16:54.351988 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:16:54.351938    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/13fe9531-744f-414a-84d9-c37c2f3c2109-service-ca\") pod \"console-9685bddcf-bflgp\" (UID: \"13fe9531-744f-414a-84d9-c37c2f3c2109\") " pod="openshift-console/console-9685bddcf-bflgp"
Aug 05 17:16:54.351988 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:16:54.351972    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/13fe9531-744f-414a-84d9-c37c2f3c2109-console-serving-cert\") pod \"console-9685bddcf-bflgp\" (UID: \"13fe9531-744f-414a-84d9-c37c2f3c2109\") " pod="openshift-console/console-9685bddcf-bflgp"
Aug 05 17:16:54.452303 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:16:54.452270    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/13fe9531-744f-414a-84d9-c37c2f3c2109-console-config\") pod \"console-9685bddcf-bflgp\" (UID: \"13fe9531-744f-414a-84d9-c37c2f3c2109\") " pod="openshift-console/console-9685bddcf-bflgp"
Aug 05 17:16:54.452303 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:16:54.452301    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/13fe9531-744f-414a-84d9-c37c2f3c2109-console-oauth-config\") pod \"console-9685bddcf-bflgp\" (UID: \"13fe9531-744f-414a-84d9-c37c2f3c2109\") " pod="openshift-console/console-9685bddcf-bflgp"
Aug 05 17:16:54.452507 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:16:54.452316    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/13fe9531-744f-414a-84d9-c37c2f3c2109-trusted-ca-bundle\") pod \"console-9685bddcf-bflgp\" (UID: \"13fe9531-744f-414a-84d9-c37c2f3c2109\") " pod="openshift-console/console-9685bddcf-bflgp"
Aug 05 17:16:54.452507 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:16:54.452333    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/13fe9531-744f-414a-84d9-c37c2f3c2109-service-ca\") pod \"console-9685bddcf-bflgp\" (UID: \"13fe9531-744f-414a-84d9-c37c2f3c2109\") " pod="openshift-console/console-9685bddcf-bflgp"
Aug 05 17:16:54.452507 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:16:54.452408    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/13fe9531-744f-414a-84d9-c37c2f3c2109-console-serving-cert\") pod \"console-9685bddcf-bflgp\" (UID: \"13fe9531-744f-414a-84d9-c37c2f3c2109\") " pod="openshift-console/console-9685bddcf-bflgp"
Aug 05 17:16:54.452507 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:16:54.452437    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-25l95\" (UniqueName: \"kubernetes.io/projected/13fe9531-744f-414a-84d9-c37c2f3c2109-kube-api-access-25l95\") pod \"console-9685bddcf-bflgp\" (UID: \"13fe9531-744f-414a-84d9-c37c2f3c2109\") " pod="openshift-console/console-9685bddcf-bflgp"
Aug 05 17:16:54.452507 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:16:54.452475    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/13fe9531-744f-414a-84d9-c37c2f3c2109-oauth-serving-cert\") pod \"console-9685bddcf-bflgp\" (UID: \"13fe9531-744f-414a-84d9-c37c2f3c2109\") " pod="openshift-console/console-9685bddcf-bflgp"
Aug 05 17:16:54.453137 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:16:54.453109    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/13fe9531-744f-414a-84d9-c37c2f3c2109-console-config\") pod \"console-9685bddcf-bflgp\" (UID: \"13fe9531-744f-414a-84d9-c37c2f3c2109\") " pod="openshift-console/console-9685bddcf-bflgp"
Aug 05 17:16:54.453242 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:16:54.453109    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/13fe9531-744f-414a-84d9-c37c2f3c2109-service-ca\") pod \"console-9685bddcf-bflgp\" (UID: \"13fe9531-744f-414a-84d9-c37c2f3c2109\") " pod="openshift-console/console-9685bddcf-bflgp"
Aug 05 17:16:54.453242 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:16:54.453200    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/13fe9531-744f-414a-84d9-c37c2f3c2109-trusted-ca-bundle\") pod \"console-9685bddcf-bflgp\" (UID: \"13fe9531-744f-414a-84d9-c37c2f3c2109\") " pod="openshift-console/console-9685bddcf-bflgp"
Aug 05 17:16:54.453242 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:16:54.453208    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/13fe9531-744f-414a-84d9-c37c2f3c2109-oauth-serving-cert\") pod \"console-9685bddcf-bflgp\" (UID: \"13fe9531-744f-414a-84d9-c37c2f3c2109\") " pod="openshift-console/console-9685bddcf-bflgp"
Aug 05 17:16:54.454803 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:16:54.454785    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/13fe9531-744f-414a-84d9-c37c2f3c2109-console-oauth-config\") pod \"console-9685bddcf-bflgp\" (UID: \"13fe9531-744f-414a-84d9-c37c2f3c2109\") " pod="openshift-console/console-9685bddcf-bflgp"
Aug 05 17:16:54.455042 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:16:54.455013    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/13fe9531-744f-414a-84d9-c37c2f3c2109-console-serving-cert\") pod \"console-9685bddcf-bflgp\" (UID: \"13fe9531-744f-414a-84d9-c37c2f3c2109\") " pod="openshift-console/console-9685bddcf-bflgp"
Aug 05 17:16:54.463000 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:16:54.462978    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-25l95\" (UniqueName: \"kubernetes.io/projected/13fe9531-744f-414a-84d9-c37c2f3c2109-kube-api-access-25l95\") pod \"console-9685bddcf-bflgp\" (UID: \"13fe9531-744f-414a-84d9-c37c2f3c2109\") " pod="openshift-console/console-9685bddcf-bflgp"
Aug 05 17:16:54.584909 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:16:54.584882    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-9685bddcf-bflgp"
Aug 05 17:16:54.711883 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:16:54.711744    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-9685bddcf-bflgp"]
Aug 05 17:16:54.714446 ip-10-0-135-2 kubenswrapper[2571]: W0805 17:16:54.714416    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod13fe9531_744f_414a_84d9_c37c2f3c2109.slice/crio-e6c6da8c5233016fbca65bedd8ad866695e99efc5283756d22ae77f699f81c3d WatchSource:0}: Error finding container e6c6da8c5233016fbca65bedd8ad866695e99efc5283756d22ae77f699f81c3d: Status 404 returned error can't find the container with id e6c6da8c5233016fbca65bedd8ad866695e99efc5283756d22ae77f699f81c3d
Aug 05 17:16:55.072822 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:16:55.072792    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-9685bddcf-bflgp" event={"ID":"13fe9531-744f-414a-84d9-c37c2f3c2109","Type":"ContainerStarted","Data":"56b288078ff8c2ccf061850fbcb30beb0f482f1ada45fe7db074a06f13517055"}
Aug 05 17:16:55.073207 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:16:55.072841    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-9685bddcf-bflgp" event={"ID":"13fe9531-744f-414a-84d9-c37c2f3c2109","Type":"ContainerStarted","Data":"e6c6da8c5233016fbca65bedd8ad866695e99efc5283756d22ae77f699f81c3d"}
Aug 05 17:16:55.101585 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:16:55.101545    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-9685bddcf-bflgp" podStartSLOduration=1.101530009 podStartE2EDuration="1.101530009s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-05 17:16:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-05 17:16:55.101193275 +0000 UTC m=+242.468559619" watchObservedRunningTime="2026-08-05 17:16:55.101530009 +0000 UTC m=+242.468896351"
Aug 05 17:17:04.585850 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:17:04.585812    2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-9685bddcf-bflgp"
Aug 05 17:17:04.586288 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:17:04.585887    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-9685bddcf-bflgp"
Aug 05 17:17:04.590595 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:17:04.590570    2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-9685bddcf-bflgp"
Aug 05 17:17:05.114661 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:17:05.114634    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-9685bddcf-bflgp"
Aug 05 17:17:05.167647 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:17:05.167616    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5d799cddfd-fhl99"]
Aug 05 17:17:30.188009 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:17:30.187969    2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-5d799cddfd-fhl99" podUID="736198e4-bb0b-45a3-8500-050c572dded4" containerName="console" containerID="cri-o://8e67af0f3aaa01205ac5d6474b9a86a34a6bdd6a931039a07832e0715be94e2b" gracePeriod=15
Aug 05 17:17:30.425527 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:17:30.425499    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5d799cddfd-fhl99"
Aug 05 17:17:30.518598 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:17:30.518572    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/736198e4-bb0b-45a3-8500-050c572dded4-console-oauth-config\") pod \"736198e4-bb0b-45a3-8500-050c572dded4\" (UID: \"736198e4-bb0b-45a3-8500-050c572dded4\") "
Aug 05 17:17:30.518734 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:17:30.518614    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/736198e4-bb0b-45a3-8500-050c572dded4-console-serving-cert\") pod \"736198e4-bb0b-45a3-8500-050c572dded4\" (UID: \"736198e4-bb0b-45a3-8500-050c572dded4\") "
Aug 05 17:17:30.518734 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:17:30.518634    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/736198e4-bb0b-45a3-8500-050c572dded4-service-ca\") pod \"736198e4-bb0b-45a3-8500-050c572dded4\" (UID: \"736198e4-bb0b-45a3-8500-050c572dded4\") "
Aug 05 17:17:30.518734 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:17:30.518664    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/736198e4-bb0b-45a3-8500-050c572dded4-trusted-ca-bundle\") pod \"736198e4-bb0b-45a3-8500-050c572dded4\" (UID: \"736198e4-bb0b-45a3-8500-050c572dded4\") "
Aug 05 17:17:30.518734 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:17:30.518705    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/736198e4-bb0b-45a3-8500-050c572dded4-oauth-serving-cert\") pod \"736198e4-bb0b-45a3-8500-050c572dded4\" (UID: \"736198e4-bb0b-45a3-8500-050c572dded4\") "
Aug 05 17:17:30.518921 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:17:30.518788    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/736198e4-bb0b-45a3-8500-050c572dded4-console-config\") pod \"736198e4-bb0b-45a3-8500-050c572dded4\" (UID: \"736198e4-bb0b-45a3-8500-050c572dded4\") "
Aug 05 17:17:30.518921 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:17:30.518835    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dv72k\" (UniqueName: \"kubernetes.io/projected/736198e4-bb0b-45a3-8500-050c572dded4-kube-api-access-dv72k\") pod \"736198e4-bb0b-45a3-8500-050c572dded4\" (UID: \"736198e4-bb0b-45a3-8500-050c572dded4\") "
Aug 05 17:17:30.519523 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:17:30.519328    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/736198e4-bb0b-45a3-8500-050c572dded4-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "736198e4-bb0b-45a3-8500-050c572dded4" (UID: "736198e4-bb0b-45a3-8500-050c572dded4"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 05 17:17:30.519645 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:17:30.519535    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/736198e4-bb0b-45a3-8500-050c572dded4-service-ca" (OuterVolumeSpecName: "service-ca") pod "736198e4-bb0b-45a3-8500-050c572dded4" (UID: "736198e4-bb0b-45a3-8500-050c572dded4"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 05 17:17:30.519645 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:17:30.519598    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/736198e4-bb0b-45a3-8500-050c572dded4-console-config" (OuterVolumeSpecName: "console-config") pod "736198e4-bb0b-45a3-8500-050c572dded4" (UID: "736198e4-bb0b-45a3-8500-050c572dded4"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 05 17:17:30.521424 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:17:30.519790    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/736198e4-bb0b-45a3-8500-050c572dded4-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "736198e4-bb0b-45a3-8500-050c572dded4" (UID: "736198e4-bb0b-45a3-8500-050c572dded4"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 05 17:17:30.525462 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:17:30.525430    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/736198e4-bb0b-45a3-8500-050c572dded4-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "736198e4-bb0b-45a3-8500-050c572dded4" (UID: "736198e4-bb0b-45a3-8500-050c572dded4"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 05 17:17:30.525462 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:17:30.525445    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/736198e4-bb0b-45a3-8500-050c572dded4-kube-api-access-dv72k" (OuterVolumeSpecName: "kube-api-access-dv72k") pod "736198e4-bb0b-45a3-8500-050c572dded4" (UID: "736198e4-bb0b-45a3-8500-050c572dded4"). InnerVolumeSpecName "kube-api-access-dv72k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 05 17:17:30.526093 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:17:30.526073    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/736198e4-bb0b-45a3-8500-050c572dded4-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "736198e4-bb0b-45a3-8500-050c572dded4" (UID: "736198e4-bb0b-45a3-8500-050c572dded4"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 05 17:17:30.620700 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:17:30.620675    2571 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/736198e4-bb0b-45a3-8500-050c572dded4-console-config\") on node \"ip-10-0-135-2.ec2.internal\" DevicePath \"\""
Aug 05 17:17:30.620700 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:17:30.620699    2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dv72k\" (UniqueName: \"kubernetes.io/projected/736198e4-bb0b-45a3-8500-050c572dded4-kube-api-access-dv72k\") on node \"ip-10-0-135-2.ec2.internal\" DevicePath \"\""
Aug 05 17:17:30.620825 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:17:30.620708    2571 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/736198e4-bb0b-45a3-8500-050c572dded4-console-oauth-config\") on node \"ip-10-0-135-2.ec2.internal\" DevicePath \"\""
Aug 05 17:17:30.620825 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:17:30.620717    2571 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/736198e4-bb0b-45a3-8500-050c572dded4-console-serving-cert\") on node \"ip-10-0-135-2.ec2.internal\" DevicePath \"\""
Aug 05 17:17:30.620825 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:17:30.620724    2571 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/736198e4-bb0b-45a3-8500-050c572dded4-service-ca\") on node \"ip-10-0-135-2.ec2.internal\" DevicePath \"\""
Aug 05 17:17:30.620825 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:17:30.620732    2571 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/736198e4-bb0b-45a3-8500-050c572dded4-trusted-ca-bundle\") on node \"ip-10-0-135-2.ec2.internal\" DevicePath \"\""
Aug 05 17:17:30.620825 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:17:30.620740    2571 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/736198e4-bb0b-45a3-8500-050c572dded4-oauth-serving-cert\") on node \"ip-10-0-135-2.ec2.internal\" DevicePath \"\""
Aug 05 17:17:31.204360 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:17:31.204322    2571 generic.go:356] "Generic (PLEG): container finished" podID="736198e4-bb0b-45a3-8500-050c572dded4" containerID="8e67af0f3aaa01205ac5d6474b9a86a34a6bdd6a931039a07832e0715be94e2b" exitCode=0
Aug 05 17:17:31.204827 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:17:31.204403    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5d799cddfd-fhl99"
Aug 05 17:17:31.204827 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:17:31.204402    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5d799cddfd-fhl99" event={"ID":"736198e4-bb0b-45a3-8500-050c572dded4","Type":"ContainerDied","Data":"8e67af0f3aaa01205ac5d6474b9a86a34a6bdd6a931039a07832e0715be94e2b"}
Aug 05 17:17:31.204827 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:17:31.204454    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5d799cddfd-fhl99" event={"ID":"736198e4-bb0b-45a3-8500-050c572dded4","Type":"ContainerDied","Data":"b5ef9bff55dd19d8703b4dfce4f7071a0e985c2562630047d468e21c8f6ec386"}
Aug 05 17:17:31.204827 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:17:31.204475    2571 scope.go:117] "RemoveContainer" containerID="8e67af0f3aaa01205ac5d6474b9a86a34a6bdd6a931039a07832e0715be94e2b"
Aug 05 17:17:31.212317 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:17:31.212300    2571 scope.go:117] "RemoveContainer" containerID="8e67af0f3aaa01205ac5d6474b9a86a34a6bdd6a931039a07832e0715be94e2b"
Aug 05 17:17:31.212580 ip-10-0-135-2 kubenswrapper[2571]: E0805 17:17:31.212562    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8e67af0f3aaa01205ac5d6474b9a86a34a6bdd6a931039a07832e0715be94e2b\": container with ID starting with 8e67af0f3aaa01205ac5d6474b9a86a34a6bdd6a931039a07832e0715be94e2b not found: ID does not exist" containerID="8e67af0f3aaa01205ac5d6474b9a86a34a6bdd6a931039a07832e0715be94e2b"
Aug 05 17:17:31.212632 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:17:31.212589    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8e67af0f3aaa01205ac5d6474b9a86a34a6bdd6a931039a07832e0715be94e2b"} err="failed to get container status \"8e67af0f3aaa01205ac5d6474b9a86a34a6bdd6a931039a07832e0715be94e2b\": rpc error: code = NotFound desc = could not find container \"8e67af0f3aaa01205ac5d6474b9a86a34a6bdd6a931039a07832e0715be94e2b\": container with ID starting with 8e67af0f3aaa01205ac5d6474b9a86a34a6bdd6a931039a07832e0715be94e2b not found: ID does not exist"
Aug 05 17:17:31.221090 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:17:31.221072    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5d799cddfd-fhl99"]
Aug 05 17:17:31.226360 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:17:31.226340    2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-5d799cddfd-fhl99"]
Aug 05 17:17:33.136666 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:17:33.136632    2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="736198e4-bb0b-45a3-8500-050c572dded4" path="/var/lib/kubelet/pods/736198e4-bb0b-45a3-8500-050c572dded4/volumes"
Aug 05 17:17:52.983508 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:17:52.983482    2571 kubelet.go:1648] "Image garbage collection succeeded"
Aug 05 17:18:37.128760 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:18:37.128722    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-85fcbc755-6xmdc"]
Aug 05 17:18:37.129256 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:18:37.129155    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="736198e4-bb0b-45a3-8500-050c572dded4" containerName="console"
Aug 05 17:18:37.129256 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:18:37.129175    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="736198e4-bb0b-45a3-8500-050c572dded4" containerName="console"
Aug 05 17:18:37.129256 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:18:37.129243    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="736198e4-bb0b-45a3-8500-050c572dded4" containerName="console"
Aug 05 17:18:37.132245 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:18:37.132218    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-85fcbc755-6xmdc"
Aug 05 17:18:37.143384 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:18:37.143350    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-85fcbc755-6xmdc"]
Aug 05 17:18:37.194046 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:18:37.194023    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/825969e8-cbe8-448d-a808-3153afda1962-console-serving-cert\") pod \"console-85fcbc755-6xmdc\" (UID: \"825969e8-cbe8-448d-a808-3153afda1962\") " pod="openshift-console/console-85fcbc755-6xmdc"
Aug 05 17:18:37.194139 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:18:37.194050    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/825969e8-cbe8-448d-a808-3153afda1962-trusted-ca-bundle\") pod \"console-85fcbc755-6xmdc\" (UID: \"825969e8-cbe8-448d-a808-3153afda1962\") " pod="openshift-console/console-85fcbc755-6xmdc"
Aug 05 17:18:37.194139 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:18:37.194069    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/825969e8-cbe8-448d-a808-3153afda1962-oauth-serving-cert\") pod \"console-85fcbc755-6xmdc\" (UID: \"825969e8-cbe8-448d-a808-3153afda1962\") " pod="openshift-console/console-85fcbc755-6xmdc"
Aug 05 17:18:37.194139 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:18:37.194093    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/825969e8-cbe8-448d-a808-3153afda1962-console-oauth-config\") pod \"console-85fcbc755-6xmdc\" (UID: \"825969e8-cbe8-448d-a808-3153afda1962\") " pod="openshift-console/console-85fcbc755-6xmdc"
Aug 05 17:18:37.194241 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:18:37.194151    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zw9wj\" (UniqueName: \"kubernetes.io/projected/825969e8-cbe8-448d-a808-3153afda1962-kube-api-access-zw9wj\") pod \"console-85fcbc755-6xmdc\" (UID: \"825969e8-cbe8-448d-a808-3153afda1962\") " pod="openshift-console/console-85fcbc755-6xmdc"
Aug 05 17:18:37.194241 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:18:37.194194    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/825969e8-cbe8-448d-a808-3153afda1962-service-ca\") pod \"console-85fcbc755-6xmdc\" (UID: \"825969e8-cbe8-448d-a808-3153afda1962\") " pod="openshift-console/console-85fcbc755-6xmdc"
Aug 05 17:18:37.194241 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:18:37.194223    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/825969e8-cbe8-448d-a808-3153afda1962-console-config\") pod \"console-85fcbc755-6xmdc\" (UID: \"825969e8-cbe8-448d-a808-3153afda1962\") " pod="openshift-console/console-85fcbc755-6xmdc"
Aug 05 17:18:37.295173 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:18:37.295151    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/825969e8-cbe8-448d-a808-3153afda1962-console-serving-cert\") pod \"console-85fcbc755-6xmdc\" (UID: \"825969e8-cbe8-448d-a808-3153afda1962\") " pod="openshift-console/console-85fcbc755-6xmdc"
Aug 05 17:18:37.295263 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:18:37.295179    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/825969e8-cbe8-448d-a808-3153afda1962-trusted-ca-bundle\") pod \"console-85fcbc755-6xmdc\" (UID: \"825969e8-cbe8-448d-a808-3153afda1962\") " pod="openshift-console/console-85fcbc755-6xmdc"
Aug 05 17:18:37.295263 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:18:37.295195    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/825969e8-cbe8-448d-a808-3153afda1962-oauth-serving-cert\") pod \"console-85fcbc755-6xmdc\" (UID: \"825969e8-cbe8-448d-a808-3153afda1962\") " pod="openshift-console/console-85fcbc755-6xmdc"
Aug 05 17:18:37.295343 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:18:37.295323    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/825969e8-cbe8-448d-a808-3153afda1962-console-oauth-config\") pod \"console-85fcbc755-6xmdc\" (UID: \"825969e8-cbe8-448d-a808-3153afda1962\") " pod="openshift-console/console-85fcbc755-6xmdc"
Aug 05 17:18:37.295400 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:18:37.295357    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zw9wj\" (UniqueName: \"kubernetes.io/projected/825969e8-cbe8-448d-a808-3153afda1962-kube-api-access-zw9wj\") pod \"console-85fcbc755-6xmdc\" (UID: \"825969e8-cbe8-448d-a808-3153afda1962\") " pod="openshift-console/console-85fcbc755-6xmdc"
Aug 05 17:18:37.295451 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:18:37.295407    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/825969e8-cbe8-448d-a808-3153afda1962-service-ca\") pod \"console-85fcbc755-6xmdc\" (UID: \"825969e8-cbe8-448d-a808-3153afda1962\") " pod="openshift-console/console-85fcbc755-6xmdc"
Aug 05 17:18:37.295506 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:18:37.295452    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/825969e8-cbe8-448d-a808-3153afda1962-console-config\") pod \"console-85fcbc755-6xmdc\" (UID: \"825969e8-cbe8-448d-a808-3153afda1962\") " pod="openshift-console/console-85fcbc755-6xmdc"
Aug 05 17:18:37.296134 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:18:37.296100    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/825969e8-cbe8-448d-a808-3153afda1962-oauth-serving-cert\") pod \"console-85fcbc755-6xmdc\" (UID: \"825969e8-cbe8-448d-a808-3153afda1962\") " pod="openshift-console/console-85fcbc755-6xmdc"
Aug 05 17:18:37.296134 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:18:37.296121    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/825969e8-cbe8-448d-a808-3153afda1962-service-ca\") pod \"console-85fcbc755-6xmdc\" (UID: \"825969e8-cbe8-448d-a808-3153afda1962\") " pod="openshift-console/console-85fcbc755-6xmdc"
Aug 05 17:18:37.296294 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:18:37.296142    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/825969e8-cbe8-448d-a808-3153afda1962-console-config\") pod \"console-85fcbc755-6xmdc\" (UID: \"825969e8-cbe8-448d-a808-3153afda1962\") " pod="openshift-console/console-85fcbc755-6xmdc"
Aug 05 17:18:37.296294 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:18:37.296210    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/825969e8-cbe8-448d-a808-3153afda1962-trusted-ca-bundle\") pod \"console-85fcbc755-6xmdc\" (UID: \"825969e8-cbe8-448d-a808-3153afda1962\") " pod="openshift-console/console-85fcbc755-6xmdc"
Aug 05 17:18:37.297711 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:18:37.297684    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/825969e8-cbe8-448d-a808-3153afda1962-console-serving-cert\") pod \"console-85fcbc755-6xmdc\" (UID: \"825969e8-cbe8-448d-a808-3153afda1962\") " pod="openshift-console/console-85fcbc755-6xmdc"
Aug 05 17:18:37.297711 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:18:37.297706    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/825969e8-cbe8-448d-a808-3153afda1962-console-oauth-config\") pod \"console-85fcbc755-6xmdc\" (UID: \"825969e8-cbe8-448d-a808-3153afda1962\") " pod="openshift-console/console-85fcbc755-6xmdc"
Aug 05 17:18:37.305385 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:18:37.305347    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zw9wj\" (UniqueName: \"kubernetes.io/projected/825969e8-cbe8-448d-a808-3153afda1962-kube-api-access-zw9wj\") pod \"console-85fcbc755-6xmdc\" (UID: \"825969e8-cbe8-448d-a808-3153afda1962\") " pod="openshift-console/console-85fcbc755-6xmdc"
Aug 05 17:18:37.441129 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:18:37.441047    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-85fcbc755-6xmdc"
Aug 05 17:18:37.580501 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:18:37.580477    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-85fcbc755-6xmdc"]
Aug 05 17:18:37.581539 ip-10-0-135-2 kubenswrapper[2571]: W0805 17:18:37.581516    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod825969e8_cbe8_448d_a808_3153afda1962.slice/crio-3ac7348efacd93499b697e463129e649462a5afddebf48b655b42d4e82d33159 WatchSource:0}: Error finding container 3ac7348efacd93499b697e463129e649462a5afddebf48b655b42d4e82d33159: Status 404 returned error can't find the container with id 3ac7348efacd93499b697e463129e649462a5afddebf48b655b42d4e82d33159
Aug 05 17:18:38.451743 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:18:38.451705    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-85fcbc755-6xmdc" event={"ID":"825969e8-cbe8-448d-a808-3153afda1962","Type":"ContainerStarted","Data":"5d49e33dd1a365ede3865e1973d67aa3ff4e9aab9f89cb4425e0bd4665e752db"}
Aug 05 17:18:38.452169 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:18:38.451780    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-85fcbc755-6xmdc" event={"ID":"825969e8-cbe8-448d-a808-3153afda1962","Type":"ContainerStarted","Data":"3ac7348efacd93499b697e463129e649462a5afddebf48b655b42d4e82d33159"}
Aug 05 17:18:38.476879 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:18:38.476835    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-85fcbc755-6xmdc" podStartSLOduration=1.476820681 podStartE2EDuration="1.476820681s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-05 17:18:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-05 17:18:38.475216491 +0000 UTC m=+345.842582835" watchObservedRunningTime="2026-08-05 17:18:38.476820681 +0000 UTC m=+345.844187025"
Aug 05 17:18:47.441976 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:18:47.441876    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-85fcbc755-6xmdc"
Aug 05 17:18:47.441976 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:18:47.441944    2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-85fcbc755-6xmdc"
Aug 05 17:18:47.446513 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:18:47.446492    2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-85fcbc755-6xmdc"
Aug 05 17:18:47.488363 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:18:47.488341    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-85fcbc755-6xmdc"
Aug 05 17:18:47.539423 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:18:47.539399    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-9685bddcf-bflgp"]
Aug 05 17:19:12.562950 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:19:12.562901    2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-9685bddcf-bflgp" podUID="13fe9531-744f-414a-84d9-c37c2f3c2109" containerName="console" containerID="cri-o://56b288078ff8c2ccf061850fbcb30beb0f482f1ada45fe7db074a06f13517055" gracePeriod=15
Aug 05 17:19:12.808389 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:19:12.808352    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-9685bddcf-bflgp"
Aug 05 17:19:12.850992 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:19:12.850930    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/13fe9531-744f-414a-84d9-c37c2f3c2109-console-serving-cert\") pod \"13fe9531-744f-414a-84d9-c37c2f3c2109\" (UID: \"13fe9531-744f-414a-84d9-c37c2f3c2109\") "
Aug 05 17:19:12.850992 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:19:12.850974    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/13fe9531-744f-414a-84d9-c37c2f3c2109-console-config\") pod \"13fe9531-744f-414a-84d9-c37c2f3c2109\" (UID: \"13fe9531-744f-414a-84d9-c37c2f3c2109\") "
Aug 05 17:19:12.850992 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:19:12.850990    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/13fe9531-744f-414a-84d9-c37c2f3c2109-console-oauth-config\") pod \"13fe9531-744f-414a-84d9-c37c2f3c2109\" (UID: \"13fe9531-744f-414a-84d9-c37c2f3c2109\") "
Aug 05 17:19:12.851229 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:19:12.851019    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-25l95\" (UniqueName: \"kubernetes.io/projected/13fe9531-744f-414a-84d9-c37c2f3c2109-kube-api-access-25l95\") pod \"13fe9531-744f-414a-84d9-c37c2f3c2109\" (UID: \"13fe9531-744f-414a-84d9-c37c2f3c2109\") "
Aug 05 17:19:12.851229 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:19:12.851043    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/13fe9531-744f-414a-84d9-c37c2f3c2109-trusted-ca-bundle\") pod \"13fe9531-744f-414a-84d9-c37c2f3c2109\" (UID: \"13fe9531-744f-414a-84d9-c37c2f3c2109\") "
Aug 05 17:19:12.851229 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:19:12.851114    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/13fe9531-744f-414a-84d9-c37c2f3c2109-service-ca\") pod \"13fe9531-744f-414a-84d9-c37c2f3c2109\" (UID: \"13fe9531-744f-414a-84d9-c37c2f3c2109\") "
Aug 05 17:19:12.851229 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:19:12.851137    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/13fe9531-744f-414a-84d9-c37c2f3c2109-oauth-serving-cert\") pod \"13fe9531-744f-414a-84d9-c37c2f3c2109\" (UID: \"13fe9531-744f-414a-84d9-c37c2f3c2109\") "
Aug 05 17:19:12.851444 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:19:12.851364    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/13fe9531-744f-414a-84d9-c37c2f3c2109-console-config" (OuterVolumeSpecName: "console-config") pod "13fe9531-744f-414a-84d9-c37c2f3c2109" (UID: "13fe9531-744f-414a-84d9-c37c2f3c2109"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 05 17:19:12.851617 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:19:12.851595    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/13fe9531-744f-414a-84d9-c37c2f3c2109-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "13fe9531-744f-414a-84d9-c37c2f3c2109" (UID: "13fe9531-744f-414a-84d9-c37c2f3c2109"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 05 17:19:12.851685 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:19:12.851660    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/13fe9531-744f-414a-84d9-c37c2f3c2109-service-ca" (OuterVolumeSpecName: "service-ca") pod "13fe9531-744f-414a-84d9-c37c2f3c2109" (UID: "13fe9531-744f-414a-84d9-c37c2f3c2109"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 05 17:19:12.851731 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:19:12.851683    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/13fe9531-744f-414a-84d9-c37c2f3c2109-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "13fe9531-744f-414a-84d9-c37c2f3c2109" (UID: "13fe9531-744f-414a-84d9-c37c2f3c2109"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 05 17:19:12.853205 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:19:12.853148    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/13fe9531-744f-414a-84d9-c37c2f3c2109-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "13fe9531-744f-414a-84d9-c37c2f3c2109" (UID: "13fe9531-744f-414a-84d9-c37c2f3c2109"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 05 17:19:12.853456 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:19:12.853429    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/13fe9531-744f-414a-84d9-c37c2f3c2109-kube-api-access-25l95" (OuterVolumeSpecName: "kube-api-access-25l95") pod "13fe9531-744f-414a-84d9-c37c2f3c2109" (UID: "13fe9531-744f-414a-84d9-c37c2f3c2109"). InnerVolumeSpecName "kube-api-access-25l95". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 05 17:19:12.853539 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:19:12.853520    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/13fe9531-744f-414a-84d9-c37c2f3c2109-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "13fe9531-744f-414a-84d9-c37c2f3c2109" (UID: "13fe9531-744f-414a-84d9-c37c2f3c2109"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 05 17:19:12.952166 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:19:12.952136    2571 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/13fe9531-744f-414a-84d9-c37c2f3c2109-console-config\") on node \"ip-10-0-135-2.ec2.internal\" DevicePath \"\""
Aug 05 17:19:12.952166 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:19:12.952160    2571 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/13fe9531-744f-414a-84d9-c37c2f3c2109-console-oauth-config\") on node \"ip-10-0-135-2.ec2.internal\" DevicePath \"\""
Aug 05 17:19:12.952330 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:19:12.952175    2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-25l95\" (UniqueName: \"kubernetes.io/projected/13fe9531-744f-414a-84d9-c37c2f3c2109-kube-api-access-25l95\") on node \"ip-10-0-135-2.ec2.internal\" DevicePath \"\""
Aug 05 17:19:12.952330 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:19:12.952188    2571 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/13fe9531-744f-414a-84d9-c37c2f3c2109-trusted-ca-bundle\") on node \"ip-10-0-135-2.ec2.internal\" DevicePath \"\""
Aug 05 17:19:12.952330 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:19:12.952199    2571 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/13fe9531-744f-414a-84d9-c37c2f3c2109-service-ca\") on node \"ip-10-0-135-2.ec2.internal\" DevicePath \"\""
Aug 05 17:19:12.952330 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:19:12.952209    2571 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/13fe9531-744f-414a-84d9-c37c2f3c2109-oauth-serving-cert\") on node \"ip-10-0-135-2.ec2.internal\" DevicePath \"\""
Aug 05 17:19:12.952330 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:19:12.952220    2571 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/13fe9531-744f-414a-84d9-c37c2f3c2109-console-serving-cert\") on node \"ip-10-0-135-2.ec2.internal\" DevicePath \"\""
Aug 05 17:19:13.592055 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:19:13.592018    2571 generic.go:356] "Generic (PLEG): container finished" podID="13fe9531-744f-414a-84d9-c37c2f3c2109" containerID="56b288078ff8c2ccf061850fbcb30beb0f482f1ada45fe7db074a06f13517055" exitCode=0
Aug 05 17:19:13.592437 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:19:13.592068    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-9685bddcf-bflgp"
Aug 05 17:19:13.592437 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:19:13.592065    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-9685bddcf-bflgp" event={"ID":"13fe9531-744f-414a-84d9-c37c2f3c2109","Type":"ContainerDied","Data":"56b288078ff8c2ccf061850fbcb30beb0f482f1ada45fe7db074a06f13517055"}
Aug 05 17:19:13.592437 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:19:13.592196    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-9685bddcf-bflgp" event={"ID":"13fe9531-744f-414a-84d9-c37c2f3c2109","Type":"ContainerDied","Data":"e6c6da8c5233016fbca65bedd8ad866695e99efc5283756d22ae77f699f81c3d"}
Aug 05 17:19:13.592437 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:19:13.592223    2571 scope.go:117] "RemoveContainer" containerID="56b288078ff8c2ccf061850fbcb30beb0f482f1ada45fe7db074a06f13517055"
Aug 05 17:19:13.602320 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:19:13.601039    2571 scope.go:117] "RemoveContainer" containerID="56b288078ff8c2ccf061850fbcb30beb0f482f1ada45fe7db074a06f13517055"
Aug 05 17:19:13.602709 ip-10-0-135-2 kubenswrapper[2571]: E0805 17:19:13.602689    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"56b288078ff8c2ccf061850fbcb30beb0f482f1ada45fe7db074a06f13517055\": container with ID starting with 56b288078ff8c2ccf061850fbcb30beb0f482f1ada45fe7db074a06f13517055 not found: ID does not exist" containerID="56b288078ff8c2ccf061850fbcb30beb0f482f1ada45fe7db074a06f13517055"
Aug 05 17:19:13.602769 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:19:13.602718    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"56b288078ff8c2ccf061850fbcb30beb0f482f1ada45fe7db074a06f13517055"} err="failed to get container status \"56b288078ff8c2ccf061850fbcb30beb0f482f1ada45fe7db074a06f13517055\": rpc error: code = NotFound desc = could not find container \"56b288078ff8c2ccf061850fbcb30beb0f482f1ada45fe7db074a06f13517055\": container with ID starting with 56b288078ff8c2ccf061850fbcb30beb0f482f1ada45fe7db074a06f13517055 not found: ID does not exist"
Aug 05 17:19:13.612838 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:19:13.612817    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-9685bddcf-bflgp"]
Aug 05 17:19:13.616769 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:19:13.616751    2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-9685bddcf-bflgp"]
Aug 05 17:19:15.141046 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:19:15.141003    2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="13fe9531-744f-414a-84d9-c37c2f3c2109" path="/var/lib/kubelet/pods/13fe9531-744f-414a-84d9-c37c2f3c2109/volumes"
Aug 05 17:20:10.681926 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:20:10.681894    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-keda/keda-operator-77f648d694-g787c"]
Aug 05 17:20:10.682343 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:20:10.682187    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="13fe9531-744f-414a-84d9-c37c2f3c2109" containerName="console"
Aug 05 17:20:10.682343 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:20:10.682199    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="13fe9531-744f-414a-84d9-c37c2f3c2109" containerName="console"
Aug 05 17:20:10.682343 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:20:10.682246    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="13fe9531-744f-414a-84d9-c37c2f3c2109" containerName="console"
Aug 05 17:20:10.685415 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:20:10.685397    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-operator-77f648d694-g787c"
Aug 05 17:20:10.688181 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:20:10.688154    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"kube-root-ca.crt\""
Aug 05 17:20:10.688332 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:20:10.688279    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"kedaorg-certs\""
Aug 05 17:20:10.689099 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:20:10.689080    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-operator-certs\""
Aug 05 17:20:10.689203 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:20:10.689089    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-operator-dockercfg-57lnq\""
Aug 05 17:20:10.689203 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:20:10.689106    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"keda-ocp-cabundle\""
Aug 05 17:20:10.689203 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:20:10.689125    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"openshift-service-ca.crt\""
Aug 05 17:20:10.695144 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:20:10.695123    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-operator-77f648d694-g787c"]
Aug 05 17:20:10.770799 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:20:10.770759    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle0\" (UniqueName: \"kubernetes.io/configmap/ca0dda53-4d5f-46e8-90d1-3ccae9e02661-cabundle0\") pod \"keda-operator-77f648d694-g787c\" (UID: \"ca0dda53-4d5f-46e8-90d1-3ccae9e02661\") " pod="openshift-keda/keda-operator-77f648d694-g787c"
Aug 05 17:20:10.770799 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:20:10.770800    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/ca0dda53-4d5f-46e8-90d1-3ccae9e02661-certificates\") pod \"keda-operator-77f648d694-g787c\" (UID: \"ca0dda53-4d5f-46e8-90d1-3ccae9e02661\") " pod="openshift-keda/keda-operator-77f648d694-g787c"
Aug 05 17:20:10.771012 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:20:10.770831    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bhthr\" (UniqueName: \"kubernetes.io/projected/ca0dda53-4d5f-46e8-90d1-3ccae9e02661-kube-api-access-bhthr\") pod \"keda-operator-77f648d694-g787c\" (UID: \"ca0dda53-4d5f-46e8-90d1-3ccae9e02661\") " pod="openshift-keda/keda-operator-77f648d694-g787c"
Aug 05 17:20:10.872119 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:20:10.872087    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle0\" (UniqueName: \"kubernetes.io/configmap/ca0dda53-4d5f-46e8-90d1-3ccae9e02661-cabundle0\") pod \"keda-operator-77f648d694-g787c\" (UID: \"ca0dda53-4d5f-46e8-90d1-3ccae9e02661\") " pod="openshift-keda/keda-operator-77f648d694-g787c"
Aug 05 17:20:10.872119 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:20:10.872124    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/ca0dda53-4d5f-46e8-90d1-3ccae9e02661-certificates\") pod \"keda-operator-77f648d694-g787c\" (UID: \"ca0dda53-4d5f-46e8-90d1-3ccae9e02661\") " pod="openshift-keda/keda-operator-77f648d694-g787c"
Aug 05 17:20:10.872352 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:20:10.872157    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bhthr\" (UniqueName: \"kubernetes.io/projected/ca0dda53-4d5f-46e8-90d1-3ccae9e02661-kube-api-access-bhthr\") pod \"keda-operator-77f648d694-g787c\" (UID: \"ca0dda53-4d5f-46e8-90d1-3ccae9e02661\") " pod="openshift-keda/keda-operator-77f648d694-g787c"
Aug 05 17:20:10.872352 ip-10-0-135-2 kubenswrapper[2571]: E0805 17:20:10.872267    2571 secret.go:281] references non-existent secret key: ca.crt
Aug 05 17:20:10.872352 ip-10-0-135-2 kubenswrapper[2571]: E0805 17:20:10.872290    2571 projected.go:279] Couldn't get secret payload openshift-keda/kedaorg-certs: references non-existent secret key: ca.crt
Aug 05 17:20:10.872352 ip-10-0-135-2 kubenswrapper[2571]: E0805 17:20:10.872301    2571 projected.go:196] Error preparing data for projected volume certificates for pod openshift-keda/keda-operator-77f648d694-g787c: references non-existent secret key: ca.crt
Aug 05 17:20:10.872566 ip-10-0-135-2 kubenswrapper[2571]: E0805 17:20:10.872411    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ca0dda53-4d5f-46e8-90d1-3ccae9e02661-certificates podName:ca0dda53-4d5f-46e8-90d1-3ccae9e02661 nodeName:}" failed. No retries permitted until 2026-08-05 17:20:11.372352249 +0000 UTC m=+438.739718576 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "certificates" (UniqueName: "kubernetes.io/projected/ca0dda53-4d5f-46e8-90d1-3ccae9e02661-certificates") pod "keda-operator-77f648d694-g787c" (UID: "ca0dda53-4d5f-46e8-90d1-3ccae9e02661") : references non-existent secret key: ca.crt
Aug 05 17:20:10.872810 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:20:10.872792    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle0\" (UniqueName: \"kubernetes.io/configmap/ca0dda53-4d5f-46e8-90d1-3ccae9e02661-cabundle0\") pod \"keda-operator-77f648d694-g787c\" (UID: \"ca0dda53-4d5f-46e8-90d1-3ccae9e02661\") " pod="openshift-keda/keda-operator-77f648d694-g787c"
Aug 05 17:20:10.881474 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:20:10.881458    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bhthr\" (UniqueName: \"kubernetes.io/projected/ca0dda53-4d5f-46e8-90d1-3ccae9e02661-kube-api-access-bhthr\") pod \"keda-operator-77f648d694-g787c\" (UID: \"ca0dda53-4d5f-46e8-90d1-3ccae9e02661\") " pod="openshift-keda/keda-operator-77f648d694-g787c"
Aug 05 17:20:11.376075 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:20:11.376041    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/ca0dda53-4d5f-46e8-90d1-3ccae9e02661-certificates\") pod \"keda-operator-77f648d694-g787c\" (UID: \"ca0dda53-4d5f-46e8-90d1-3ccae9e02661\") " pod="openshift-keda/keda-operator-77f648d694-g787c"
Aug 05 17:20:11.376251 ip-10-0-135-2 kubenswrapper[2571]: E0805 17:20:11.376154    2571 secret.go:281] references non-existent secret key: ca.crt
Aug 05 17:20:11.376251 ip-10-0-135-2 kubenswrapper[2571]: E0805 17:20:11.376166    2571 projected.go:279] Couldn't get secret payload openshift-keda/kedaorg-certs: references non-existent secret key: ca.crt
Aug 05 17:20:11.376251 ip-10-0-135-2 kubenswrapper[2571]: E0805 17:20:11.376174    2571 projected.go:196] Error preparing data for projected volume certificates for pod openshift-keda/keda-operator-77f648d694-g787c: references non-existent secret key: ca.crt
Aug 05 17:20:11.376251 ip-10-0-135-2 kubenswrapper[2571]: E0805 17:20:11.376218    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ca0dda53-4d5f-46e8-90d1-3ccae9e02661-certificates podName:ca0dda53-4d5f-46e8-90d1-3ccae9e02661 nodeName:}" failed. No retries permitted until 2026-08-05 17:20:12.376206597 +0000 UTC m=+439.743572919 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "certificates" (UniqueName: "kubernetes.io/projected/ca0dda53-4d5f-46e8-90d1-3ccae9e02661-certificates") pod "keda-operator-77f648d694-g787c" (UID: "ca0dda53-4d5f-46e8-90d1-3ccae9e02661") : references non-existent secret key: ca.crt
Aug 05 17:20:11.444655 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:20:11.444621    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-keda/keda-metrics-apiserver-55557df8c-lvglf"]
Aug 05 17:20:11.447764 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:20:11.447747    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-metrics-apiserver-55557df8c-lvglf"
Aug 05 17:20:11.450551 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:20:11.450531    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-metrics-apiserver-certs\""
Aug 05 17:20:11.461230 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:20:11.461210    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-metrics-apiserver-55557df8c-lvglf"]
Aug 05 17:20:11.577893 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:20:11.577854    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/9949e492-1770-4049-9879-5996cf2a97d4-certificates\") pod \"keda-metrics-apiserver-55557df8c-lvglf\" (UID: \"9949e492-1770-4049-9879-5996cf2a97d4\") " pod="openshift-keda/keda-metrics-apiserver-55557df8c-lvglf"
Aug 05 17:20:11.577893 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:20:11.577889    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jfdvl\" (UniqueName: \"kubernetes.io/projected/9949e492-1770-4049-9879-5996cf2a97d4-kube-api-access-jfdvl\") pod \"keda-metrics-apiserver-55557df8c-lvglf\" (UID: \"9949e492-1770-4049-9879-5996cf2a97d4\") " pod="openshift-keda/keda-metrics-apiserver-55557df8c-lvglf"
Aug 05 17:20:11.578105 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:20:11.577919    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"temp-vol\" (UniqueName: \"kubernetes.io/empty-dir/9949e492-1770-4049-9879-5996cf2a97d4-temp-vol\") pod \"keda-metrics-apiserver-55557df8c-lvglf\" (UID: \"9949e492-1770-4049-9879-5996cf2a97d4\") " pod="openshift-keda/keda-metrics-apiserver-55557df8c-lvglf"
Aug 05 17:20:11.679105 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:20:11.679033    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/9949e492-1770-4049-9879-5996cf2a97d4-certificates\") pod \"keda-metrics-apiserver-55557df8c-lvglf\" (UID: \"9949e492-1770-4049-9879-5996cf2a97d4\") " pod="openshift-keda/keda-metrics-apiserver-55557df8c-lvglf"
Aug 05 17:20:11.679105 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:20:11.679070    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jfdvl\" (UniqueName: \"kubernetes.io/projected/9949e492-1770-4049-9879-5996cf2a97d4-kube-api-access-jfdvl\") pod \"keda-metrics-apiserver-55557df8c-lvglf\" (UID: \"9949e492-1770-4049-9879-5996cf2a97d4\") " pod="openshift-keda/keda-metrics-apiserver-55557df8c-lvglf"
Aug 05 17:20:11.679295 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:20:11.679189    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"temp-vol\" (UniqueName: \"kubernetes.io/empty-dir/9949e492-1770-4049-9879-5996cf2a97d4-temp-vol\") pod \"keda-metrics-apiserver-55557df8c-lvglf\" (UID: \"9949e492-1770-4049-9879-5996cf2a97d4\") " pod="openshift-keda/keda-metrics-apiserver-55557df8c-lvglf"
Aug 05 17:20:11.679592 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:20:11.679576    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"temp-vol\" (UniqueName: \"kubernetes.io/empty-dir/9949e492-1770-4049-9879-5996cf2a97d4-temp-vol\") pod \"keda-metrics-apiserver-55557df8c-lvglf\" (UID: \"9949e492-1770-4049-9879-5996cf2a97d4\") " pod="openshift-keda/keda-metrics-apiserver-55557df8c-lvglf"
Aug 05 17:20:11.681628 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:20:11.681611    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/9949e492-1770-4049-9879-5996cf2a97d4-certificates\") pod \"keda-metrics-apiserver-55557df8c-lvglf\" (UID: \"9949e492-1770-4049-9879-5996cf2a97d4\") " pod="openshift-keda/keda-metrics-apiserver-55557df8c-lvglf"
Aug 05 17:20:11.691797 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:20:11.691774    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jfdvl\" (UniqueName: \"kubernetes.io/projected/9949e492-1770-4049-9879-5996cf2a97d4-kube-api-access-jfdvl\") pod \"keda-metrics-apiserver-55557df8c-lvglf\" (UID: \"9949e492-1770-4049-9879-5996cf2a97d4\") " pod="openshift-keda/keda-metrics-apiserver-55557df8c-lvglf"
Aug 05 17:20:11.745846 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:20:11.745811    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-keda/keda-admission-56f9798bb4-2gg7t"]
Aug 05 17:20:11.749040 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:20:11.749025    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-admission-56f9798bb4-2gg7t"
Aug 05 17:20:11.752609 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:20:11.752590    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-admission-webhooks-certs\""
Aug 05 17:20:11.757386 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:20:11.757342    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-metrics-apiserver-55557df8c-lvglf"
Aug 05 17:20:11.762691 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:20:11.762673    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-admission-56f9798bb4-2gg7t"]
Aug 05 17:20:11.880929 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:20:11.880896    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7zzrd\" (UniqueName: \"kubernetes.io/projected/48275ce8-9299-4585-acb3-3df000e66094-kube-api-access-7zzrd\") pod \"keda-admission-56f9798bb4-2gg7t\" (UID: \"48275ce8-9299-4585-acb3-3df000e66094\") " pod="openshift-keda/keda-admission-56f9798bb4-2gg7t"
Aug 05 17:20:11.881080 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:20:11.880942    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/48275ce8-9299-4585-acb3-3df000e66094-certificates\") pod \"keda-admission-56f9798bb4-2gg7t\" (UID: \"48275ce8-9299-4585-acb3-3df000e66094\") " pod="openshift-keda/keda-admission-56f9798bb4-2gg7t"
Aug 05 17:20:11.898643 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:20:11.898565    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-metrics-apiserver-55557df8c-lvglf"]
Aug 05 17:20:11.901509 ip-10-0-135-2 kubenswrapper[2571]: W0805 17:20:11.901482    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9949e492_1770_4049_9879_5996cf2a97d4.slice/crio-07dd5ee1b7dce073ae74fb601f7945d109d07f828200bb192475621b2c43fe37 WatchSource:0}: Error finding container 07dd5ee1b7dce073ae74fb601f7945d109d07f828200bb192475621b2c43fe37: Status 404 returned error can't find the container with id 07dd5ee1b7dce073ae74fb601f7945d109d07f828200bb192475621b2c43fe37
Aug 05 17:20:11.903165 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:20:11.903144    2571 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Aug 05 17:20:11.981840 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:20:11.981811    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/48275ce8-9299-4585-acb3-3df000e66094-certificates\") pod \"keda-admission-56f9798bb4-2gg7t\" (UID: \"48275ce8-9299-4585-acb3-3df000e66094\") " pod="openshift-keda/keda-admission-56f9798bb4-2gg7t"
Aug 05 17:20:11.981993 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:20:11.981900    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7zzrd\" (UniqueName: \"kubernetes.io/projected/48275ce8-9299-4585-acb3-3df000e66094-kube-api-access-7zzrd\") pod \"keda-admission-56f9798bb4-2gg7t\" (UID: \"48275ce8-9299-4585-acb3-3df000e66094\") " pod="openshift-keda/keda-admission-56f9798bb4-2gg7t"
Aug 05 17:20:11.984249 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:20:11.984226    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/48275ce8-9299-4585-acb3-3df000e66094-certificates\") pod \"keda-admission-56f9798bb4-2gg7t\" (UID: \"48275ce8-9299-4585-acb3-3df000e66094\") " pod="openshift-keda/keda-admission-56f9798bb4-2gg7t"
Aug 05 17:20:11.991366 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:20:11.991339    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7zzrd\" (UniqueName: \"kubernetes.io/projected/48275ce8-9299-4585-acb3-3df000e66094-kube-api-access-7zzrd\") pod \"keda-admission-56f9798bb4-2gg7t\" (UID: \"48275ce8-9299-4585-acb3-3df000e66094\") " pod="openshift-keda/keda-admission-56f9798bb4-2gg7t"
Aug 05 17:20:12.058974 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:20:12.058947    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-admission-56f9798bb4-2gg7t"
Aug 05 17:20:12.185275 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:20:12.185246    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-admission-56f9798bb4-2gg7t"]
Aug 05 17:20:12.188324 ip-10-0-135-2 kubenswrapper[2571]: W0805 17:20:12.188298    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod48275ce8_9299_4585_acb3_3df000e66094.slice/crio-1f8fb65945f577afd2a8ce62ae1559c37ce4f537fc42f38f548462151c1619de WatchSource:0}: Error finding container 1f8fb65945f577afd2a8ce62ae1559c37ce4f537fc42f38f548462151c1619de: Status 404 returned error can't find the container with id 1f8fb65945f577afd2a8ce62ae1559c37ce4f537fc42f38f548462151c1619de
Aug 05 17:20:12.385356 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:20:12.385262    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/ca0dda53-4d5f-46e8-90d1-3ccae9e02661-certificates\") pod \"keda-operator-77f648d694-g787c\" (UID: \"ca0dda53-4d5f-46e8-90d1-3ccae9e02661\") " pod="openshift-keda/keda-operator-77f648d694-g787c"
Aug 05 17:20:12.387627 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:20:12.387600    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/ca0dda53-4d5f-46e8-90d1-3ccae9e02661-certificates\") pod \"keda-operator-77f648d694-g787c\" (UID: \"ca0dda53-4d5f-46e8-90d1-3ccae9e02661\") " pod="openshift-keda/keda-operator-77f648d694-g787c"
Aug 05 17:20:12.495943 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:20:12.495904    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-operator-77f648d694-g787c"
Aug 05 17:20:12.648847 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:20:12.648719    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-operator-77f648d694-g787c"]
Aug 05 17:20:12.652059 ip-10-0-135-2 kubenswrapper[2571]: W0805 17:20:12.652028    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podca0dda53_4d5f_46e8_90d1_3ccae9e02661.slice/crio-bf02a3f03da6b041a01a4f7c4969e035cfd2cc03a04dcd87c39649e46ddc3dcb WatchSource:0}: Error finding container bf02a3f03da6b041a01a4f7c4969e035cfd2cc03a04dcd87c39649e46ddc3dcb: Status 404 returned error can't find the container with id bf02a3f03da6b041a01a4f7c4969e035cfd2cc03a04dcd87c39649e46ddc3dcb
Aug 05 17:20:12.813462 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:20:12.813421    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-admission-56f9798bb4-2gg7t" event={"ID":"48275ce8-9299-4585-acb3-3df000e66094","Type":"ContainerStarted","Data":"1f8fb65945f577afd2a8ce62ae1559c37ce4f537fc42f38f548462151c1619de"}
Aug 05 17:20:12.815070 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:20:12.815042    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-operator-77f648d694-g787c" event={"ID":"ca0dda53-4d5f-46e8-90d1-3ccae9e02661","Type":"ContainerStarted","Data":"bf02a3f03da6b041a01a4f7c4969e035cfd2cc03a04dcd87c39649e46ddc3dcb"}
Aug 05 17:20:12.817088 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:20:12.817031    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-metrics-apiserver-55557df8c-lvglf" event={"ID":"9949e492-1770-4049-9879-5996cf2a97d4","Type":"ContainerStarted","Data":"07dd5ee1b7dce073ae74fb601f7945d109d07f828200bb192475621b2c43fe37"}
Aug 05 17:20:17.845524 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:20:17.845485    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-operator-77f648d694-g787c" event={"ID":"ca0dda53-4d5f-46e8-90d1-3ccae9e02661","Type":"ContainerStarted","Data":"c737e225c4245fd826c25d8f76dadb2862b7937c7f9a4b4a58aa838957d48596"}
Aug 05 17:20:17.845985 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:20:17.845594    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-keda/keda-operator-77f648d694-g787c"
Aug 05 17:20:17.847076 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:20:17.847048    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-metrics-apiserver-55557df8c-lvglf" event={"ID":"9949e492-1770-4049-9879-5996cf2a97d4","Type":"ContainerStarted","Data":"c84b5831df768ecc96b99dd493fdb3845790c15a773165e377a0da6452c34ad7"}
Aug 05 17:20:17.847217 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:20:17.847101    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-keda/keda-metrics-apiserver-55557df8c-lvglf"
Aug 05 17:20:17.848304 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:20:17.848279    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-admission-56f9798bb4-2gg7t" event={"ID":"48275ce8-9299-4585-acb3-3df000e66094","Type":"ContainerStarted","Data":"360045e1ae97a27386f0498513ec08ff210244603e48a114a32f2041faa67f18"}
Aug 05 17:20:17.848442 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:20:17.848424    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-keda/keda-admission-56f9798bb4-2gg7t"
Aug 05 17:20:17.873034 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:20:17.872996    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-keda/keda-operator-77f648d694-g787c" podStartSLOduration=3.712710949 podStartE2EDuration="7.872983812s" totalImagesPullingTime="4.160272863s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-05 17:20:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-05 17:20:17.869065744 +0000 UTC m=+445.236432088" watchObservedRunningTime="2026-08-05 17:20:17.872983812 +0000 UTC m=+445.240350134"
Aug 05 17:20:17.907873 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:20:17.907820    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-keda/keda-metrics-apiserver-55557df8c-lvglf" podStartSLOduration=2.011721289 podStartE2EDuration="6.907800384s" totalImagesPullingTime="4.896079095s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-05 17:20:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-05 17:20:17.907681997 +0000 UTC m=+445.275048341" watchObservedRunningTime="2026-08-05 17:20:17.907800384 +0000 UTC m=+445.275166728"
Aug 05 17:20:17.951000 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:20:17.950952    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-keda/keda-admission-56f9798bb4-2gg7t" podStartSLOduration=2.335323316 podStartE2EDuration="6.950937792s" totalImagesPullingTime="4.615614476s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-05 17:20:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-05 17:20:17.949634878 +0000 UTC m=+445.317001220" watchObservedRunningTime="2026-08-05 17:20:17.950937792 +0000 UTC m=+445.318304134"
Aug 05 17:20:28.857175 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:20:28.857144    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-keda/keda-metrics-apiserver-55557df8c-lvglf"
Aug 05 17:20:38.854902 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:20:38.854868    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-keda/keda-operator-77f648d694-g787c"
Aug 05 17:20:38.855332 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:20:38.854944    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-keda/keda-admission-56f9798bb4-2gg7t"
Aug 05 17:21:24.345597 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:21:24.345563    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-ffb9f44c4-xvvq9"]
Aug 05 17:21:24.348766 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:21:24.348747    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-ffb9f44c4-xvvq9"
Aug 05 17:21:24.367622 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:21:24.367590    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-ffb9f44c4-xvvq9"]
Aug 05 17:21:24.431467 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:21:24.431417    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/90eef495-5e96-4ffa-ae69-d1a0a4ae6da6-service-ca\") pod \"console-ffb9f44c4-xvvq9\" (UID: \"90eef495-5e96-4ffa-ae69-d1a0a4ae6da6\") " pod="openshift-console/console-ffb9f44c4-xvvq9"
Aug 05 17:21:24.431467 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:21:24.431480    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/90eef495-5e96-4ffa-ae69-d1a0a4ae6da6-console-config\") pod \"console-ffb9f44c4-xvvq9\" (UID: \"90eef495-5e96-4ffa-ae69-d1a0a4ae6da6\") " pod="openshift-console/console-ffb9f44c4-xvvq9"
Aug 05 17:21:24.431694 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:21:24.431545    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/90eef495-5e96-4ffa-ae69-d1a0a4ae6da6-trusted-ca-bundle\") pod \"console-ffb9f44c4-xvvq9\" (UID: \"90eef495-5e96-4ffa-ae69-d1a0a4ae6da6\") " pod="openshift-console/console-ffb9f44c4-xvvq9"
Aug 05 17:21:24.431694 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:21:24.431590    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/90eef495-5e96-4ffa-ae69-d1a0a4ae6da6-console-serving-cert\") pod \"console-ffb9f44c4-xvvq9\" (UID: \"90eef495-5e96-4ffa-ae69-d1a0a4ae6da6\") " pod="openshift-console/console-ffb9f44c4-xvvq9"
Aug 05 17:21:24.431694 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:21:24.431608    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/90eef495-5e96-4ffa-ae69-d1a0a4ae6da6-console-oauth-config\") pod \"console-ffb9f44c4-xvvq9\" (UID: \"90eef495-5e96-4ffa-ae69-d1a0a4ae6da6\") " pod="openshift-console/console-ffb9f44c4-xvvq9"
Aug 05 17:21:24.431694 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:21:24.431631    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j2x7p\" (UniqueName: \"kubernetes.io/projected/90eef495-5e96-4ffa-ae69-d1a0a4ae6da6-kube-api-access-j2x7p\") pod \"console-ffb9f44c4-xvvq9\" (UID: \"90eef495-5e96-4ffa-ae69-d1a0a4ae6da6\") " pod="openshift-console/console-ffb9f44c4-xvvq9"
Aug 05 17:21:24.431694 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:21:24.431651    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/90eef495-5e96-4ffa-ae69-d1a0a4ae6da6-oauth-serving-cert\") pod \"console-ffb9f44c4-xvvq9\" (UID: \"90eef495-5e96-4ffa-ae69-d1a0a4ae6da6\") " pod="openshift-console/console-ffb9f44c4-xvvq9"
Aug 05 17:21:24.532789 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:21:24.532737    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/90eef495-5e96-4ffa-ae69-d1a0a4ae6da6-oauth-serving-cert\") pod \"console-ffb9f44c4-xvvq9\" (UID: \"90eef495-5e96-4ffa-ae69-d1a0a4ae6da6\") " pod="openshift-console/console-ffb9f44c4-xvvq9"
Aug 05 17:21:24.532991 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:21:24.532809    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/90eef495-5e96-4ffa-ae69-d1a0a4ae6da6-service-ca\") pod \"console-ffb9f44c4-xvvq9\" (UID: \"90eef495-5e96-4ffa-ae69-d1a0a4ae6da6\") " pod="openshift-console/console-ffb9f44c4-xvvq9"
Aug 05 17:21:24.532991 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:21:24.532868    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/90eef495-5e96-4ffa-ae69-d1a0a4ae6da6-console-config\") pod \"console-ffb9f44c4-xvvq9\" (UID: \"90eef495-5e96-4ffa-ae69-d1a0a4ae6da6\") " pod="openshift-console/console-ffb9f44c4-xvvq9"
Aug 05 17:21:24.532991 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:21:24.532903    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/90eef495-5e96-4ffa-ae69-d1a0a4ae6da6-trusted-ca-bundle\") pod \"console-ffb9f44c4-xvvq9\" (UID: \"90eef495-5e96-4ffa-ae69-d1a0a4ae6da6\") " pod="openshift-console/console-ffb9f44c4-xvvq9"
Aug 05 17:21:24.533183 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:21:24.533057    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/90eef495-5e96-4ffa-ae69-d1a0a4ae6da6-console-serving-cert\") pod \"console-ffb9f44c4-xvvq9\" (UID: \"90eef495-5e96-4ffa-ae69-d1a0a4ae6da6\") " pod="openshift-console/console-ffb9f44c4-xvvq9"
Aug 05 17:21:24.533183 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:21:24.533104    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/90eef495-5e96-4ffa-ae69-d1a0a4ae6da6-console-oauth-config\") pod \"console-ffb9f44c4-xvvq9\" (UID: \"90eef495-5e96-4ffa-ae69-d1a0a4ae6da6\") " pod="openshift-console/console-ffb9f44c4-xvvq9"
Aug 05 17:21:24.533183 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:21:24.533140    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j2x7p\" (UniqueName: \"kubernetes.io/projected/90eef495-5e96-4ffa-ae69-d1a0a4ae6da6-kube-api-access-j2x7p\") pod \"console-ffb9f44c4-xvvq9\" (UID: \"90eef495-5e96-4ffa-ae69-d1a0a4ae6da6\") " pod="openshift-console/console-ffb9f44c4-xvvq9"
Aug 05 17:21:24.533612 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:21:24.533580    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/90eef495-5e96-4ffa-ae69-d1a0a4ae6da6-service-ca\") pod \"console-ffb9f44c4-xvvq9\" (UID: \"90eef495-5e96-4ffa-ae69-d1a0a4ae6da6\") " pod="openshift-console/console-ffb9f44c4-xvvq9"
Aug 05 17:21:24.533718 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:21:24.533619    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/90eef495-5e96-4ffa-ae69-d1a0a4ae6da6-console-config\") pod \"console-ffb9f44c4-xvvq9\" (UID: \"90eef495-5e96-4ffa-ae69-d1a0a4ae6da6\") " pod="openshift-console/console-ffb9f44c4-xvvq9"
Aug 05 17:21:24.533718 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:21:24.533619    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/90eef495-5e96-4ffa-ae69-d1a0a4ae6da6-oauth-serving-cert\") pod \"console-ffb9f44c4-xvvq9\" (UID: \"90eef495-5e96-4ffa-ae69-d1a0a4ae6da6\") " pod="openshift-console/console-ffb9f44c4-xvvq9"
Aug 05 17:21:24.533790 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:21:24.533765    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/90eef495-5e96-4ffa-ae69-d1a0a4ae6da6-trusted-ca-bundle\") pod \"console-ffb9f44c4-xvvq9\" (UID: \"90eef495-5e96-4ffa-ae69-d1a0a4ae6da6\") " pod="openshift-console/console-ffb9f44c4-xvvq9"
Aug 05 17:21:24.535631 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:21:24.535608    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/90eef495-5e96-4ffa-ae69-d1a0a4ae6da6-console-serving-cert\") pod \"console-ffb9f44c4-xvvq9\" (UID: \"90eef495-5e96-4ffa-ae69-d1a0a4ae6da6\") " pod="openshift-console/console-ffb9f44c4-xvvq9"
Aug 05 17:21:24.535631 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:21:24.535623    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/90eef495-5e96-4ffa-ae69-d1a0a4ae6da6-console-oauth-config\") pod \"console-ffb9f44c4-xvvq9\" (UID: \"90eef495-5e96-4ffa-ae69-d1a0a4ae6da6\") " pod="openshift-console/console-ffb9f44c4-xvvq9"
Aug 05 17:21:24.542134 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:21:24.542113    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j2x7p\" (UniqueName: \"kubernetes.io/projected/90eef495-5e96-4ffa-ae69-d1a0a4ae6da6-kube-api-access-j2x7p\") pod \"console-ffb9f44c4-xvvq9\" (UID: \"90eef495-5e96-4ffa-ae69-d1a0a4ae6da6\") " pod="openshift-console/console-ffb9f44c4-xvvq9"
Aug 05 17:21:24.658318 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:21:24.658222    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-ffb9f44c4-xvvq9"
Aug 05 17:21:24.801042 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:21:24.801010    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-ffb9f44c4-xvvq9"]
Aug 05 17:21:24.804518 ip-10-0-135-2 kubenswrapper[2571]: W0805 17:21:24.804489    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod90eef495_5e96_4ffa_ae69_d1a0a4ae6da6.slice/crio-69c149d6f12ab19532e00ae613d843238431938a6ed282f56985d5b1c1de9d39 WatchSource:0}: Error finding container 69c149d6f12ab19532e00ae613d843238431938a6ed282f56985d5b1c1de9d39: Status 404 returned error can't find the container with id 69c149d6f12ab19532e00ae613d843238431938a6ed282f56985d5b1c1de9d39
Aug 05 17:21:25.113069 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:21:25.113033    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-ffb9f44c4-xvvq9" event={"ID":"90eef495-5e96-4ffa-ae69-d1a0a4ae6da6","Type":"ContainerStarted","Data":"556a42a139b5404ace65480a6fe299168325a527f1dfe313489fab05ad9462d3"}
Aug 05 17:21:25.113261 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:21:25.113112    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-ffb9f44c4-xvvq9" event={"ID":"90eef495-5e96-4ffa-ae69-d1a0a4ae6da6","Type":"ContainerStarted","Data":"69c149d6f12ab19532e00ae613d843238431938a6ed282f56985d5b1c1de9d39"}
Aug 05 17:21:25.145595 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:21:25.145551    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-ffb9f44c4-xvvq9" podStartSLOduration=1.1455364000000001 podStartE2EDuration="1.1455364s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-05 17:21:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-05 17:21:25.143907976 +0000 UTC m=+512.511274319" watchObservedRunningTime="2026-08-05 17:21:25.1455364 +0000 UTC m=+512.512902742"
Aug 05 17:21:34.635665 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:21:34.635631    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/seaweedfs-64568bcd49-7qwp7"]
Aug 05 17:21:34.638901 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:21:34.638881    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-64568bcd49-7qwp7"
Aug 05 17:21:34.641420 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:21:34.641388    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"mlpipeline-s3-artifact\""
Aug 05 17:21:34.641420 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:21:34.641395    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"openshift-service-ca.crt\""
Aug 05 17:21:34.641696 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:21:34.641681    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"kube-root-ca.crt\""
Aug 05 17:21:34.642324 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:21:34.642308    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"default-dockercfg-sxhkx\""
Aug 05 17:21:34.651687 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:21:34.651669    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-64568bcd49-7qwp7"]
Aug 05 17:21:34.658309 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:21:34.658289    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-ffb9f44c4-xvvq9"
Aug 05 17:21:34.658416 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:21:34.658330    2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-ffb9f44c4-xvvq9"
Aug 05 17:21:34.663205 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:21:34.663183    2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-ffb9f44c4-xvvq9"
Aug 05 17:21:34.717516 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:21:34.717487    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-npl2r\" (UniqueName: \"kubernetes.io/projected/851488c2-41ee-4d47-95da-5710eca8d9e7-kube-api-access-npl2r\") pod \"seaweedfs-64568bcd49-7qwp7\" (UID: \"851488c2-41ee-4d47-95da-5710eca8d9e7\") " pod="kserve/seaweedfs-64568bcd49-7qwp7"
Aug 05 17:21:34.717622 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:21:34.717569    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/851488c2-41ee-4d47-95da-5710eca8d9e7-data\") pod \"seaweedfs-64568bcd49-7qwp7\" (UID: \"851488c2-41ee-4d47-95da-5710eca8d9e7\") " pod="kserve/seaweedfs-64568bcd49-7qwp7"
Aug 05 17:21:34.818536 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:21:34.818507    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-npl2r\" (UniqueName: \"kubernetes.io/projected/851488c2-41ee-4d47-95da-5710eca8d9e7-kube-api-access-npl2r\") pod \"seaweedfs-64568bcd49-7qwp7\" (UID: \"851488c2-41ee-4d47-95da-5710eca8d9e7\") " pod="kserve/seaweedfs-64568bcd49-7qwp7"
Aug 05 17:21:34.818646 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:21:34.818561    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/851488c2-41ee-4d47-95da-5710eca8d9e7-data\") pod \"seaweedfs-64568bcd49-7qwp7\" (UID: \"851488c2-41ee-4d47-95da-5710eca8d9e7\") " pod="kserve/seaweedfs-64568bcd49-7qwp7"
Aug 05 17:21:34.818882 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:21:34.818866    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/851488c2-41ee-4d47-95da-5710eca8d9e7-data\") pod \"seaweedfs-64568bcd49-7qwp7\" (UID: \"851488c2-41ee-4d47-95da-5710eca8d9e7\") " pod="kserve/seaweedfs-64568bcd49-7qwp7"
Aug 05 17:21:34.828537 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:21:34.828517    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-npl2r\" (UniqueName: \"kubernetes.io/projected/851488c2-41ee-4d47-95da-5710eca8d9e7-kube-api-access-npl2r\") pod \"seaweedfs-64568bcd49-7qwp7\" (UID: \"851488c2-41ee-4d47-95da-5710eca8d9e7\") " pod="kserve/seaweedfs-64568bcd49-7qwp7"
Aug 05 17:21:34.948318 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:21:34.948268    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-64568bcd49-7qwp7"
Aug 05 17:21:35.091319 ip-10-0-135-2 kubenswrapper[2571]: W0805 17:21:35.091287    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod851488c2_41ee_4d47_95da_5710eca8d9e7.slice/crio-f01d8c39ad9f81ee72e225de8bdf455c5eff5444a7804d51bca52a40530c7e29 WatchSource:0}: Error finding container f01d8c39ad9f81ee72e225de8bdf455c5eff5444a7804d51bca52a40530c7e29: Status 404 returned error can't find the container with id f01d8c39ad9f81ee72e225de8bdf455c5eff5444a7804d51bca52a40530c7e29
Aug 05 17:21:35.092962 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:21:35.092938    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-64568bcd49-7qwp7"]
Aug 05 17:21:35.150870 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:21:35.150842    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-64568bcd49-7qwp7" event={"ID":"851488c2-41ee-4d47-95da-5710eca8d9e7","Type":"ContainerStarted","Data":"f01d8c39ad9f81ee72e225de8bdf455c5eff5444a7804d51bca52a40530c7e29"}
Aug 05 17:21:35.154617 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:21:35.154598    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-ffb9f44c4-xvvq9"
Aug 05 17:21:35.232940 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:21:35.232919    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-85fcbc755-6xmdc"]
Aug 05 17:21:39.172448 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:21:39.172415    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-64568bcd49-7qwp7" event={"ID":"851488c2-41ee-4d47-95da-5710eca8d9e7","Type":"ContainerStarted","Data":"c8b9469cc9c60faafbcca7abca48e832041d3990d09fe738139715a066704a53"}
Aug 05 17:21:39.172808 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:21:39.172534    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/seaweedfs-64568bcd49-7qwp7"
Aug 05 17:21:39.193757 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:21:39.193712    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/seaweedfs-64568bcd49-7qwp7" podStartSLOduration=1.87587595 podStartE2EDuration="5.193699385s" totalImagesPullingTime="3.317823435s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-05 17:21:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-05 17:21:39.193366203 +0000 UTC m=+526.560732523" watchObservedRunningTime="2026-08-05 17:21:39.193699385 +0000 UTC m=+526.561065731"
Aug 05 17:21:45.178817 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:21:45.178746    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/seaweedfs-64568bcd49-7qwp7"
Aug 05 17:22:00.251449 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:22:00.251395    2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-85fcbc755-6xmdc" podUID="825969e8-cbe8-448d-a808-3153afda1962" containerName="console" containerID="cri-o://5d49e33dd1a365ede3865e1973d67aa3ff4e9aab9f89cb4425e0bd4665e752db" gracePeriod=15
Aug 05 17:22:00.497620 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:22:00.497598    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-85fcbc755-6xmdc"
Aug 05 17:22:00.534759 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:22:00.534699    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zw9wj\" (UniqueName: \"kubernetes.io/projected/825969e8-cbe8-448d-a808-3153afda1962-kube-api-access-zw9wj\") pod \"825969e8-cbe8-448d-a808-3153afda1962\" (UID: \"825969e8-cbe8-448d-a808-3153afda1962\") "
Aug 05 17:22:00.534759 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:22:00.534739    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/825969e8-cbe8-448d-a808-3153afda1962-console-serving-cert\") pod \"825969e8-cbe8-448d-a808-3153afda1962\" (UID: \"825969e8-cbe8-448d-a808-3153afda1962\") "
Aug 05 17:22:00.534959 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:22:00.534763    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/825969e8-cbe8-448d-a808-3153afda1962-console-oauth-config\") pod \"825969e8-cbe8-448d-a808-3153afda1962\" (UID: \"825969e8-cbe8-448d-a808-3153afda1962\") "
Aug 05 17:22:00.534959 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:22:00.534784    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/825969e8-cbe8-448d-a808-3153afda1962-trusted-ca-bundle\") pod \"825969e8-cbe8-448d-a808-3153afda1962\" (UID: \"825969e8-cbe8-448d-a808-3153afda1962\") "
Aug 05 17:22:00.534959 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:22:00.534836    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/825969e8-cbe8-448d-a808-3153afda1962-service-ca\") pod \"825969e8-cbe8-448d-a808-3153afda1962\" (UID: \"825969e8-cbe8-448d-a808-3153afda1962\") "
Aug 05 17:22:00.534959 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:22:00.534874    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/825969e8-cbe8-448d-a808-3153afda1962-oauth-serving-cert\") pod \"825969e8-cbe8-448d-a808-3153afda1962\" (UID: \"825969e8-cbe8-448d-a808-3153afda1962\") "
Aug 05 17:22:00.534959 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:22:00.534938    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/825969e8-cbe8-448d-a808-3153afda1962-console-config\") pod \"825969e8-cbe8-448d-a808-3153afda1962\" (UID: \"825969e8-cbe8-448d-a808-3153afda1962\") "
Aug 05 17:22:00.535311 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:22:00.535283    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/825969e8-cbe8-448d-a808-3153afda1962-service-ca" (OuterVolumeSpecName: "service-ca") pod "825969e8-cbe8-448d-a808-3153afda1962" (UID: "825969e8-cbe8-448d-a808-3153afda1962"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 05 17:22:00.535451 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:22:00.535300    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/825969e8-cbe8-448d-a808-3153afda1962-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "825969e8-cbe8-448d-a808-3153afda1962" (UID: "825969e8-cbe8-448d-a808-3153afda1962"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 05 17:22:00.535518 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:22:00.535418    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/825969e8-cbe8-448d-a808-3153afda1962-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "825969e8-cbe8-448d-a808-3153afda1962" (UID: "825969e8-cbe8-448d-a808-3153afda1962"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 05 17:22:00.535572 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:22:00.535534    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/825969e8-cbe8-448d-a808-3153afda1962-console-config" (OuterVolumeSpecName: "console-config") pod "825969e8-cbe8-448d-a808-3153afda1962" (UID: "825969e8-cbe8-448d-a808-3153afda1962"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 05 17:22:00.536879 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:22:00.536850    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/825969e8-cbe8-448d-a808-3153afda1962-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "825969e8-cbe8-448d-a808-3153afda1962" (UID: "825969e8-cbe8-448d-a808-3153afda1962"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 05 17:22:00.537297 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:22:00.537277    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/825969e8-cbe8-448d-a808-3153afda1962-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "825969e8-cbe8-448d-a808-3153afda1962" (UID: "825969e8-cbe8-448d-a808-3153afda1962"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 05 17:22:00.537398 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:22:00.537294    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/825969e8-cbe8-448d-a808-3153afda1962-kube-api-access-zw9wj" (OuterVolumeSpecName: "kube-api-access-zw9wj") pod "825969e8-cbe8-448d-a808-3153afda1962" (UID: "825969e8-cbe8-448d-a808-3153afda1962"). InnerVolumeSpecName "kube-api-access-zw9wj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 05 17:22:00.635971 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:22:00.635945    2571 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/825969e8-cbe8-448d-a808-3153afda1962-console-config\") on node \"ip-10-0-135-2.ec2.internal\" DevicePath \"\""
Aug 05 17:22:00.635971 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:22:00.635971    2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zw9wj\" (UniqueName: \"kubernetes.io/projected/825969e8-cbe8-448d-a808-3153afda1962-kube-api-access-zw9wj\") on node \"ip-10-0-135-2.ec2.internal\" DevicePath \"\""
Aug 05 17:22:00.636141 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:22:00.635981    2571 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/825969e8-cbe8-448d-a808-3153afda1962-console-serving-cert\") on node \"ip-10-0-135-2.ec2.internal\" DevicePath \"\""
Aug 05 17:22:00.636141 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:22:00.635990    2571 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/825969e8-cbe8-448d-a808-3153afda1962-console-oauth-config\") on node \"ip-10-0-135-2.ec2.internal\" DevicePath \"\""
Aug 05 17:22:00.636141 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:22:00.635999    2571 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/825969e8-cbe8-448d-a808-3153afda1962-trusted-ca-bundle\") on node \"ip-10-0-135-2.ec2.internal\" DevicePath \"\""
Aug 05 17:22:00.636141 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:22:00.636008    2571 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/825969e8-cbe8-448d-a808-3153afda1962-service-ca\") on node \"ip-10-0-135-2.ec2.internal\" DevicePath \"\""
Aug 05 17:22:00.636141 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:22:00.636017    2571 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/825969e8-cbe8-448d-a808-3153afda1962-oauth-serving-cert\") on node \"ip-10-0-135-2.ec2.internal\" DevicePath \"\""
Aug 05 17:22:01.254792 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:22:01.254759    2571 generic.go:356] "Generic (PLEG): container finished" podID="825969e8-cbe8-448d-a808-3153afda1962" containerID="5d49e33dd1a365ede3865e1973d67aa3ff4e9aab9f89cb4425e0bd4665e752db" exitCode=0
Aug 05 17:22:01.255235 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:22:01.254806    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-85fcbc755-6xmdc"
Aug 05 17:22:01.255235 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:22:01.254825    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-85fcbc755-6xmdc" event={"ID":"825969e8-cbe8-448d-a808-3153afda1962","Type":"ContainerDied","Data":"5d49e33dd1a365ede3865e1973d67aa3ff4e9aab9f89cb4425e0bd4665e752db"}
Aug 05 17:22:01.255235 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:22:01.254863    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-85fcbc755-6xmdc" event={"ID":"825969e8-cbe8-448d-a808-3153afda1962","Type":"ContainerDied","Data":"3ac7348efacd93499b697e463129e649462a5afddebf48b655b42d4e82d33159"}
Aug 05 17:22:01.255235 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:22:01.254882    2571 scope.go:117] "RemoveContainer" containerID="5d49e33dd1a365ede3865e1973d67aa3ff4e9aab9f89cb4425e0bd4665e752db"
Aug 05 17:22:01.263229 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:22:01.263209    2571 scope.go:117] "RemoveContainer" containerID="5d49e33dd1a365ede3865e1973d67aa3ff4e9aab9f89cb4425e0bd4665e752db"
Aug 05 17:22:01.263522 ip-10-0-135-2 kubenswrapper[2571]: E0805 17:22:01.263501    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5d49e33dd1a365ede3865e1973d67aa3ff4e9aab9f89cb4425e0bd4665e752db\": container with ID starting with 5d49e33dd1a365ede3865e1973d67aa3ff4e9aab9f89cb4425e0bd4665e752db not found: ID does not exist" containerID="5d49e33dd1a365ede3865e1973d67aa3ff4e9aab9f89cb4425e0bd4665e752db"
Aug 05 17:22:01.263566 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:22:01.263533    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5d49e33dd1a365ede3865e1973d67aa3ff4e9aab9f89cb4425e0bd4665e752db"} err="failed to get container status \"5d49e33dd1a365ede3865e1973d67aa3ff4e9aab9f89cb4425e0bd4665e752db\": rpc error: code = NotFound desc = could not find container \"5d49e33dd1a365ede3865e1973d67aa3ff4e9aab9f89cb4425e0bd4665e752db\": container with ID starting with 5d49e33dd1a365ede3865e1973d67aa3ff4e9aab9f89cb4425e0bd4665e752db not found: ID does not exist"
Aug 05 17:22:01.273201 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:22:01.273181    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-85fcbc755-6xmdc"]
Aug 05 17:22:01.277921 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:22:01.277900    2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-85fcbc755-6xmdc"]
Aug 05 17:22:03.141820 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:22:03.141779    2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="825969e8-cbe8-448d-a808-3153afda1962" path="/var/lib/kubelet/pods/825969e8-cbe8-448d-a808-3153afda1962/volumes"
Aug 05 17:22:52.972634 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:22:52.972598    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/odh-model-controller-8588677cfb-f2rz7"]
Aug 05 17:22:52.973077 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:22:52.972912    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="825969e8-cbe8-448d-a808-3153afda1962" containerName="console"
Aug 05 17:22:52.973077 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:22:52.972924    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="825969e8-cbe8-448d-a808-3153afda1962" containerName="console"
Aug 05 17:22:52.973077 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:22:52.972980    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="825969e8-cbe8-448d-a808-3153afda1962" containerName="console"
Aug 05 17:22:52.975713 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:22:52.975696    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/odh-model-controller-8588677cfb-f2rz7"
Aug 05 17:22:52.981591 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:22:52.981565    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"odh-model-controller-webhook-cert\""
Aug 05 17:22:52.981753 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:22:52.981730    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"odh-model-controller-dockercfg-htbnn\""
Aug 05 17:22:52.981877 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:22:52.981777    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"odh-model-controller-metrics-tls\""
Aug 05 17:22:52.993099 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:22:52.993074    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/odh-model-controller-8588677cfb-f2rz7"]
Aug 05 17:22:53.156882 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:22:53.156850    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/47655fb1-9898-4205-9753-a80dbc39a6ce-metrics-certs\") pod \"odh-model-controller-8588677cfb-f2rz7\" (UID: \"47655fb1-9898-4205-9753-a80dbc39a6ce\") " pod="kserve/odh-model-controller-8588677cfb-f2rz7"
Aug 05 17:22:53.156882 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:22:53.156885    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g9gts\" (UniqueName: \"kubernetes.io/projected/47655fb1-9898-4205-9753-a80dbc39a6ce-kube-api-access-g9gts\") pod \"odh-model-controller-8588677cfb-f2rz7\" (UID: \"47655fb1-9898-4205-9753-a80dbc39a6ce\") " pod="kserve/odh-model-controller-8588677cfb-f2rz7"
Aug 05 17:22:53.157065 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:22:53.156913    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/47655fb1-9898-4205-9753-a80dbc39a6ce-cert\") pod \"odh-model-controller-8588677cfb-f2rz7\" (UID: \"47655fb1-9898-4205-9753-a80dbc39a6ce\") " pod="kserve/odh-model-controller-8588677cfb-f2rz7"
Aug 05 17:22:53.257720 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:22:53.257672    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/47655fb1-9898-4205-9753-a80dbc39a6ce-metrics-certs\") pod \"odh-model-controller-8588677cfb-f2rz7\" (UID: \"47655fb1-9898-4205-9753-a80dbc39a6ce\") " pod="kserve/odh-model-controller-8588677cfb-f2rz7"
Aug 05 17:22:53.257902 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:22:53.257740    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g9gts\" (UniqueName: \"kubernetes.io/projected/47655fb1-9898-4205-9753-a80dbc39a6ce-kube-api-access-g9gts\") pod \"odh-model-controller-8588677cfb-f2rz7\" (UID: \"47655fb1-9898-4205-9753-a80dbc39a6ce\") " pod="kserve/odh-model-controller-8588677cfb-f2rz7"
Aug 05 17:22:53.257902 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:22:53.257797    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/47655fb1-9898-4205-9753-a80dbc39a6ce-cert\") pod \"odh-model-controller-8588677cfb-f2rz7\" (UID: \"47655fb1-9898-4205-9753-a80dbc39a6ce\") " pod="kserve/odh-model-controller-8588677cfb-f2rz7"
Aug 05 17:22:53.258015 ip-10-0-135-2 kubenswrapper[2571]: E0805 17:22:53.257959    2571 secret.go:189] Couldn't get secret kserve/odh-model-controller-webhook-cert: secret "odh-model-controller-webhook-cert" not found
Aug 05 17:22:53.258101 ip-10-0-135-2 kubenswrapper[2571]: E0805 17:22:53.258089    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/47655fb1-9898-4205-9753-a80dbc39a6ce-cert podName:47655fb1-9898-4205-9753-a80dbc39a6ce nodeName:}" failed. No retries permitted until 2026-08-05 17:22:53.758059826 +0000 UTC m=+601.125426163 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/47655fb1-9898-4205-9753-a80dbc39a6ce-cert") pod "odh-model-controller-8588677cfb-f2rz7" (UID: "47655fb1-9898-4205-9753-a80dbc39a6ce") : secret "odh-model-controller-webhook-cert" not found
Aug 05 17:22:53.260250 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:22:53.260221    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/47655fb1-9898-4205-9753-a80dbc39a6ce-metrics-certs\") pod \"odh-model-controller-8588677cfb-f2rz7\" (UID: \"47655fb1-9898-4205-9753-a80dbc39a6ce\") " pod="kserve/odh-model-controller-8588677cfb-f2rz7"
Aug 05 17:22:53.267439 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:22:53.267418    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g9gts\" (UniqueName: \"kubernetes.io/projected/47655fb1-9898-4205-9753-a80dbc39a6ce-kube-api-access-g9gts\") pod \"odh-model-controller-8588677cfb-f2rz7\" (UID: \"47655fb1-9898-4205-9753-a80dbc39a6ce\") " pod="kserve/odh-model-controller-8588677cfb-f2rz7"
Aug 05 17:22:53.761911 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:22:53.761875    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/47655fb1-9898-4205-9753-a80dbc39a6ce-cert\") pod \"odh-model-controller-8588677cfb-f2rz7\" (UID: \"47655fb1-9898-4205-9753-a80dbc39a6ce\") " pod="kserve/odh-model-controller-8588677cfb-f2rz7"
Aug 05 17:22:53.764140 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:22:53.764118    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/47655fb1-9898-4205-9753-a80dbc39a6ce-cert\") pod \"odh-model-controller-8588677cfb-f2rz7\" (UID: \"47655fb1-9898-4205-9753-a80dbc39a6ce\") " pod="kserve/odh-model-controller-8588677cfb-f2rz7"
Aug 05 17:22:53.886609 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:22:53.886574    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/odh-model-controller-8588677cfb-f2rz7"
Aug 05 17:22:54.014436 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:22:54.014369    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/odh-model-controller-8588677cfb-f2rz7"]
Aug 05 17:22:54.017755 ip-10-0-135-2 kubenswrapper[2571]: W0805 17:22:54.017728    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod47655fb1_9898_4205_9753_a80dbc39a6ce.slice/crio-7c0ad143493bc5b58e8a644893cbd215f010ec04931786c211b515da155de1b0 WatchSource:0}: Error finding container 7c0ad143493bc5b58e8a644893cbd215f010ec04931786c211b515da155de1b0: Status 404 returned error can't find the container with id 7c0ad143493bc5b58e8a644893cbd215f010ec04931786c211b515da155de1b0
Aug 05 17:22:54.458475 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:22:54.458395    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/odh-model-controller-8588677cfb-f2rz7" event={"ID":"47655fb1-9898-4205-9753-a80dbc39a6ce","Type":"ContainerStarted","Data":"7c0ad143493bc5b58e8a644893cbd215f010ec04931786c211b515da155de1b0"}
Aug 05 17:22:58.480919 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:22:58.480879    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/odh-model-controller-8588677cfb-f2rz7" event={"ID":"47655fb1-9898-4205-9753-a80dbc39a6ce","Type":"ContainerStarted","Data":"c671514cf4db65de248ce97d8cdce1f0bd68eac3d833e240db37447d458a747b"}
Aug 05 17:22:58.481330 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:22:58.481015    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/odh-model-controller-8588677cfb-f2rz7"
Aug 05 17:22:58.501732 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:22:58.501680    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/odh-model-controller-8588677cfb-f2rz7" podStartSLOduration=2.934489162 podStartE2EDuration="6.501665123s" totalImagesPullingTime="3.567175961s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-05 17:22:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-05 17:22:58.500722132 +0000 UTC m=+605.868088475" watchObservedRunningTime="2026-08-05 17:22:58.501665123 +0000 UTC m=+605.869031467"
Aug 05 17:23:09.486840 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:23:09.486802    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/odh-model-controller-8588677cfb-f2rz7"
Aug 05 17:24:41.096247 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:24:41.096216    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-829a9-predictor-54f97d665d-jlvfn"]
Aug 05 17:24:41.099556 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:24:41.099540    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-829a9-predictor-54f97d665d-jlvfn"
Aug 05 17:24:41.102279 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:24:41.102255    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-829a9-kube-rbac-proxy-sar-config\""
Aug 05 17:24:41.102427 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:24:41.102292    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-829a9-predictor-serving-cert\""
Aug 05 17:24:41.102482 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:24:41.102466    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Aug 05 17:24:41.103258 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:24:41.103239    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Aug 05 17:24:41.103346 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:24:41.103251    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-5hw69\""
Aug 05 17:24:41.109919 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:24:41.109900    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-829a9-predictor-54f97d665d-jlvfn"]
Aug 05 17:24:41.210369 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:24:41.210338    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xkpvq\" (UniqueName: \"kubernetes.io/projected/54086e09-ff4e-4168-94f9-58df784d7811-kube-api-access-xkpvq\") pod \"success-200-isvc-829a9-predictor-54f97d665d-jlvfn\" (UID: \"54086e09-ff4e-4168-94f9-58df784d7811\") " pod="kserve-ci-e2e-test/success-200-isvc-829a9-predictor-54f97d665d-jlvfn"
Aug 05 17:24:41.210543 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:24:41.210407    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/54086e09-ff4e-4168-94f9-58df784d7811-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-829a9-predictor-54f97d665d-jlvfn\" (UID: \"54086e09-ff4e-4168-94f9-58df784d7811\") " pod="kserve-ci-e2e-test/success-200-isvc-829a9-predictor-54f97d665d-jlvfn"
Aug 05 17:24:41.210543 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:24:41.210436    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/54086e09-ff4e-4168-94f9-58df784d7811-proxy-tls\") pod \"success-200-isvc-829a9-predictor-54f97d665d-jlvfn\" (UID: \"54086e09-ff4e-4168-94f9-58df784d7811\") " pod="kserve-ci-e2e-test/success-200-isvc-829a9-predictor-54f97d665d-jlvfn"
Aug 05 17:24:41.310854 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:24:41.310817    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xkpvq\" (UniqueName: \"kubernetes.io/projected/54086e09-ff4e-4168-94f9-58df784d7811-kube-api-access-xkpvq\") pod \"success-200-isvc-829a9-predictor-54f97d665d-jlvfn\" (UID: \"54086e09-ff4e-4168-94f9-58df784d7811\") " pod="kserve-ci-e2e-test/success-200-isvc-829a9-predictor-54f97d665d-jlvfn"
Aug 05 17:24:41.311033 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:24:41.310873    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/54086e09-ff4e-4168-94f9-58df784d7811-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-829a9-predictor-54f97d665d-jlvfn\" (UID: \"54086e09-ff4e-4168-94f9-58df784d7811\") " pod="kserve-ci-e2e-test/success-200-isvc-829a9-predictor-54f97d665d-jlvfn"
Aug 05 17:24:41.311033 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:24:41.310902    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/54086e09-ff4e-4168-94f9-58df784d7811-proxy-tls\") pod \"success-200-isvc-829a9-predictor-54f97d665d-jlvfn\" (UID: \"54086e09-ff4e-4168-94f9-58df784d7811\") " pod="kserve-ci-e2e-test/success-200-isvc-829a9-predictor-54f97d665d-jlvfn"
Aug 05 17:24:41.311565 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:24:41.311530    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/54086e09-ff4e-4168-94f9-58df784d7811-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-829a9-predictor-54f97d665d-jlvfn\" (UID: \"54086e09-ff4e-4168-94f9-58df784d7811\") " pod="kserve-ci-e2e-test/success-200-isvc-829a9-predictor-54f97d665d-jlvfn"
Aug 05 17:24:41.313459 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:24:41.313431    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/54086e09-ff4e-4168-94f9-58df784d7811-proxy-tls\") pod \"success-200-isvc-829a9-predictor-54f97d665d-jlvfn\" (UID: \"54086e09-ff4e-4168-94f9-58df784d7811\") " pod="kserve-ci-e2e-test/success-200-isvc-829a9-predictor-54f97d665d-jlvfn"
Aug 05 17:24:41.322394 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:24:41.322351    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xkpvq\" (UniqueName: \"kubernetes.io/projected/54086e09-ff4e-4168-94f9-58df784d7811-kube-api-access-xkpvq\") pod \"success-200-isvc-829a9-predictor-54f97d665d-jlvfn\" (UID: \"54086e09-ff4e-4168-94f9-58df784d7811\") " pod="kserve-ci-e2e-test/success-200-isvc-829a9-predictor-54f97d665d-jlvfn"
Aug 05 17:24:41.383992 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:24:41.383924    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-829a9-predictor-67f4589d75-vnfjn"]
Aug 05 17:24:41.387500 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:24:41.387482    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-829a9-predictor-67f4589d75-vnfjn"
Aug 05 17:24:41.390079 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:24:41.390059    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-829a9-kube-rbac-proxy-sar-config\""
Aug 05 17:24:41.390298 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:24:41.390285    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-829a9-predictor-serving-cert\""
Aug 05 17:24:41.398682 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:24:41.398662    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-829a9-predictor-67f4589d75-vnfjn"]
Aug 05 17:24:41.410302 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:24:41.410283    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-829a9-predictor-54f97d665d-jlvfn"
Aug 05 17:24:41.512886 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:24:41.512848    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/419c1eaa-2d46-4226-94b6-9d1bbe38e8f0-proxy-tls\") pod \"error-404-isvc-829a9-predictor-67f4589d75-vnfjn\" (UID: \"419c1eaa-2d46-4226-94b6-9d1bbe38e8f0\") " pod="kserve-ci-e2e-test/error-404-isvc-829a9-predictor-67f4589d75-vnfjn"
Aug 05 17:24:41.513038 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:24:41.512977    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5b56t\" (UniqueName: \"kubernetes.io/projected/419c1eaa-2d46-4226-94b6-9d1bbe38e8f0-kube-api-access-5b56t\") pod \"error-404-isvc-829a9-predictor-67f4589d75-vnfjn\" (UID: \"419c1eaa-2d46-4226-94b6-9d1bbe38e8f0\") " pod="kserve-ci-e2e-test/error-404-isvc-829a9-predictor-67f4589d75-vnfjn"
Aug 05 17:24:41.513168 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:24:41.513039    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/419c1eaa-2d46-4226-94b6-9d1bbe38e8f0-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-829a9-predictor-67f4589d75-vnfjn\" (UID: \"419c1eaa-2d46-4226-94b6-9d1bbe38e8f0\") " pod="kserve-ci-e2e-test/error-404-isvc-829a9-predictor-67f4589d75-vnfjn"
Aug 05 17:24:41.543536 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:24:41.543491    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-829a9-predictor-54f97d665d-jlvfn"]
Aug 05 17:24:41.545985 ip-10-0-135-2 kubenswrapper[2571]: W0805 17:24:41.545948    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod54086e09_ff4e_4168_94f9_58df784d7811.slice/crio-7b56f71fc2dd75096b1b1e504b610b353feb8a0f2b574227c36b8255d2532bac WatchSource:0}: Error finding container 7b56f71fc2dd75096b1b1e504b610b353feb8a0f2b574227c36b8255d2532bac: Status 404 returned error can't find the container with id 7b56f71fc2dd75096b1b1e504b610b353feb8a0f2b574227c36b8255d2532bac
Aug 05 17:24:41.614296 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:24:41.614267    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5b56t\" (UniqueName: \"kubernetes.io/projected/419c1eaa-2d46-4226-94b6-9d1bbe38e8f0-kube-api-access-5b56t\") pod \"error-404-isvc-829a9-predictor-67f4589d75-vnfjn\" (UID: \"419c1eaa-2d46-4226-94b6-9d1bbe38e8f0\") " pod="kserve-ci-e2e-test/error-404-isvc-829a9-predictor-67f4589d75-vnfjn"
Aug 05 17:24:41.614428 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:24:41.614304    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/419c1eaa-2d46-4226-94b6-9d1bbe38e8f0-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-829a9-predictor-67f4589d75-vnfjn\" (UID: \"419c1eaa-2d46-4226-94b6-9d1bbe38e8f0\") " pod="kserve-ci-e2e-test/error-404-isvc-829a9-predictor-67f4589d75-vnfjn"
Aug 05 17:24:41.614428 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:24:41.614351    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/419c1eaa-2d46-4226-94b6-9d1bbe38e8f0-proxy-tls\") pod \"error-404-isvc-829a9-predictor-67f4589d75-vnfjn\" (UID: \"419c1eaa-2d46-4226-94b6-9d1bbe38e8f0\") " pod="kserve-ci-e2e-test/error-404-isvc-829a9-predictor-67f4589d75-vnfjn"
Aug 05 17:24:41.614930 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:24:41.614908    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/419c1eaa-2d46-4226-94b6-9d1bbe38e8f0-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-829a9-predictor-67f4589d75-vnfjn\" (UID: \"419c1eaa-2d46-4226-94b6-9d1bbe38e8f0\") " pod="kserve-ci-e2e-test/error-404-isvc-829a9-predictor-67f4589d75-vnfjn"
Aug 05 17:24:41.616665 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:24:41.616646    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/419c1eaa-2d46-4226-94b6-9d1bbe38e8f0-proxy-tls\") pod \"error-404-isvc-829a9-predictor-67f4589d75-vnfjn\" (UID: \"419c1eaa-2d46-4226-94b6-9d1bbe38e8f0\") " pod="kserve-ci-e2e-test/error-404-isvc-829a9-predictor-67f4589d75-vnfjn"
Aug 05 17:24:41.622971 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:24:41.622955    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5b56t\" (UniqueName: \"kubernetes.io/projected/419c1eaa-2d46-4226-94b6-9d1bbe38e8f0-kube-api-access-5b56t\") pod \"error-404-isvc-829a9-predictor-67f4589d75-vnfjn\" (UID: \"419c1eaa-2d46-4226-94b6-9d1bbe38e8f0\") " pod="kserve-ci-e2e-test/error-404-isvc-829a9-predictor-67f4589d75-vnfjn"
Aug 05 17:24:41.698021 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:24:41.697965    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-829a9-predictor-67f4589d75-vnfjn"
Aug 05 17:24:41.825537 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:24:41.825514    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-829a9-predictor-67f4589d75-vnfjn"]
Aug 05 17:24:41.827413 ip-10-0-135-2 kubenswrapper[2571]: W0805 17:24:41.827387    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod419c1eaa_2d46_4226_94b6_9d1bbe38e8f0.slice/crio-a6303051f0f7c29622013caab4771e8352900c4c089359da5b3940e0eeb428cd WatchSource:0}: Error finding container a6303051f0f7c29622013caab4771e8352900c4c089359da5b3940e0eeb428cd: Status 404 returned error can't find the container with id a6303051f0f7c29622013caab4771e8352900c4c089359da5b3940e0eeb428cd
Aug 05 17:24:41.883594 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:24:41.883563    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-829a9-predictor-67f4589d75-vnfjn" event={"ID":"419c1eaa-2d46-4226-94b6-9d1bbe38e8f0","Type":"ContainerStarted","Data":"a6303051f0f7c29622013caab4771e8352900c4c089359da5b3940e0eeb428cd"}
Aug 05 17:24:41.884747 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:24:41.884717    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-829a9-predictor-54f97d665d-jlvfn" event={"ID":"54086e09-ff4e-4168-94f9-58df784d7811","Type":"ContainerStarted","Data":"7b56f71fc2dd75096b1b1e504b610b353feb8a0f2b574227c36b8255d2532bac"}
Aug 05 17:24:57.986276 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:24:57.986239    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-829a9-predictor-67f4589d75-vnfjn" event={"ID":"419c1eaa-2d46-4226-94b6-9d1bbe38e8f0","Type":"ContainerStarted","Data":"4d04358342b0668d72c552d3ae9b29cecd93a82b63e4e89540749c066466c62b"}
Aug 05 17:24:57.987842 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:24:57.987818    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-829a9-predictor-54f97d665d-jlvfn" event={"ID":"54086e09-ff4e-4168-94f9-58df784d7811","Type":"ContainerStarted","Data":"30c806bcd7bbb7d6fbb627475a4c798e795653c27c359b128cf83f708cf8d645"}
Aug 05 17:25:01.006398 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:25:01.006342    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-829a9-predictor-54f97d665d-jlvfn" event={"ID":"54086e09-ff4e-4168-94f9-58df784d7811","Type":"ContainerStarted","Data":"e600b2dc50291f6d6b7c9d2f3fcee84a864341d1cb20db3555a82c8f17c7d4e5"}
Aug 05 17:25:01.006850 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:25:01.006517    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-829a9-predictor-54f97d665d-jlvfn"
Aug 05 17:25:01.006850 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:25:01.006572    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-829a9-predictor-54f97d665d-jlvfn"
Aug 05 17:25:01.007913 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:25:01.007876    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-829a9-predictor-54f97d665d-jlvfn" podUID="54086e09-ff4e-4168-94f9-58df784d7811" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.28:8080: connect: connection refused"
Aug 05 17:25:01.008293 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:25:01.008269    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-829a9-predictor-67f4589d75-vnfjn" event={"ID":"419c1eaa-2d46-4226-94b6-9d1bbe38e8f0","Type":"ContainerStarted","Data":"07edd901daf5fbe346b6d5104d7e913d0042ea05d62b9b932f28574db58a0095"}
Aug 05 17:25:01.008450 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:25:01.008434    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-829a9-predictor-67f4589d75-vnfjn"
Aug 05 17:25:01.008500 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:25:01.008477    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-829a9-predictor-67f4589d75-vnfjn"
Aug 05 17:25:01.009589 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:25:01.009567    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-829a9-predictor-67f4589d75-vnfjn" podUID="419c1eaa-2d46-4226-94b6-9d1bbe38e8f0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused"
Aug 05 17:25:01.031612 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:25:01.031568    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/success-200-isvc-829a9-predictor-54f97d665d-jlvfn" podStartSLOduration=1.37298475 podStartE2EDuration="20.03155666s" totalImagesPullingTime="18.65857191s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-05 17:24:41 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-05 17:25:01.030246686 +0000 UTC m=+728.397613029" watchObservedRunningTime="2026-08-05 17:25:01.03155666 +0000 UTC m=+728.398923002"
Aug 05 17:25:01.050240 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:25:01.050195    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/error-404-isvc-829a9-predictor-67f4589d75-vnfjn" podStartSLOduration=1.684873481 podStartE2EDuration="20.050182165s" totalImagesPullingTime="18.365308684s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-05 17:24:41 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-05 17:25:01.049208985 +0000 UTC m=+728.416575354" watchObservedRunningTime="2026-08-05 17:25:01.050182165 +0000 UTC m=+728.417548508"
Aug 05 17:25:02.013309 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:25:02.013269    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-829a9-predictor-67f4589d75-vnfjn" podUID="419c1eaa-2d46-4226-94b6-9d1bbe38e8f0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused"
Aug 05 17:25:02.013708 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:25:02.013269    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-829a9-predictor-54f97d665d-jlvfn" podUID="54086e09-ff4e-4168-94f9-58df784d7811" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.28:8080: connect: connection refused"
Aug 05 17:25:07.018159 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:25:07.018122    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-829a9-predictor-67f4589d75-vnfjn"
Aug 05 17:25:07.018573 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:25:07.018440    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-829a9-predictor-54f97d665d-jlvfn"
Aug 05 17:25:07.018621 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:25:07.018558    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-829a9-predictor-67f4589d75-vnfjn" podUID="419c1eaa-2d46-4226-94b6-9d1bbe38e8f0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused"
Aug 05 17:25:07.018856 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:25:07.018831    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-829a9-predictor-54f97d665d-jlvfn" podUID="54086e09-ff4e-4168-94f9-58df784d7811" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.28:8080: connect: connection refused"
Aug 05 17:25:17.018606 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:25:17.018564    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-829a9-predictor-67f4589d75-vnfjn" podUID="419c1eaa-2d46-4226-94b6-9d1bbe38e8f0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused"
Aug 05 17:25:17.019067 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:25:17.018808    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-829a9-predictor-54f97d665d-jlvfn" podUID="54086e09-ff4e-4168-94f9-58df784d7811" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.28:8080: connect: connection refused"
Aug 05 17:25:27.019406 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:25:27.019354    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-829a9-predictor-54f97d665d-jlvfn" podUID="54086e09-ff4e-4168-94f9-58df784d7811" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.28:8080: connect: connection refused"
Aug 05 17:25:27.019794 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:25:27.019356    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-829a9-predictor-67f4589d75-vnfjn" podUID="419c1eaa-2d46-4226-94b6-9d1bbe38e8f0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused"
Aug 05 17:25:37.019206 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:25:37.019165    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-829a9-predictor-54f97d665d-jlvfn" podUID="54086e09-ff4e-4168-94f9-58df784d7811" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.28:8080: connect: connection refused"
Aug 05 17:25:37.019599 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:25:37.019174    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-829a9-predictor-67f4589d75-vnfjn" podUID="419c1eaa-2d46-4226-94b6-9d1bbe38e8f0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused"
Aug 05 17:25:47.019535 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:25:47.019504    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-829a9-predictor-67f4589d75-vnfjn"
Aug 05 17:25:47.019928 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:25:47.019580    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-829a9-predictor-54f97d665d-jlvfn"
Aug 05 17:26:11.239860 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:26:11.239827    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-829a9-predictor-54f97d665d-jlvfn"]
Aug 05 17:26:11.240351 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:26:11.240095    2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-829a9-predictor-54f97d665d-jlvfn" podUID="54086e09-ff4e-4168-94f9-58df784d7811" containerName="kserve-container" containerID="cri-o://30c806bcd7bbb7d6fbb627475a4c798e795653c27c359b128cf83f708cf8d645" gracePeriod=30
Aug 05 17:26:11.240351 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:26:11.240159    2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-829a9-predictor-54f97d665d-jlvfn" podUID="54086e09-ff4e-4168-94f9-58df784d7811" containerName="kube-rbac-proxy" containerID="cri-o://e600b2dc50291f6d6b7c9d2f3fcee84a864341d1cb20db3555a82c8f17c7d4e5" gracePeriod=30
Aug 05 17:26:11.297479 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:26:11.297448    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-a0ee2-predictor-69b645cc54-6zzvp"]
Aug 05 17:26:11.301020 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:26:11.300995    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-a0ee2-predictor-69b645cc54-6zzvp"
Aug 05 17:26:11.303281 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:26:11.303261    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-a0ee2-kube-rbac-proxy-sar-config\""
Aug 05 17:26:11.303415 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:26:11.303259    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-a0ee2-predictor-serving-cert\""
Aug 05 17:26:11.313950 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:26:11.311792    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-829a9-predictor-67f4589d75-vnfjn"]
Aug 05 17:26:11.313950 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:26:11.312128    2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-829a9-predictor-67f4589d75-vnfjn" podUID="419c1eaa-2d46-4226-94b6-9d1bbe38e8f0" containerName="kserve-container" containerID="cri-o://4d04358342b0668d72c552d3ae9b29cecd93a82b63e4e89540749c066466c62b" gracePeriod=30
Aug 05 17:26:11.313950 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:26:11.312314    2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-829a9-predictor-67f4589d75-vnfjn" podUID="419c1eaa-2d46-4226-94b6-9d1bbe38e8f0" containerName="kube-rbac-proxy" containerID="cri-o://07edd901daf5fbe346b6d5104d7e913d0042ea05d62b9b932f28574db58a0095" gracePeriod=30
Aug 05 17:26:11.316937 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:26:11.316913    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-a0ee2-predictor-69b645cc54-6zzvp"]
Aug 05 17:26:11.372450 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:26:11.372418    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/cf5fb621-292d-454c-88d6-e66edcd9dfb5-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-a0ee2-predictor-69b645cc54-6zzvp\" (UID: \"cf5fb621-292d-454c-88d6-e66edcd9dfb5\") " pod="kserve-ci-e2e-test/success-200-isvc-a0ee2-predictor-69b645cc54-6zzvp"
Aug 05 17:26:11.372608 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:26:11.372478    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/cf5fb621-292d-454c-88d6-e66edcd9dfb5-proxy-tls\") pod \"success-200-isvc-a0ee2-predictor-69b645cc54-6zzvp\" (UID: \"cf5fb621-292d-454c-88d6-e66edcd9dfb5\") " pod="kserve-ci-e2e-test/success-200-isvc-a0ee2-predictor-69b645cc54-6zzvp"
Aug 05 17:26:11.372608 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:26:11.372562    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-brnzk\" (UniqueName: \"kubernetes.io/projected/cf5fb621-292d-454c-88d6-e66edcd9dfb5-kube-api-access-brnzk\") pod \"success-200-isvc-a0ee2-predictor-69b645cc54-6zzvp\" (UID: \"cf5fb621-292d-454c-88d6-e66edcd9dfb5\") " pod="kserve-ci-e2e-test/success-200-isvc-a0ee2-predictor-69b645cc54-6zzvp"
Aug 05 17:26:11.415054 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:26:11.415018    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-a0ee2-predictor-7f8d8c965f-kj7sw"]
Aug 05 17:26:11.418445 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:26:11.418414    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-a0ee2-predictor-7f8d8c965f-kj7sw"
Aug 05 17:26:11.420851 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:26:11.420830    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-a0ee2-predictor-serving-cert\""
Aug 05 17:26:11.420851 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:26:11.420848    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-a0ee2-kube-rbac-proxy-sar-config\""
Aug 05 17:26:11.429662 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:26:11.429637    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-a0ee2-predictor-7f8d8c965f-kj7sw"]
Aug 05 17:26:11.473760 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:26:11.473730    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gvv8l\" (UniqueName: \"kubernetes.io/projected/108727d9-7bf4-4042-9a0f-8b07a5a12f91-kube-api-access-gvv8l\") pod \"error-404-isvc-a0ee2-predictor-7f8d8c965f-kj7sw\" (UID: \"108727d9-7bf4-4042-9a0f-8b07a5a12f91\") " pod="kserve-ci-e2e-test/error-404-isvc-a0ee2-predictor-7f8d8c965f-kj7sw"
Aug 05 17:26:11.473880 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:26:11.473782    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/cf5fb621-292d-454c-88d6-e66edcd9dfb5-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-a0ee2-predictor-69b645cc54-6zzvp\" (UID: \"cf5fb621-292d-454c-88d6-e66edcd9dfb5\") " pod="kserve-ci-e2e-test/success-200-isvc-a0ee2-predictor-69b645cc54-6zzvp"
Aug 05 17:26:11.473880 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:26:11.473818    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/108727d9-7bf4-4042-9a0f-8b07a5a12f91-proxy-tls\") pod \"error-404-isvc-a0ee2-predictor-7f8d8c965f-kj7sw\" (UID: \"108727d9-7bf4-4042-9a0f-8b07a5a12f91\") " pod="kserve-ci-e2e-test/error-404-isvc-a0ee2-predictor-7f8d8c965f-kj7sw"
Aug 05 17:26:11.473880 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:26:11.473838    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/108727d9-7bf4-4042-9a0f-8b07a5a12f91-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-a0ee2-predictor-7f8d8c965f-kj7sw\" (UID: \"108727d9-7bf4-4042-9a0f-8b07a5a12f91\") " pod="kserve-ci-e2e-test/error-404-isvc-a0ee2-predictor-7f8d8c965f-kj7sw"
Aug 05 17:26:11.473880 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:26:11.473871    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/cf5fb621-292d-454c-88d6-e66edcd9dfb5-proxy-tls\") pod \"success-200-isvc-a0ee2-predictor-69b645cc54-6zzvp\" (UID: \"cf5fb621-292d-454c-88d6-e66edcd9dfb5\") " pod="kserve-ci-e2e-test/success-200-isvc-a0ee2-predictor-69b645cc54-6zzvp"
Aug 05 17:26:11.474048 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:26:11.473940    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-brnzk\" (UniqueName: \"kubernetes.io/projected/cf5fb621-292d-454c-88d6-e66edcd9dfb5-kube-api-access-brnzk\") pod \"success-200-isvc-a0ee2-predictor-69b645cc54-6zzvp\" (UID: \"cf5fb621-292d-454c-88d6-e66edcd9dfb5\") " pod="kserve-ci-e2e-test/success-200-isvc-a0ee2-predictor-69b645cc54-6zzvp"
Aug 05 17:26:11.474048 ip-10-0-135-2 kubenswrapper[2571]: E0805 17:26:11.473975    2571 secret.go:189] Couldn't get secret kserve-ci-e2e-test/success-200-isvc-a0ee2-predictor-serving-cert: secret "success-200-isvc-a0ee2-predictor-serving-cert" not found
Aug 05 17:26:11.474048 ip-10-0-135-2 kubenswrapper[2571]: E0805 17:26:11.474035    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cf5fb621-292d-454c-88d6-e66edcd9dfb5-proxy-tls podName:cf5fb621-292d-454c-88d6-e66edcd9dfb5 nodeName:}" failed. No retries permitted until 2026-08-05 17:26:11.974015824 +0000 UTC m=+799.341382148 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/cf5fb621-292d-454c-88d6-e66edcd9dfb5-proxy-tls") pod "success-200-isvc-a0ee2-predictor-69b645cc54-6zzvp" (UID: "cf5fb621-292d-454c-88d6-e66edcd9dfb5") : secret "success-200-isvc-a0ee2-predictor-serving-cert" not found
Aug 05 17:26:11.474424 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:26:11.474407    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/cf5fb621-292d-454c-88d6-e66edcd9dfb5-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-a0ee2-predictor-69b645cc54-6zzvp\" (UID: \"cf5fb621-292d-454c-88d6-e66edcd9dfb5\") " pod="kserve-ci-e2e-test/success-200-isvc-a0ee2-predictor-69b645cc54-6zzvp"
Aug 05 17:26:11.482752 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:26:11.482730    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-brnzk\" (UniqueName: \"kubernetes.io/projected/cf5fb621-292d-454c-88d6-e66edcd9dfb5-kube-api-access-brnzk\") pod \"success-200-isvc-a0ee2-predictor-69b645cc54-6zzvp\" (UID: \"cf5fb621-292d-454c-88d6-e66edcd9dfb5\") " pod="kserve-ci-e2e-test/success-200-isvc-a0ee2-predictor-69b645cc54-6zzvp"
Aug 05 17:26:11.575518 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:26:11.575431    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gvv8l\" (UniqueName: \"kubernetes.io/projected/108727d9-7bf4-4042-9a0f-8b07a5a12f91-kube-api-access-gvv8l\") pod \"error-404-isvc-a0ee2-predictor-7f8d8c965f-kj7sw\" (UID: \"108727d9-7bf4-4042-9a0f-8b07a5a12f91\") " pod="kserve-ci-e2e-test/error-404-isvc-a0ee2-predictor-7f8d8c965f-kj7sw"
Aug 05 17:26:11.575518 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:26:11.575484    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/108727d9-7bf4-4042-9a0f-8b07a5a12f91-proxy-tls\") pod \"error-404-isvc-a0ee2-predictor-7f8d8c965f-kj7sw\" (UID: \"108727d9-7bf4-4042-9a0f-8b07a5a12f91\") " pod="kserve-ci-e2e-test/error-404-isvc-a0ee2-predictor-7f8d8c965f-kj7sw"
Aug 05 17:26:11.575518 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:26:11.575512    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/108727d9-7bf4-4042-9a0f-8b07a5a12f91-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-a0ee2-predictor-7f8d8c965f-kj7sw\" (UID: \"108727d9-7bf4-4042-9a0f-8b07a5a12f91\") " pod="kserve-ci-e2e-test/error-404-isvc-a0ee2-predictor-7f8d8c965f-kj7sw"
Aug 05 17:26:11.576221 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:26:11.576196    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/108727d9-7bf4-4042-9a0f-8b07a5a12f91-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-a0ee2-predictor-7f8d8c965f-kj7sw\" (UID: \"108727d9-7bf4-4042-9a0f-8b07a5a12f91\") " pod="kserve-ci-e2e-test/error-404-isvc-a0ee2-predictor-7f8d8c965f-kj7sw"
Aug 05 17:26:11.577952 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:26:11.577935    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/108727d9-7bf4-4042-9a0f-8b07a5a12f91-proxy-tls\") pod \"error-404-isvc-a0ee2-predictor-7f8d8c965f-kj7sw\" (UID: \"108727d9-7bf4-4042-9a0f-8b07a5a12f91\") " pod="kserve-ci-e2e-test/error-404-isvc-a0ee2-predictor-7f8d8c965f-kj7sw"
Aug 05 17:26:11.584334 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:26:11.584308    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gvv8l\" (UniqueName: \"kubernetes.io/projected/108727d9-7bf4-4042-9a0f-8b07a5a12f91-kube-api-access-gvv8l\") pod \"error-404-isvc-a0ee2-predictor-7f8d8c965f-kj7sw\" (UID: \"108727d9-7bf4-4042-9a0f-8b07a5a12f91\") " pod="kserve-ci-e2e-test/error-404-isvc-a0ee2-predictor-7f8d8c965f-kj7sw"
Aug 05 17:26:11.731494 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:26:11.731462    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-a0ee2-predictor-7f8d8c965f-kj7sw"
Aug 05 17:26:11.857049 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:26:11.857021    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-a0ee2-predictor-7f8d8c965f-kj7sw"]
Aug 05 17:26:11.859424 ip-10-0-135-2 kubenswrapper[2571]: W0805 17:26:11.859394    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod108727d9_7bf4_4042_9a0f_8b07a5a12f91.slice/crio-73563aa6b80a115aa05b5a5b80c98e67996572ade320ac6cb8b6cc48f8630648 WatchSource:0}: Error finding container 73563aa6b80a115aa05b5a5b80c98e67996572ade320ac6cb8b6cc48f8630648: Status 404 returned error can't find the container with id 73563aa6b80a115aa05b5a5b80c98e67996572ade320ac6cb8b6cc48f8630648
Aug 05 17:26:11.979064 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:26:11.979039    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/cf5fb621-292d-454c-88d6-e66edcd9dfb5-proxy-tls\") pod \"success-200-isvc-a0ee2-predictor-69b645cc54-6zzvp\" (UID: \"cf5fb621-292d-454c-88d6-e66edcd9dfb5\") " pod="kserve-ci-e2e-test/success-200-isvc-a0ee2-predictor-69b645cc54-6zzvp"
Aug 05 17:26:11.981054 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:26:11.981037    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/cf5fb621-292d-454c-88d6-e66edcd9dfb5-proxy-tls\") pod \"success-200-isvc-a0ee2-predictor-69b645cc54-6zzvp\" (UID: \"cf5fb621-292d-454c-88d6-e66edcd9dfb5\") " pod="kserve-ci-e2e-test/success-200-isvc-a0ee2-predictor-69b645cc54-6zzvp"
Aug 05 17:26:12.013711 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:26:12.013680    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-829a9-predictor-67f4589d75-vnfjn" podUID="419c1eaa-2d46-4226-94b6-9d1bbe38e8f0" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.29:8643/healthz\": dial tcp 10.134.0.29:8643: connect: connection refused"
Aug 05 17:26:12.013784 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:26:12.013687    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-829a9-predictor-54f97d665d-jlvfn" podUID="54086e09-ff4e-4168-94f9-58df784d7811" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.28:8643/healthz\": dial tcp 10.134.0.28:8643: connect: connection refused"
Aug 05 17:26:12.217191 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:26:12.217121    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-a0ee2-predictor-69b645cc54-6zzvp"
Aug 05 17:26:12.314091 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:26:12.314060    2571 generic.go:356] "Generic (PLEG): container finished" podID="419c1eaa-2d46-4226-94b6-9d1bbe38e8f0" containerID="07edd901daf5fbe346b6d5104d7e913d0042ea05d62b9b932f28574db58a0095" exitCode=2
Aug 05 17:26:12.314495 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:26:12.314109    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-829a9-predictor-67f4589d75-vnfjn" event={"ID":"419c1eaa-2d46-4226-94b6-9d1bbe38e8f0","Type":"ContainerDied","Data":"07edd901daf5fbe346b6d5104d7e913d0042ea05d62b9b932f28574db58a0095"}
Aug 05 17:26:12.316181 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:26:12.316158    2571 generic.go:356] "Generic (PLEG): container finished" podID="54086e09-ff4e-4168-94f9-58df784d7811" containerID="e600b2dc50291f6d6b7c9d2f3fcee84a864341d1cb20db3555a82c8f17c7d4e5" exitCode=2
Aug 05 17:26:12.316288 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:26:12.316224    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-829a9-predictor-54f97d665d-jlvfn" event={"ID":"54086e09-ff4e-4168-94f9-58df784d7811","Type":"ContainerDied","Data":"e600b2dc50291f6d6b7c9d2f3fcee84a864341d1cb20db3555a82c8f17c7d4e5"}
Aug 05 17:26:12.318175 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:26:12.318152    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-a0ee2-predictor-7f8d8c965f-kj7sw" event={"ID":"108727d9-7bf4-4042-9a0f-8b07a5a12f91","Type":"ContainerStarted","Data":"039d99edd06cbc2525a5d5cb18022a9f46d5bf67aec4003dcda6d164ed6fee5c"}
Aug 05 17:26:12.318274 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:26:12.318203    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-a0ee2-predictor-7f8d8c965f-kj7sw" event={"ID":"108727d9-7bf4-4042-9a0f-8b07a5a12f91","Type":"ContainerStarted","Data":"d38aed0ea994ddf62b53cbf2956dca2d006197c2df217c002047e7b081c68613"}
Aug 05 17:26:12.318274 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:26:12.318235    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-a0ee2-predictor-7f8d8c965f-kj7sw" event={"ID":"108727d9-7bf4-4042-9a0f-8b07a5a12f91","Type":"ContainerStarted","Data":"73563aa6b80a115aa05b5a5b80c98e67996572ade320ac6cb8b6cc48f8630648"}
Aug 05 17:26:12.318274 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:26:12.318266    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-a0ee2-predictor-7f8d8c965f-kj7sw"
Aug 05 17:26:12.318415 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:26:12.318294    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-a0ee2-predictor-7f8d8c965f-kj7sw"
Aug 05 17:26:12.319394 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:26:12.319351    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-a0ee2-predictor-7f8d8c965f-kj7sw" podUID="108727d9-7bf4-4042-9a0f-8b07a5a12f91" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:8080: connect: connection refused"
Aug 05 17:26:12.338249 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:26:12.338207    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/error-404-isvc-a0ee2-predictor-7f8d8c965f-kj7sw" podStartSLOduration=1.3381945929999999 podStartE2EDuration="1.338194593s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-05 17:26:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-05 17:26:12.336480587 +0000 UTC m=+799.703846933" watchObservedRunningTime="2026-08-05 17:26:12.338194593 +0000 UTC m=+799.705561005"
Aug 05 17:26:12.352950 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:26:12.352929    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-a0ee2-predictor-69b645cc54-6zzvp"]
Aug 05 17:26:12.354782 ip-10-0-135-2 kubenswrapper[2571]: W0805 17:26:12.354758    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcf5fb621_292d_454c_88d6_e66edcd9dfb5.slice/crio-868fe003eaf7e2a544236d7c7e9b643b118b49a4051213408b5d449b1dcd9bcc WatchSource:0}: Error finding container 868fe003eaf7e2a544236d7c7e9b643b118b49a4051213408b5d449b1dcd9bcc: Status 404 returned error can't find the container with id 868fe003eaf7e2a544236d7c7e9b643b118b49a4051213408b5d449b1dcd9bcc
Aug 05 17:26:13.323761 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:26:13.323725    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-a0ee2-predictor-69b645cc54-6zzvp" event={"ID":"cf5fb621-292d-454c-88d6-e66edcd9dfb5","Type":"ContainerStarted","Data":"7e2dbcef41e03cee3ac11588a096d677760fa135377e473abdcead871d635512"}
Aug 05 17:26:13.324182 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:26:13.323789    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-a0ee2-predictor-69b645cc54-6zzvp" event={"ID":"cf5fb621-292d-454c-88d6-e66edcd9dfb5","Type":"ContainerStarted","Data":"bbf9e44a35b24a552fa29c1a62006ba0185d457ce4a7df161062333038edb0aa"}
Aug 05 17:26:13.324182 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:26:13.323822    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-a0ee2-predictor-69b645cc54-6zzvp" event={"ID":"cf5fb621-292d-454c-88d6-e66edcd9dfb5","Type":"ContainerStarted","Data":"868fe003eaf7e2a544236d7c7e9b643b118b49a4051213408b5d449b1dcd9bcc"}
Aug 05 17:26:13.324182 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:26:13.323968    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-a0ee2-predictor-69b645cc54-6zzvp"
Aug 05 17:26:13.324182 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:26:13.323997    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-a0ee2-predictor-69b645cc54-6zzvp"
Aug 05 17:26:13.324182 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:26:13.324128    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-a0ee2-predictor-7f8d8c965f-kj7sw" podUID="108727d9-7bf4-4042-9a0f-8b07a5a12f91" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:8080: connect: connection refused"
Aug 05 17:26:13.325197 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:26:13.325170    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-a0ee2-predictor-69b645cc54-6zzvp" podUID="cf5fb621-292d-454c-88d6-e66edcd9dfb5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.30:8080: connect: connection refused"
Aug 05 17:26:13.346663 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:26:13.346621    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/success-200-isvc-a0ee2-predictor-69b645cc54-6zzvp" podStartSLOduration=2.346607031 podStartE2EDuration="2.346607031s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-05 17:26:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-05 17:26:13.345145354 +0000 UTC m=+800.712511698" watchObservedRunningTime="2026-08-05 17:26:13.346607031 +0000 UTC m=+800.713973374"
Aug 05 17:26:14.330583 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:26:14.330552    2571 generic.go:356] "Generic (PLEG): container finished" podID="54086e09-ff4e-4168-94f9-58df784d7811" containerID="30c806bcd7bbb7d6fbb627475a4c798e795653c27c359b128cf83f708cf8d645" exitCode=0
Aug 05 17:26:14.330935 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:26:14.330627    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-829a9-predictor-54f97d665d-jlvfn" event={"ID":"54086e09-ff4e-4168-94f9-58df784d7811","Type":"ContainerDied","Data":"30c806bcd7bbb7d6fbb627475a4c798e795653c27c359b128cf83f708cf8d645"}
Aug 05 17:26:14.330994 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:26:14.330973    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-a0ee2-predictor-69b645cc54-6zzvp" podUID="cf5fb621-292d-454c-88d6-e66edcd9dfb5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.30:8080: connect: connection refused"
Aug 05 17:26:14.387233 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:26:14.387207    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-829a9-predictor-54f97d665d-jlvfn"
Aug 05 17:26:14.499905 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:26:14.499881    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xkpvq\" (UniqueName: \"kubernetes.io/projected/54086e09-ff4e-4168-94f9-58df784d7811-kube-api-access-xkpvq\") pod \"54086e09-ff4e-4168-94f9-58df784d7811\" (UID: \"54086e09-ff4e-4168-94f9-58df784d7811\") "
Aug 05 17:26:14.500055 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:26:14.499967    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/54086e09-ff4e-4168-94f9-58df784d7811-proxy-tls\") pod \"54086e09-ff4e-4168-94f9-58df784d7811\" (UID: \"54086e09-ff4e-4168-94f9-58df784d7811\") "
Aug 05 17:26:14.500097 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:26:14.500068    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/54086e09-ff4e-4168-94f9-58df784d7811-kube-rbac-proxy-sar-config\") pod \"54086e09-ff4e-4168-94f9-58df784d7811\" (UID: \"54086e09-ff4e-4168-94f9-58df784d7811\") "
Aug 05 17:26:14.500425 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:26:14.500398    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/54086e09-ff4e-4168-94f9-58df784d7811-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "54086e09-ff4e-4168-94f9-58df784d7811" (UID: "54086e09-ff4e-4168-94f9-58df784d7811"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 05 17:26:14.502023 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:26:14.502000    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/54086e09-ff4e-4168-94f9-58df784d7811-kube-api-access-xkpvq" (OuterVolumeSpecName: "kube-api-access-xkpvq") pod "54086e09-ff4e-4168-94f9-58df784d7811" (UID: "54086e09-ff4e-4168-94f9-58df784d7811"). InnerVolumeSpecName "kube-api-access-xkpvq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 05 17:26:14.502023 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:26:14.502011    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/54086e09-ff4e-4168-94f9-58df784d7811-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "54086e09-ff4e-4168-94f9-58df784d7811" (UID: "54086e09-ff4e-4168-94f9-58df784d7811"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 05 17:26:14.601266 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:26:14.601229    2571 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/54086e09-ff4e-4168-94f9-58df784d7811-proxy-tls\") on node \"ip-10-0-135-2.ec2.internal\" DevicePath \"\""
Aug 05 17:26:14.601266 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:26:14.601263    2571 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/54086e09-ff4e-4168-94f9-58df784d7811-kube-rbac-proxy-sar-config\") on node \"ip-10-0-135-2.ec2.internal\" DevicePath \"\""
Aug 05 17:26:14.601266 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:26:14.601275    2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xkpvq\" (UniqueName: \"kubernetes.io/projected/54086e09-ff4e-4168-94f9-58df784d7811-kube-api-access-xkpvq\") on node \"ip-10-0-135-2.ec2.internal\" DevicePath \"\""
Aug 05 17:26:14.950638 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:26:14.950613    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-829a9-predictor-67f4589d75-vnfjn"
Aug 05 17:26:15.006266 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:26:15.006230    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/419c1eaa-2d46-4226-94b6-9d1bbe38e8f0-kube-rbac-proxy-sar-config\") pod \"419c1eaa-2d46-4226-94b6-9d1bbe38e8f0\" (UID: \"419c1eaa-2d46-4226-94b6-9d1bbe38e8f0\") "
Aug 05 17:26:15.006426 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:26:15.006279    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5b56t\" (UniqueName: \"kubernetes.io/projected/419c1eaa-2d46-4226-94b6-9d1bbe38e8f0-kube-api-access-5b56t\") pod \"419c1eaa-2d46-4226-94b6-9d1bbe38e8f0\" (UID: \"419c1eaa-2d46-4226-94b6-9d1bbe38e8f0\") "
Aug 05 17:26:15.006426 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:26:15.006356    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/419c1eaa-2d46-4226-94b6-9d1bbe38e8f0-proxy-tls\") pod \"419c1eaa-2d46-4226-94b6-9d1bbe38e8f0\" (UID: \"419c1eaa-2d46-4226-94b6-9d1bbe38e8f0\") "
Aug 05 17:26:15.006599 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:26:15.006573    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/419c1eaa-2d46-4226-94b6-9d1bbe38e8f0-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "419c1eaa-2d46-4226-94b6-9d1bbe38e8f0" (UID: "419c1eaa-2d46-4226-94b6-9d1bbe38e8f0"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 05 17:26:15.008253 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:26:15.008229    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/419c1eaa-2d46-4226-94b6-9d1bbe38e8f0-kube-api-access-5b56t" (OuterVolumeSpecName: "kube-api-access-5b56t") pod "419c1eaa-2d46-4226-94b6-9d1bbe38e8f0" (UID: "419c1eaa-2d46-4226-94b6-9d1bbe38e8f0"). InnerVolumeSpecName "kube-api-access-5b56t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 05 17:26:15.008340 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:26:15.008264    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/419c1eaa-2d46-4226-94b6-9d1bbe38e8f0-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "419c1eaa-2d46-4226-94b6-9d1bbe38e8f0" (UID: "419c1eaa-2d46-4226-94b6-9d1bbe38e8f0"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 05 17:26:15.107515 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:26:15.107409    2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5b56t\" (UniqueName: \"kubernetes.io/projected/419c1eaa-2d46-4226-94b6-9d1bbe38e8f0-kube-api-access-5b56t\") on node \"ip-10-0-135-2.ec2.internal\" DevicePath \"\""
Aug 05 17:26:15.107515 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:26:15.107440    2571 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/419c1eaa-2d46-4226-94b6-9d1bbe38e8f0-proxy-tls\") on node \"ip-10-0-135-2.ec2.internal\" DevicePath \"\""
Aug 05 17:26:15.107515 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:26:15.107454    2571 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/419c1eaa-2d46-4226-94b6-9d1bbe38e8f0-kube-rbac-proxy-sar-config\") on node \"ip-10-0-135-2.ec2.internal\" DevicePath \"\""
Aug 05 17:26:15.335904 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:26:15.335868    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-829a9-predictor-54f97d665d-jlvfn"
Aug 05 17:26:15.335904 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:26:15.335893    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-829a9-predictor-54f97d665d-jlvfn" event={"ID":"54086e09-ff4e-4168-94f9-58df784d7811","Type":"ContainerDied","Data":"7b56f71fc2dd75096b1b1e504b610b353feb8a0f2b574227c36b8255d2532bac"}
Aug 05 17:26:15.336325 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:26:15.335967    2571 scope.go:117] "RemoveContainer" containerID="e600b2dc50291f6d6b7c9d2f3fcee84a864341d1cb20db3555a82c8f17c7d4e5"
Aug 05 17:26:15.337757 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:26:15.337675    2571 generic.go:356] "Generic (PLEG): container finished" podID="419c1eaa-2d46-4226-94b6-9d1bbe38e8f0" containerID="4d04358342b0668d72c552d3ae9b29cecd93a82b63e4e89540749c066466c62b" exitCode=0
Aug 05 17:26:15.337757 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:26:15.337705    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-829a9-predictor-67f4589d75-vnfjn" event={"ID":"419c1eaa-2d46-4226-94b6-9d1bbe38e8f0","Type":"ContainerDied","Data":"4d04358342b0668d72c552d3ae9b29cecd93a82b63e4e89540749c066466c62b"}
Aug 05 17:26:15.337757 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:26:15.337750    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-829a9-predictor-67f4589d75-vnfjn" event={"ID":"419c1eaa-2d46-4226-94b6-9d1bbe38e8f0","Type":"ContainerDied","Data":"a6303051f0f7c29622013caab4771e8352900c4c089359da5b3940e0eeb428cd"}
Aug 05 17:26:15.337757 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:26:15.337753    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-829a9-predictor-67f4589d75-vnfjn"
Aug 05 17:26:15.344347 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:26:15.344322    2571 scope.go:117] "RemoveContainer" containerID="30c806bcd7bbb7d6fbb627475a4c798e795653c27c359b128cf83f708cf8d645"
Aug 05 17:26:15.352170 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:26:15.352154    2571 scope.go:117] "RemoveContainer" containerID="07edd901daf5fbe346b6d5104d7e913d0042ea05d62b9b932f28574db58a0095"
Aug 05 17:26:15.358463 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:26:15.358417    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-829a9-predictor-54f97d665d-jlvfn"]
Aug 05 17:26:15.359737 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:26:15.359720    2571 scope.go:117] "RemoveContainer" containerID="4d04358342b0668d72c552d3ae9b29cecd93a82b63e4e89540749c066466c62b"
Aug 05 17:26:15.361469 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:26:15.361451    2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-829a9-predictor-54f97d665d-jlvfn"]
Aug 05 17:26:15.366873 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:26:15.366853    2571 scope.go:117] "RemoveContainer" containerID="07edd901daf5fbe346b6d5104d7e913d0042ea05d62b9b932f28574db58a0095"
Aug 05 17:26:15.367119 ip-10-0-135-2 kubenswrapper[2571]: E0805 17:26:15.367099    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"07edd901daf5fbe346b6d5104d7e913d0042ea05d62b9b932f28574db58a0095\": container with ID starting with 07edd901daf5fbe346b6d5104d7e913d0042ea05d62b9b932f28574db58a0095 not found: ID does not exist" containerID="07edd901daf5fbe346b6d5104d7e913d0042ea05d62b9b932f28574db58a0095"
Aug 05 17:26:15.367162 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:26:15.367127    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"07edd901daf5fbe346b6d5104d7e913d0042ea05d62b9b932f28574db58a0095"} err="failed to get container status \"07edd901daf5fbe346b6d5104d7e913d0042ea05d62b9b932f28574db58a0095\": rpc error: code = NotFound desc = could not find container \"07edd901daf5fbe346b6d5104d7e913d0042ea05d62b9b932f28574db58a0095\": container with ID starting with 07edd901daf5fbe346b6d5104d7e913d0042ea05d62b9b932f28574db58a0095 not found: ID does not exist"
Aug 05 17:26:15.367162 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:26:15.367144    2571 scope.go:117] "RemoveContainer" containerID="4d04358342b0668d72c552d3ae9b29cecd93a82b63e4e89540749c066466c62b"
Aug 05 17:26:15.367406 ip-10-0-135-2 kubenswrapper[2571]: E0805 17:26:15.367389    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4d04358342b0668d72c552d3ae9b29cecd93a82b63e4e89540749c066466c62b\": container with ID starting with 4d04358342b0668d72c552d3ae9b29cecd93a82b63e4e89540749c066466c62b not found: ID does not exist" containerID="4d04358342b0668d72c552d3ae9b29cecd93a82b63e4e89540749c066466c62b"
Aug 05 17:26:15.367461 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:26:15.367411    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4d04358342b0668d72c552d3ae9b29cecd93a82b63e4e89540749c066466c62b"} err="failed to get container status \"4d04358342b0668d72c552d3ae9b29cecd93a82b63e4e89540749c066466c62b\": rpc error: code = NotFound desc = could not find container \"4d04358342b0668d72c552d3ae9b29cecd93a82b63e4e89540749c066466c62b\": container with ID starting with 4d04358342b0668d72c552d3ae9b29cecd93a82b63e4e89540749c066466c62b not found: ID does not exist"
Aug 05 17:26:15.372781 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:26:15.372759    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-829a9-predictor-67f4589d75-vnfjn"]
Aug 05 17:26:15.376878 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:26:15.376860    2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-829a9-predictor-67f4589d75-vnfjn"]
Aug 05 17:26:17.136317 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:26:17.136286    2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="419c1eaa-2d46-4226-94b6-9d1bbe38e8f0" path="/var/lib/kubelet/pods/419c1eaa-2d46-4226-94b6-9d1bbe38e8f0/volumes"
Aug 05 17:26:17.136868 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:26:17.136851    2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="54086e09-ff4e-4168-94f9-58df784d7811" path="/var/lib/kubelet/pods/54086e09-ff4e-4168-94f9-58df784d7811/volumes"
Aug 05 17:26:18.328657 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:26:18.328629    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-a0ee2-predictor-7f8d8c965f-kj7sw"
Aug 05 17:26:18.329229 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:26:18.329203    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-a0ee2-predictor-7f8d8c965f-kj7sw" podUID="108727d9-7bf4-4042-9a0f-8b07a5a12f91" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:8080: connect: connection refused"
Aug 05 17:26:19.336567 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:26:19.336536    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-a0ee2-predictor-69b645cc54-6zzvp"
Aug 05 17:26:19.337006 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:26:19.336955    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-a0ee2-predictor-69b645cc54-6zzvp" podUID="cf5fb621-292d-454c-88d6-e66edcd9dfb5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.30:8080: connect: connection refused"
Aug 05 17:26:28.330194 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:26:28.330154    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-a0ee2-predictor-7f8d8c965f-kj7sw" podUID="108727d9-7bf4-4042-9a0f-8b07a5a12f91" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:8080: connect: connection refused"
Aug 05 17:26:29.337277 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:26:29.337235    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-a0ee2-predictor-69b645cc54-6zzvp" podUID="cf5fb621-292d-454c-88d6-e66edcd9dfb5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.30:8080: connect: connection refused"
Aug 05 17:26:38.329617 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:26:38.329571    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-a0ee2-predictor-7f8d8c965f-kj7sw" podUID="108727d9-7bf4-4042-9a0f-8b07a5a12f91" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:8080: connect: connection refused"
Aug 05 17:26:39.337809 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:26:39.337770    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-a0ee2-predictor-69b645cc54-6zzvp" podUID="cf5fb621-292d-454c-88d6-e66edcd9dfb5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.30:8080: connect: connection refused"
Aug 05 17:26:48.330153 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:26:48.330108    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-a0ee2-predictor-7f8d8c965f-kj7sw" podUID="108727d9-7bf4-4042-9a0f-8b07a5a12f91" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:8080: connect: connection refused"
Aug 05 17:26:49.337198 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:26:49.337160    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-a0ee2-predictor-69b645cc54-6zzvp" podUID="cf5fb621-292d-454c-88d6-e66edcd9dfb5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.30:8080: connect: connection refused"
Aug 05 17:26:58.330254 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:26:58.330210    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-a0ee2-predictor-7f8d8c965f-kj7sw"
Aug 05 17:26:59.338049 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:26:59.338018    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-a0ee2-predictor-69b645cc54-6zzvp"
Aug 05 17:27:11.249736 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:27:11.249705    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-d9755-predictor-d8c999f56-sg8lf"]
Aug 05 17:27:11.250083 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:27:11.250026    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="419c1eaa-2d46-4226-94b6-9d1bbe38e8f0" containerName="kube-rbac-proxy"
Aug 05 17:27:11.250083 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:27:11.250036    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="419c1eaa-2d46-4226-94b6-9d1bbe38e8f0" containerName="kube-rbac-proxy"
Aug 05 17:27:11.250083 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:27:11.250044    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="54086e09-ff4e-4168-94f9-58df784d7811" containerName="kube-rbac-proxy"
Aug 05 17:27:11.250083 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:27:11.250050    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="54086e09-ff4e-4168-94f9-58df784d7811" containerName="kube-rbac-proxy"
Aug 05 17:27:11.250083 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:27:11.250061    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="54086e09-ff4e-4168-94f9-58df784d7811" containerName="kserve-container"
Aug 05 17:27:11.250083 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:27:11.250068    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="54086e09-ff4e-4168-94f9-58df784d7811" containerName="kserve-container"
Aug 05 17:27:11.250083 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:27:11.250073    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="419c1eaa-2d46-4226-94b6-9d1bbe38e8f0" containerName="kserve-container"
Aug 05 17:27:11.250083 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:27:11.250078    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="419c1eaa-2d46-4226-94b6-9d1bbe38e8f0" containerName="kserve-container"
Aug 05 17:27:11.250342 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:27:11.250139    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="419c1eaa-2d46-4226-94b6-9d1bbe38e8f0" containerName="kube-rbac-proxy"
Aug 05 17:27:11.250342 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:27:11.250148    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="54086e09-ff4e-4168-94f9-58df784d7811" containerName="kserve-container"
Aug 05 17:27:11.250342 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:27:11.250153    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="419c1eaa-2d46-4226-94b6-9d1bbe38e8f0" containerName="kserve-container"
Aug 05 17:27:11.250342 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:27:11.250161    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="54086e09-ff4e-4168-94f9-58df784d7811" containerName="kube-rbac-proxy"
Aug 05 17:27:11.254464 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:27:11.254447    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-d9755-predictor-d8c999f56-sg8lf"
Aug 05 17:27:11.258880 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:27:11.258856    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-d9755-predictor-serving-cert\""
Aug 05 17:27:11.259016 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:27:11.258860    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-d9755-kube-rbac-proxy-sar-config\""
Aug 05 17:27:11.271313 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:27:11.269495    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-d9755-predictor-d8c999f56-sg8lf"]
Aug 05 17:27:11.347181 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:27:11.347151    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/37d7cb85-d235-490e-80dc-9e73cc8cfa29-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-d9755-predictor-d8c999f56-sg8lf\" (UID: \"37d7cb85-d235-490e-80dc-9e73cc8cfa29\") " pod="kserve-ci-e2e-test/success-200-isvc-d9755-predictor-d8c999f56-sg8lf"
Aug 05 17:27:11.347329 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:27:11.347199    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gwl9l\" (UniqueName: \"kubernetes.io/projected/37d7cb85-d235-490e-80dc-9e73cc8cfa29-kube-api-access-gwl9l\") pod \"success-200-isvc-d9755-predictor-d8c999f56-sg8lf\" (UID: \"37d7cb85-d235-490e-80dc-9e73cc8cfa29\") " pod="kserve-ci-e2e-test/success-200-isvc-d9755-predictor-d8c999f56-sg8lf"
Aug 05 17:27:11.347329 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:27:11.347254    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/37d7cb85-d235-490e-80dc-9e73cc8cfa29-proxy-tls\") pod \"success-200-isvc-d9755-predictor-d8c999f56-sg8lf\" (UID: \"37d7cb85-d235-490e-80dc-9e73cc8cfa29\") " pod="kserve-ci-e2e-test/success-200-isvc-d9755-predictor-d8c999f56-sg8lf"
Aug 05 17:27:11.380186 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:27:11.380149    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-d9755-predictor-78b5549d57-wh2st"]
Aug 05 17:27:11.383579 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:27:11.383558    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-d9755-predictor-78b5549d57-wh2st"
Aug 05 17:27:11.385989 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:27:11.385963    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-d9755-predictor-serving-cert\""
Aug 05 17:27:11.386363 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:27:11.386344    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-d9755-kube-rbac-proxy-sar-config\""
Aug 05 17:27:11.393965 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:27:11.393946    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-d9755-predictor-78b5549d57-wh2st"]
Aug 05 17:27:11.447707 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:27:11.447684    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0fe55532-2a2b-41c1-92d4-dc7f0e201ccc-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-d9755-predictor-78b5549d57-wh2st\" (UID: \"0fe55532-2a2b-41c1-92d4-dc7f0e201ccc\") " pod="kserve-ci-e2e-test/error-404-isvc-d9755-predictor-78b5549d57-wh2st"
Aug 05 17:27:11.447807 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:27:11.447726    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/37d7cb85-d235-490e-80dc-9e73cc8cfa29-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-d9755-predictor-d8c999f56-sg8lf\" (UID: \"37d7cb85-d235-490e-80dc-9e73cc8cfa29\") " pod="kserve-ci-e2e-test/success-200-isvc-d9755-predictor-d8c999f56-sg8lf"
Aug 05 17:27:11.447807 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:27:11.447764    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0fe55532-2a2b-41c1-92d4-dc7f0e201ccc-proxy-tls\") pod \"error-404-isvc-d9755-predictor-78b5549d57-wh2st\" (UID: \"0fe55532-2a2b-41c1-92d4-dc7f0e201ccc\") " pod="kserve-ci-e2e-test/error-404-isvc-d9755-predictor-78b5549d57-wh2st"
Aug 05 17:27:11.447899 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:27:11.447813    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gwl9l\" (UniqueName: \"kubernetes.io/projected/37d7cb85-d235-490e-80dc-9e73cc8cfa29-kube-api-access-gwl9l\") pod \"success-200-isvc-d9755-predictor-d8c999f56-sg8lf\" (UID: \"37d7cb85-d235-490e-80dc-9e73cc8cfa29\") " pod="kserve-ci-e2e-test/success-200-isvc-d9755-predictor-d8c999f56-sg8lf"
Aug 05 17:27:11.447899 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:27:11.447834    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/37d7cb85-d235-490e-80dc-9e73cc8cfa29-proxy-tls\") pod \"success-200-isvc-d9755-predictor-d8c999f56-sg8lf\" (UID: \"37d7cb85-d235-490e-80dc-9e73cc8cfa29\") " pod="kserve-ci-e2e-test/success-200-isvc-d9755-predictor-d8c999f56-sg8lf"
Aug 05 17:27:11.447899 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:27:11.447856    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v2wtg\" (UniqueName: \"kubernetes.io/projected/0fe55532-2a2b-41c1-92d4-dc7f0e201ccc-kube-api-access-v2wtg\") pod \"error-404-isvc-d9755-predictor-78b5549d57-wh2st\" (UID: \"0fe55532-2a2b-41c1-92d4-dc7f0e201ccc\") " pod="kserve-ci-e2e-test/error-404-isvc-d9755-predictor-78b5549d57-wh2st"
Aug 05 17:27:11.448024 ip-10-0-135-2 kubenswrapper[2571]: E0805 17:27:11.447957    2571 secret.go:189] Couldn't get secret kserve-ci-e2e-test/success-200-isvc-d9755-predictor-serving-cert: secret "success-200-isvc-d9755-predictor-serving-cert" not found
Aug 05 17:27:11.448024 ip-10-0-135-2 kubenswrapper[2571]: E0805 17:27:11.448022    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/37d7cb85-d235-490e-80dc-9e73cc8cfa29-proxy-tls podName:37d7cb85-d235-490e-80dc-9e73cc8cfa29 nodeName:}" failed. No retries permitted until 2026-08-05 17:27:11.948007463 +0000 UTC m=+859.315373787 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/37d7cb85-d235-490e-80dc-9e73cc8cfa29-proxy-tls") pod "success-200-isvc-d9755-predictor-d8c999f56-sg8lf" (UID: "37d7cb85-d235-490e-80dc-9e73cc8cfa29") : secret "success-200-isvc-d9755-predictor-serving-cert" not found
Aug 05 17:27:11.448318 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:27:11.448298    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/37d7cb85-d235-490e-80dc-9e73cc8cfa29-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-d9755-predictor-d8c999f56-sg8lf\" (UID: \"37d7cb85-d235-490e-80dc-9e73cc8cfa29\") " pod="kserve-ci-e2e-test/success-200-isvc-d9755-predictor-d8c999f56-sg8lf"
Aug 05 17:27:11.458854 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:27:11.458825    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gwl9l\" (UniqueName: \"kubernetes.io/projected/37d7cb85-d235-490e-80dc-9e73cc8cfa29-kube-api-access-gwl9l\") pod \"success-200-isvc-d9755-predictor-d8c999f56-sg8lf\" (UID: \"37d7cb85-d235-490e-80dc-9e73cc8cfa29\") " pod="kserve-ci-e2e-test/success-200-isvc-d9755-predictor-d8c999f56-sg8lf"
Aug 05 17:27:11.548530 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:27:11.548475    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v2wtg\" (UniqueName: \"kubernetes.io/projected/0fe55532-2a2b-41c1-92d4-dc7f0e201ccc-kube-api-access-v2wtg\") pod \"error-404-isvc-d9755-predictor-78b5549d57-wh2st\" (UID: \"0fe55532-2a2b-41c1-92d4-dc7f0e201ccc\") " pod="kserve-ci-e2e-test/error-404-isvc-d9755-predictor-78b5549d57-wh2st"
Aug 05 17:27:11.548530 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:27:11.548529    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0fe55532-2a2b-41c1-92d4-dc7f0e201ccc-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-d9755-predictor-78b5549d57-wh2st\" (UID: \"0fe55532-2a2b-41c1-92d4-dc7f0e201ccc\") " pod="kserve-ci-e2e-test/error-404-isvc-d9755-predictor-78b5549d57-wh2st"
Aug 05 17:27:11.548668 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:27:11.548559    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0fe55532-2a2b-41c1-92d4-dc7f0e201ccc-proxy-tls\") pod \"error-404-isvc-d9755-predictor-78b5549d57-wh2st\" (UID: \"0fe55532-2a2b-41c1-92d4-dc7f0e201ccc\") " pod="kserve-ci-e2e-test/error-404-isvc-d9755-predictor-78b5549d57-wh2st"
Aug 05 17:27:11.548668 ip-10-0-135-2 kubenswrapper[2571]: E0805 17:27:11.548657    2571 secret.go:189] Couldn't get secret kserve-ci-e2e-test/error-404-isvc-d9755-predictor-serving-cert: secret "error-404-isvc-d9755-predictor-serving-cert" not found
Aug 05 17:27:11.548754 ip-10-0-135-2 kubenswrapper[2571]: E0805 17:27:11.548703    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0fe55532-2a2b-41c1-92d4-dc7f0e201ccc-proxy-tls podName:0fe55532-2a2b-41c1-92d4-dc7f0e201ccc nodeName:}" failed. No retries permitted until 2026-08-05 17:27:12.048689238 +0000 UTC m=+859.416055559 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/0fe55532-2a2b-41c1-92d4-dc7f0e201ccc-proxy-tls") pod "error-404-isvc-d9755-predictor-78b5549d57-wh2st" (UID: "0fe55532-2a2b-41c1-92d4-dc7f0e201ccc") : secret "error-404-isvc-d9755-predictor-serving-cert" not found
Aug 05 17:27:11.549130 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:27:11.549113    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0fe55532-2a2b-41c1-92d4-dc7f0e201ccc-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-d9755-predictor-78b5549d57-wh2st\" (UID: \"0fe55532-2a2b-41c1-92d4-dc7f0e201ccc\") " pod="kserve-ci-e2e-test/error-404-isvc-d9755-predictor-78b5549d57-wh2st"
Aug 05 17:27:11.556983 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:27:11.556963    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v2wtg\" (UniqueName: \"kubernetes.io/projected/0fe55532-2a2b-41c1-92d4-dc7f0e201ccc-kube-api-access-v2wtg\") pod \"error-404-isvc-d9755-predictor-78b5549d57-wh2st\" (UID: \"0fe55532-2a2b-41c1-92d4-dc7f0e201ccc\") " pod="kserve-ci-e2e-test/error-404-isvc-d9755-predictor-78b5549d57-wh2st"
Aug 05 17:27:11.952568 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:27:11.952503    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/37d7cb85-d235-490e-80dc-9e73cc8cfa29-proxy-tls\") pod \"success-200-isvc-d9755-predictor-d8c999f56-sg8lf\" (UID: \"37d7cb85-d235-490e-80dc-9e73cc8cfa29\") " pod="kserve-ci-e2e-test/success-200-isvc-d9755-predictor-d8c999f56-sg8lf"
Aug 05 17:27:11.952687 ip-10-0-135-2 kubenswrapper[2571]: E0805 17:27:11.952619    2571 secret.go:189] Couldn't get secret kserve-ci-e2e-test/success-200-isvc-d9755-predictor-serving-cert: secret "success-200-isvc-d9755-predictor-serving-cert" not found
Aug 05 17:27:11.952687 ip-10-0-135-2 kubenswrapper[2571]: E0805 17:27:11.952665    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/37d7cb85-d235-490e-80dc-9e73cc8cfa29-proxy-tls podName:37d7cb85-d235-490e-80dc-9e73cc8cfa29 nodeName:}" failed. No retries permitted until 2026-08-05 17:27:12.952652569 +0000 UTC m=+860.320018889 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/37d7cb85-d235-490e-80dc-9e73cc8cfa29-proxy-tls") pod "success-200-isvc-d9755-predictor-d8c999f56-sg8lf" (UID: "37d7cb85-d235-490e-80dc-9e73cc8cfa29") : secret "success-200-isvc-d9755-predictor-serving-cert" not found
Aug 05 17:27:12.053170 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:27:12.053141    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0fe55532-2a2b-41c1-92d4-dc7f0e201ccc-proxy-tls\") pod \"error-404-isvc-d9755-predictor-78b5549d57-wh2st\" (UID: \"0fe55532-2a2b-41c1-92d4-dc7f0e201ccc\") " pod="kserve-ci-e2e-test/error-404-isvc-d9755-predictor-78b5549d57-wh2st"
Aug 05 17:27:12.055497 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:27:12.055476    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0fe55532-2a2b-41c1-92d4-dc7f0e201ccc-proxy-tls\") pod \"error-404-isvc-d9755-predictor-78b5549d57-wh2st\" (UID: \"0fe55532-2a2b-41c1-92d4-dc7f0e201ccc\") " pod="kserve-ci-e2e-test/error-404-isvc-d9755-predictor-78b5549d57-wh2st"
Aug 05 17:27:12.294320 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:27:12.294292    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-d9755-predictor-78b5549d57-wh2st"
Aug 05 17:27:12.424462 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:27:12.424437    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-d9755-predictor-78b5549d57-wh2st"]
Aug 05 17:27:12.426779 ip-10-0-135-2 kubenswrapper[2571]: W0805 17:27:12.426755    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0fe55532_2a2b_41c1_92d4_dc7f0e201ccc.slice/crio-bb1f769057dec9aab1d63b5acce65178ce9fbb94f7243f2376db548bf4b0505a WatchSource:0}: Error finding container bb1f769057dec9aab1d63b5acce65178ce9fbb94f7243f2376db548bf4b0505a: Status 404 returned error can't find the container with id bb1f769057dec9aab1d63b5acce65178ce9fbb94f7243f2376db548bf4b0505a
Aug 05 17:27:12.590125 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:27:12.590096    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-d9755-predictor-78b5549d57-wh2st" event={"ID":"0fe55532-2a2b-41c1-92d4-dc7f0e201ccc","Type":"ContainerStarted","Data":"f10060501d672f4cbd8ae351765abce050c0e4643ae67970d41545f3de958e3c"}
Aug 05 17:27:12.590227 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:27:12.590155    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-d9755-predictor-78b5549d57-wh2st" event={"ID":"0fe55532-2a2b-41c1-92d4-dc7f0e201ccc","Type":"ContainerStarted","Data":"29331d6ac473f82798132463a470532d5a7d87b5e0d210b81ffbc82b114172dc"}
Aug 05 17:27:12.590227 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:27:12.590175    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-d9755-predictor-78b5549d57-wh2st" event={"ID":"0fe55532-2a2b-41c1-92d4-dc7f0e201ccc","Type":"ContainerStarted","Data":"bb1f769057dec9aab1d63b5acce65178ce9fbb94f7243f2376db548bf4b0505a"}
Aug 05 17:27:12.590316 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:27:12.590227    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-d9755-predictor-78b5549d57-wh2st"
Aug 05 17:27:12.610709 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:27:12.610665    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/error-404-isvc-d9755-predictor-78b5549d57-wh2st" podStartSLOduration=1.610651531 podStartE2EDuration="1.610651531s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-05 17:27:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-05 17:27:12.608894168 +0000 UTC m=+859.976260524" watchObservedRunningTime="2026-08-05 17:27:12.610651531 +0000 UTC m=+859.978017874"
Aug 05 17:27:12.960680 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:27:12.960598    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/37d7cb85-d235-490e-80dc-9e73cc8cfa29-proxy-tls\") pod \"success-200-isvc-d9755-predictor-d8c999f56-sg8lf\" (UID: \"37d7cb85-d235-490e-80dc-9e73cc8cfa29\") " pod="kserve-ci-e2e-test/success-200-isvc-d9755-predictor-d8c999f56-sg8lf"
Aug 05 17:27:12.962920 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:27:12.962890    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/37d7cb85-d235-490e-80dc-9e73cc8cfa29-proxy-tls\") pod \"success-200-isvc-d9755-predictor-d8c999f56-sg8lf\" (UID: \"37d7cb85-d235-490e-80dc-9e73cc8cfa29\") " pod="kserve-ci-e2e-test/success-200-isvc-d9755-predictor-d8c999f56-sg8lf"
Aug 05 17:27:13.065152 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:27:13.065112    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-d9755-predictor-d8c999f56-sg8lf"
Aug 05 17:27:13.192924 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:27:13.192899    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-d9755-predictor-d8c999f56-sg8lf"]
Aug 05 17:27:13.194943 ip-10-0-135-2 kubenswrapper[2571]: W0805 17:27:13.194919    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod37d7cb85_d235_490e_80dc_9e73cc8cfa29.slice/crio-a735d79c24d914a8bcb26dcf43ebaa1127210f8915e1b44c5f76c49a406f2229 WatchSource:0}: Error finding container a735d79c24d914a8bcb26dcf43ebaa1127210f8915e1b44c5f76c49a406f2229: Status 404 returned error can't find the container with id a735d79c24d914a8bcb26dcf43ebaa1127210f8915e1b44c5f76c49a406f2229
Aug 05 17:27:13.596446 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:27:13.596409    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-d9755-predictor-d8c999f56-sg8lf" event={"ID":"37d7cb85-d235-490e-80dc-9e73cc8cfa29","Type":"ContainerStarted","Data":"eadae4a9c5e1fd7118df87e82da719a3b59c612a20d54fd0ec8e4f678f46c384"}
Aug 05 17:27:13.596873 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:27:13.596568    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-d9755-predictor-d8c999f56-sg8lf"
Aug 05 17:27:13.596873 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:27:13.596620    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-d9755-predictor-d8c999f56-sg8lf" event={"ID":"37d7cb85-d235-490e-80dc-9e73cc8cfa29","Type":"ContainerStarted","Data":"45dcb43937a43b7375beaf5bb4431800e574349b409baa0252e629af9021c9c3"}
Aug 05 17:27:13.596873 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:27:13.596655    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-d9755-predictor-d8c999f56-sg8lf" event={"ID":"37d7cb85-d235-490e-80dc-9e73cc8cfa29","Type":"ContainerStarted","Data":"a735d79c24d914a8bcb26dcf43ebaa1127210f8915e1b44c5f76c49a406f2229"}
Aug 05 17:27:13.597022 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:27:13.596890    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-d9755-predictor-d8c999f56-sg8lf"
Aug 05 17:27:13.597022 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:27:13.596927    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-d9755-predictor-78b5549d57-wh2st"
Aug 05 17:27:13.597921 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:27:13.597893    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-d9755-predictor-78b5549d57-wh2st" podUID="0fe55532-2a2b-41c1-92d4-dc7f0e201ccc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused"
Aug 05 17:27:13.597921 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:27:13.597907    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-d9755-predictor-d8c999f56-sg8lf" podUID="37d7cb85-d235-490e-80dc-9e73cc8cfa29" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:8080: connect: connection refused"
Aug 05 17:27:13.615521 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:27:13.615482    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/success-200-isvc-d9755-predictor-d8c999f56-sg8lf" podStartSLOduration=2.615467179 podStartE2EDuration="2.615467179s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-05 17:27:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-05 17:27:13.613955789 +0000 UTC m=+860.981322134" watchObservedRunningTime="2026-08-05 17:27:13.615467179 +0000 UTC m=+860.982833523"
Aug 05 17:27:14.600943 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:27:14.600901    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-d9755-predictor-d8c999f56-sg8lf" podUID="37d7cb85-d235-490e-80dc-9e73cc8cfa29" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:8080: connect: connection refused"
Aug 05 17:27:14.601367 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:27:14.600983    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-d9755-predictor-78b5549d57-wh2st" podUID="0fe55532-2a2b-41c1-92d4-dc7f0e201ccc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused"
Aug 05 17:27:15.605362 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:27:15.605313    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-d9755-predictor-d8c999f56-sg8lf" podUID="37d7cb85-d235-490e-80dc-9e73cc8cfa29" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:8080: connect: connection refused"
Aug 05 17:27:19.605520 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:27:19.605493    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-d9755-predictor-78b5549d57-wh2st"
Aug 05 17:27:19.605932 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:27:19.605904    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-d9755-predictor-78b5549d57-wh2st" podUID="0fe55532-2a2b-41c1-92d4-dc7f0e201ccc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused"
Aug 05 17:27:20.609867 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:27:20.609838    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-d9755-predictor-d8c999f56-sg8lf"
Aug 05 17:27:20.610276 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:27:20.610251    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-d9755-predictor-d8c999f56-sg8lf" podUID="37d7cb85-d235-490e-80dc-9e73cc8cfa29" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:8080: connect: connection refused"
Aug 05 17:27:29.606945 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:27:29.606900    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-d9755-predictor-78b5549d57-wh2st" podUID="0fe55532-2a2b-41c1-92d4-dc7f0e201ccc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused"
Aug 05 17:27:30.610981 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:27:30.610939    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-d9755-predictor-d8c999f56-sg8lf" podUID="37d7cb85-d235-490e-80dc-9e73cc8cfa29" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:8080: connect: connection refused"
Aug 05 17:27:39.606229 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:27:39.606187    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-d9755-predictor-78b5549d57-wh2st" podUID="0fe55532-2a2b-41c1-92d4-dc7f0e201ccc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused"
Aug 05 17:27:40.610697 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:27:40.610651    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-d9755-predictor-d8c999f56-sg8lf" podUID="37d7cb85-d235-490e-80dc-9e73cc8cfa29" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:8080: connect: connection refused"
Aug 05 17:27:49.605956 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:27:49.605880    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-d9755-predictor-78b5549d57-wh2st" podUID="0fe55532-2a2b-41c1-92d4-dc7f0e201ccc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused"
Aug 05 17:27:50.611102 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:27:50.611057    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-d9755-predictor-d8c999f56-sg8lf" podUID="37d7cb85-d235-490e-80dc-9e73cc8cfa29" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:8080: connect: connection refused"
Aug 05 17:27:59.606892 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:27:59.606860    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-d9755-predictor-78b5549d57-wh2st"
Aug 05 17:28:00.611570 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:28:00.611542    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-d9755-predictor-d8c999f56-sg8lf"
Aug 05 17:35:26.115365 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:35:26.115333    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-a0ee2-predictor-69b645cc54-6zzvp"]
Aug 05 17:35:26.117814 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:35:26.115572    2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-a0ee2-predictor-69b645cc54-6zzvp" podUID="cf5fb621-292d-454c-88d6-e66edcd9dfb5" containerName="kserve-container" containerID="cri-o://bbf9e44a35b24a552fa29c1a62006ba0185d457ce4a7df161062333038edb0aa" gracePeriod=30
Aug 05 17:35:26.117814 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:35:26.115606    2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-a0ee2-predictor-69b645cc54-6zzvp" podUID="cf5fb621-292d-454c-88d6-e66edcd9dfb5" containerName="kube-rbac-proxy" containerID="cri-o://7e2dbcef41e03cee3ac11588a096d677760fa135377e473abdcead871d635512" gracePeriod=30
Aug 05 17:35:26.204039 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:35:26.204009    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-fba55-predictor-77c4dfc47d-j8skd"]
Aug 05 17:35:26.207332 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:35:26.207317    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-fba55-predictor-77c4dfc47d-j8skd"
Aug 05 17:35:26.209844 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:35:26.209825    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-fba55-predictor-serving-cert\""
Aug 05 17:35:26.210102 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:35:26.210086    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-fba55-kube-rbac-proxy-sar-config\""
Aug 05 17:35:26.219670 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:35:26.219649    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-fba55-predictor-77c4dfc47d-j8skd"]
Aug 05 17:35:26.275443 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:35:26.275412    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-a0ee2-predictor-7f8d8c965f-kj7sw"]
Aug 05 17:35:26.275668 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:35:26.275636    2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-a0ee2-predictor-7f8d8c965f-kj7sw" podUID="108727d9-7bf4-4042-9a0f-8b07a5a12f91" containerName="kserve-container" containerID="cri-o://d38aed0ea994ddf62b53cbf2956dca2d006197c2df217c002047e7b081c68613" gracePeriod=30
Aug 05 17:35:26.275813 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:35:26.275700    2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-a0ee2-predictor-7f8d8c965f-kj7sw" podUID="108727d9-7bf4-4042-9a0f-8b07a5a12f91" containerName="kube-rbac-proxy" containerID="cri-o://039d99edd06cbc2525a5d5cb18022a9f46d5bf67aec4003dcda6d164ed6fee5c" gracePeriod=30
Aug 05 17:35:26.306500 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:35:26.306466    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a489ce9e-f83f-47f4-8faf-2358a9349294-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-fba55-predictor-77c4dfc47d-j8skd\" (UID: \"a489ce9e-f83f-47f4-8faf-2358a9349294\") " pod="kserve-ci-e2e-test/success-200-isvc-fba55-predictor-77c4dfc47d-j8skd"
Aug 05 17:35:26.306646 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:35:26.306618    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ts2wq\" (UniqueName: \"kubernetes.io/projected/a489ce9e-f83f-47f4-8faf-2358a9349294-kube-api-access-ts2wq\") pod \"success-200-isvc-fba55-predictor-77c4dfc47d-j8skd\" (UID: \"a489ce9e-f83f-47f4-8faf-2358a9349294\") " pod="kserve-ci-e2e-test/success-200-isvc-fba55-predictor-77c4dfc47d-j8skd"
Aug 05 17:35:26.306722 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:35:26.306672    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a489ce9e-f83f-47f4-8faf-2358a9349294-proxy-tls\") pod \"success-200-isvc-fba55-predictor-77c4dfc47d-j8skd\" (UID: \"a489ce9e-f83f-47f4-8faf-2358a9349294\") " pod="kserve-ci-e2e-test/success-200-isvc-fba55-predictor-77c4dfc47d-j8skd"
Aug 05 17:35:26.309779 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:35:26.309756    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-fba55-predictor-676475b57f-fljvk"]
Aug 05 17:35:26.313224 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:35:26.313209    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-fba55-predictor-676475b57f-fljvk"
Aug 05 17:35:26.315616 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:35:26.315596    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-fba55-predictor-serving-cert\""
Aug 05 17:35:26.315616 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:35:26.315606    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-fba55-kube-rbac-proxy-sar-config\""
Aug 05 17:35:26.325492 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:35:26.325474    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-fba55-predictor-676475b57f-fljvk"]
Aug 05 17:35:26.407783 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:35:26.407753    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/366121e0-36b3-4704-a224-ee0e872f94e4-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-fba55-predictor-676475b57f-fljvk\" (UID: \"366121e0-36b3-4704-a224-ee0e872f94e4\") " pod="kserve-ci-e2e-test/error-404-isvc-fba55-predictor-676475b57f-fljvk"
Aug 05 17:35:26.407866 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:35:26.407793    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ts2wq\" (UniqueName: \"kubernetes.io/projected/a489ce9e-f83f-47f4-8faf-2358a9349294-kube-api-access-ts2wq\") pod \"success-200-isvc-fba55-predictor-77c4dfc47d-j8skd\" (UID: \"a489ce9e-f83f-47f4-8faf-2358a9349294\") " pod="kserve-ci-e2e-test/success-200-isvc-fba55-predictor-77c4dfc47d-j8skd"
Aug 05 17:35:26.407866 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:35:26.407816    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a489ce9e-f83f-47f4-8faf-2358a9349294-proxy-tls\") pod \"success-200-isvc-fba55-predictor-77c4dfc47d-j8skd\" (UID: \"a489ce9e-f83f-47f4-8faf-2358a9349294\") " pod="kserve-ci-e2e-test/success-200-isvc-fba55-predictor-77c4dfc47d-j8skd"
Aug 05 17:35:26.407939 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:35:26.407882    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a489ce9e-f83f-47f4-8faf-2358a9349294-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-fba55-predictor-77c4dfc47d-j8skd\" (UID: \"a489ce9e-f83f-47f4-8faf-2358a9349294\") " pod="kserve-ci-e2e-test/success-200-isvc-fba55-predictor-77c4dfc47d-j8skd"
Aug 05 17:35:26.407939 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:35:26.407910    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/366121e0-36b3-4704-a224-ee0e872f94e4-proxy-tls\") pod \"error-404-isvc-fba55-predictor-676475b57f-fljvk\" (UID: \"366121e0-36b3-4704-a224-ee0e872f94e4\") " pod="kserve-ci-e2e-test/error-404-isvc-fba55-predictor-676475b57f-fljvk"
Aug 05 17:35:26.408020 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:35:26.407942    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-knb6q\" (UniqueName: \"kubernetes.io/projected/366121e0-36b3-4704-a224-ee0e872f94e4-kube-api-access-knb6q\") pod \"error-404-isvc-fba55-predictor-676475b57f-fljvk\" (UID: \"366121e0-36b3-4704-a224-ee0e872f94e4\") " pod="kserve-ci-e2e-test/error-404-isvc-fba55-predictor-676475b57f-fljvk"
Aug 05 17:35:26.408496 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:35:26.408476    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a489ce9e-f83f-47f4-8faf-2358a9349294-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-fba55-predictor-77c4dfc47d-j8skd\" (UID: \"a489ce9e-f83f-47f4-8faf-2358a9349294\") " pod="kserve-ci-e2e-test/success-200-isvc-fba55-predictor-77c4dfc47d-j8skd"
Aug 05 17:35:26.409997 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:35:26.409976    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a489ce9e-f83f-47f4-8faf-2358a9349294-proxy-tls\") pod \"success-200-isvc-fba55-predictor-77c4dfc47d-j8skd\" (UID: \"a489ce9e-f83f-47f4-8faf-2358a9349294\") " pod="kserve-ci-e2e-test/success-200-isvc-fba55-predictor-77c4dfc47d-j8skd"
Aug 05 17:35:26.416401 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:35:26.416366    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ts2wq\" (UniqueName: \"kubernetes.io/projected/a489ce9e-f83f-47f4-8faf-2358a9349294-kube-api-access-ts2wq\") pod \"success-200-isvc-fba55-predictor-77c4dfc47d-j8skd\" (UID: \"a489ce9e-f83f-47f4-8faf-2358a9349294\") " pod="kserve-ci-e2e-test/success-200-isvc-fba55-predictor-77c4dfc47d-j8skd"
Aug 05 17:35:26.508612 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:35:26.508585    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/366121e0-36b3-4704-a224-ee0e872f94e4-proxy-tls\") pod \"error-404-isvc-fba55-predictor-676475b57f-fljvk\" (UID: \"366121e0-36b3-4704-a224-ee0e872f94e4\") " pod="kserve-ci-e2e-test/error-404-isvc-fba55-predictor-676475b57f-fljvk"
Aug 05 17:35:26.508724 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:35:26.508627    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-knb6q\" (UniqueName: \"kubernetes.io/projected/366121e0-36b3-4704-a224-ee0e872f94e4-kube-api-access-knb6q\") pod \"error-404-isvc-fba55-predictor-676475b57f-fljvk\" (UID: \"366121e0-36b3-4704-a224-ee0e872f94e4\") " pod="kserve-ci-e2e-test/error-404-isvc-fba55-predictor-676475b57f-fljvk"
Aug 05 17:35:26.508724 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:35:26.508676    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/366121e0-36b3-4704-a224-ee0e872f94e4-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-fba55-predictor-676475b57f-fljvk\" (UID: \"366121e0-36b3-4704-a224-ee0e872f94e4\") " pod="kserve-ci-e2e-test/error-404-isvc-fba55-predictor-676475b57f-fljvk"
Aug 05 17:35:26.508846 ip-10-0-135-2 kubenswrapper[2571]: E0805 17:35:26.508729    2571 secret.go:189] Couldn't get secret kserve-ci-e2e-test/error-404-isvc-fba55-predictor-serving-cert: secret "error-404-isvc-fba55-predictor-serving-cert" not found
Aug 05 17:35:26.508846 ip-10-0-135-2 kubenswrapper[2571]: E0805 17:35:26.508793    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/366121e0-36b3-4704-a224-ee0e872f94e4-proxy-tls podName:366121e0-36b3-4704-a224-ee0e872f94e4 nodeName:}" failed. No retries permitted until 2026-08-05 17:35:27.00877712 +0000 UTC m=+1354.376143446 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/366121e0-36b3-4704-a224-ee0e872f94e4-proxy-tls") pod "error-404-isvc-fba55-predictor-676475b57f-fljvk" (UID: "366121e0-36b3-4704-a224-ee0e872f94e4") : secret "error-404-isvc-fba55-predictor-serving-cert" not found
Aug 05 17:35:26.509252 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:35:26.509231    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/366121e0-36b3-4704-a224-ee0e872f94e4-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-fba55-predictor-676475b57f-fljvk\" (UID: \"366121e0-36b3-4704-a224-ee0e872f94e4\") " pod="kserve-ci-e2e-test/error-404-isvc-fba55-predictor-676475b57f-fljvk"
Aug 05 17:35:26.516415 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:35:26.516392    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-fba55-predictor-77c4dfc47d-j8skd"
Aug 05 17:35:26.517204 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:35:26.517177    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-knb6q\" (UniqueName: \"kubernetes.io/projected/366121e0-36b3-4704-a224-ee0e872f94e4-kube-api-access-knb6q\") pod \"error-404-isvc-fba55-predictor-676475b57f-fljvk\" (UID: \"366121e0-36b3-4704-a224-ee0e872f94e4\") " pod="kserve-ci-e2e-test/error-404-isvc-fba55-predictor-676475b57f-fljvk"
Aug 05 17:35:26.630807 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:35:26.630778    2571 generic.go:356] "Generic (PLEG): container finished" podID="108727d9-7bf4-4042-9a0f-8b07a5a12f91" containerID="039d99edd06cbc2525a5d5cb18022a9f46d5bf67aec4003dcda6d164ed6fee5c" exitCode=2
Aug 05 17:35:26.630954 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:35:26.630859    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-a0ee2-predictor-7f8d8c965f-kj7sw" event={"ID":"108727d9-7bf4-4042-9a0f-8b07a5a12f91","Type":"ContainerDied","Data":"039d99edd06cbc2525a5d5cb18022a9f46d5bf67aec4003dcda6d164ed6fee5c"}
Aug 05 17:35:26.632692 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:35:26.632668    2571 generic.go:356] "Generic (PLEG): container finished" podID="cf5fb621-292d-454c-88d6-e66edcd9dfb5" containerID="7e2dbcef41e03cee3ac11588a096d677760fa135377e473abdcead871d635512" exitCode=2
Aug 05 17:35:26.632789 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:35:26.632709    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-a0ee2-predictor-69b645cc54-6zzvp" event={"ID":"cf5fb621-292d-454c-88d6-e66edcd9dfb5","Type":"ContainerDied","Data":"7e2dbcef41e03cee3ac11588a096d677760fa135377e473abdcead871d635512"}
Aug 05 17:35:26.649985 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:35:26.649927    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-fba55-predictor-77c4dfc47d-j8skd"]
Aug 05 17:35:26.652515 ip-10-0-135-2 kubenswrapper[2571]: W0805 17:35:26.652491    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda489ce9e_f83f_47f4_8faf_2358a9349294.slice/crio-87fbc49d7625a4b22baada8f8c3768c5aef5edb86ed6e41b7806915085c050b5 WatchSource:0}: Error finding container 87fbc49d7625a4b22baada8f8c3768c5aef5edb86ed6e41b7806915085c050b5: Status 404 returned error can't find the container with id 87fbc49d7625a4b22baada8f8c3768c5aef5edb86ed6e41b7806915085c050b5
Aug 05 17:35:27.013039 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:35:27.013009    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/366121e0-36b3-4704-a224-ee0e872f94e4-proxy-tls\") pod \"error-404-isvc-fba55-predictor-676475b57f-fljvk\" (UID: \"366121e0-36b3-4704-a224-ee0e872f94e4\") " pod="kserve-ci-e2e-test/error-404-isvc-fba55-predictor-676475b57f-fljvk"
Aug 05 17:35:27.015388 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:35:27.015348    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/366121e0-36b3-4704-a224-ee0e872f94e4-proxy-tls\") pod \"error-404-isvc-fba55-predictor-676475b57f-fljvk\" (UID: \"366121e0-36b3-4704-a224-ee0e872f94e4\") " pod="kserve-ci-e2e-test/error-404-isvc-fba55-predictor-676475b57f-fljvk"
Aug 05 17:35:27.223920 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:35:27.223887    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-fba55-predictor-676475b57f-fljvk"
Aug 05 17:35:27.356967 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:35:27.356944    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-fba55-predictor-676475b57f-fljvk"]
Aug 05 17:35:27.359119 ip-10-0-135-2 kubenswrapper[2571]: W0805 17:35:27.359093    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod366121e0_36b3_4704_a224_ee0e872f94e4.slice/crio-2512c28dd4312702a0bf20ed0f6c158445cb4b91d3500e2faa4922e8e8d11aeb WatchSource:0}: Error finding container 2512c28dd4312702a0bf20ed0f6c158445cb4b91d3500e2faa4922e8e8d11aeb: Status 404 returned error can't find the container with id 2512c28dd4312702a0bf20ed0f6c158445cb4b91d3500e2faa4922e8e8d11aeb
Aug 05 17:35:27.639148 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:35:27.639049    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-fba55-predictor-77c4dfc47d-j8skd" event={"ID":"a489ce9e-f83f-47f4-8faf-2358a9349294","Type":"ContainerStarted","Data":"f611045c7e6a80b8faa32f5a458b7330ab0274b615a572a70be008960c02ad2e"}
Aug 05 17:35:27.639291 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:35:27.639205    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-fba55-predictor-77c4dfc47d-j8skd"
Aug 05 17:35:27.639291 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:35:27.639262    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-fba55-predictor-77c4dfc47d-j8skd" event={"ID":"a489ce9e-f83f-47f4-8faf-2358a9349294","Type":"ContainerStarted","Data":"520b1b020911d8435020fb2e94c66b0c29e0f32c33b92653ba9ea7d2c7c9103d"}
Aug 05 17:35:27.639444 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:35:27.639300    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-fba55-predictor-77c4dfc47d-j8skd" event={"ID":"a489ce9e-f83f-47f4-8faf-2358a9349294","Type":"ContainerStarted","Data":"87fbc49d7625a4b22baada8f8c3768c5aef5edb86ed6e41b7806915085c050b5"}
Aug 05 17:35:27.641854 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:35:27.641299    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-fba55-predictor-676475b57f-fljvk" event={"ID":"366121e0-36b3-4704-a224-ee0e872f94e4","Type":"ContainerStarted","Data":"c07e1e66686daa816d2114bef7210b4e4abda360266cca9d86672933983ba610"}
Aug 05 17:35:27.641854 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:35:27.641353    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-fba55-predictor-676475b57f-fljvk" event={"ID":"366121e0-36b3-4704-a224-ee0e872f94e4","Type":"ContainerStarted","Data":"c08f54086eca55ccab5f173c65e9745cda91215666caf391c6e83a19be2a89fc"}
Aug 05 17:35:27.641854 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:35:27.641409    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-fba55-predictor-676475b57f-fljvk"
Aug 05 17:35:27.641854 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:35:27.641438    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-fba55-predictor-676475b57f-fljvk" event={"ID":"366121e0-36b3-4704-a224-ee0e872f94e4","Type":"ContainerStarted","Data":"2512c28dd4312702a0bf20ed0f6c158445cb4b91d3500e2faa4922e8e8d11aeb"}
Aug 05 17:35:27.641854 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:35:27.641469    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-fba55-predictor-676475b57f-fljvk"
Aug 05 17:35:27.643252 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:35:27.643208    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-fba55-predictor-676475b57f-fljvk" podUID="366121e0-36b3-4704-a224-ee0e872f94e4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.35:8080: connect: connection refused"
Aug 05 17:35:27.692137 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:35:27.692080    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/error-404-isvc-fba55-predictor-676475b57f-fljvk" podStartSLOduration=1.692063533 podStartE2EDuration="1.692063533s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-05 17:35:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-05 17:35:27.69198864 +0000 UTC m=+1355.059354984" watchObservedRunningTime="2026-08-05 17:35:27.692063533 +0000 UTC m=+1355.059429878"
Aug 05 17:35:27.693484 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:35:27.693453    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/success-200-isvc-fba55-predictor-77c4dfc47d-j8skd" podStartSLOduration=1.693444431 podStartE2EDuration="1.693444431s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-05 17:35:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-05 17:35:27.663121577 +0000 UTC m=+1355.030487920" watchObservedRunningTime="2026-08-05 17:35:27.693444431 +0000 UTC m=+1355.060810774"
Aug 05 17:35:28.324478 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:35:28.324437    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-a0ee2-predictor-7f8d8c965f-kj7sw" podUID="108727d9-7bf4-4042-9a0f-8b07a5a12f91" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.31:8643/healthz\": dial tcp 10.134.0.31:8643: connect: connection refused"
Aug 05 17:35:28.329912 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:35:28.329882    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-a0ee2-predictor-7f8d8c965f-kj7sw" podUID="108727d9-7bf4-4042-9a0f-8b07a5a12f91" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:8080: connect: connection refused"
Aug 05 17:35:28.647677 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:35:28.647585    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-fba55-predictor-676475b57f-fljvk" podUID="366121e0-36b3-4704-a224-ee0e872f94e4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.35:8080: connect: connection refused"
Aug 05 17:35:28.647994 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:35:28.647742    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-fba55-predictor-77c4dfc47d-j8skd"
Aug 05 17:35:28.649029 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:35:28.649002    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-fba55-predictor-77c4dfc47d-j8skd" podUID="a489ce9e-f83f-47f4-8faf-2358a9349294" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused"
Aug 05 17:35:29.265957 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:35:29.265935    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-a0ee2-predictor-69b645cc54-6zzvp"
Aug 05 17:35:29.434307 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:35:29.434217    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-brnzk\" (UniqueName: \"kubernetes.io/projected/cf5fb621-292d-454c-88d6-e66edcd9dfb5-kube-api-access-brnzk\") pod \"cf5fb621-292d-454c-88d6-e66edcd9dfb5\" (UID: \"cf5fb621-292d-454c-88d6-e66edcd9dfb5\") "
Aug 05 17:35:29.434307 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:35:29.434263    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/cf5fb621-292d-454c-88d6-e66edcd9dfb5-proxy-tls\") pod \"cf5fb621-292d-454c-88d6-e66edcd9dfb5\" (UID: \"cf5fb621-292d-454c-88d6-e66edcd9dfb5\") "
Aug 05 17:35:29.434769 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:35:29.434325    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/cf5fb621-292d-454c-88d6-e66edcd9dfb5-kube-rbac-proxy-sar-config\") pod \"cf5fb621-292d-454c-88d6-e66edcd9dfb5\" (UID: \"cf5fb621-292d-454c-88d6-e66edcd9dfb5\") "
Aug 05 17:35:29.434769 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:35:29.434638    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cf5fb621-292d-454c-88d6-e66edcd9dfb5-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "cf5fb621-292d-454c-88d6-e66edcd9dfb5" (UID: "cf5fb621-292d-454c-88d6-e66edcd9dfb5"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 05 17:35:29.436429 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:35:29.436401    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cf5fb621-292d-454c-88d6-e66edcd9dfb5-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "cf5fb621-292d-454c-88d6-e66edcd9dfb5" (UID: "cf5fb621-292d-454c-88d6-e66edcd9dfb5"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 05 17:35:29.436429 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:35:29.436416    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cf5fb621-292d-454c-88d6-e66edcd9dfb5-kube-api-access-brnzk" (OuterVolumeSpecName: "kube-api-access-brnzk") pod "cf5fb621-292d-454c-88d6-e66edcd9dfb5" (UID: "cf5fb621-292d-454c-88d6-e66edcd9dfb5"). InnerVolumeSpecName "kube-api-access-brnzk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 05 17:35:29.534995 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:35:29.534951    2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-brnzk\" (UniqueName: \"kubernetes.io/projected/cf5fb621-292d-454c-88d6-e66edcd9dfb5-kube-api-access-brnzk\") on node \"ip-10-0-135-2.ec2.internal\" DevicePath \"\""
Aug 05 17:35:29.534995 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:35:29.534989    2571 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/cf5fb621-292d-454c-88d6-e66edcd9dfb5-proxy-tls\") on node \"ip-10-0-135-2.ec2.internal\" DevicePath \"\""
Aug 05 17:35:29.534995 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:35:29.534999    2571 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/cf5fb621-292d-454c-88d6-e66edcd9dfb5-kube-rbac-proxy-sar-config\") on node \"ip-10-0-135-2.ec2.internal\" DevicePath \"\""
Aug 05 17:35:29.654097 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:35:29.654066    2571 generic.go:356] "Generic (PLEG): container finished" podID="cf5fb621-292d-454c-88d6-e66edcd9dfb5" containerID="bbf9e44a35b24a552fa29c1a62006ba0185d457ce4a7df161062333038edb0aa" exitCode=0
Aug 05 17:35:29.654238 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:35:29.654129    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-a0ee2-predictor-69b645cc54-6zzvp"
Aug 05 17:35:29.654238 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:35:29.654163    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-a0ee2-predictor-69b645cc54-6zzvp" event={"ID":"cf5fb621-292d-454c-88d6-e66edcd9dfb5","Type":"ContainerDied","Data":"bbf9e44a35b24a552fa29c1a62006ba0185d457ce4a7df161062333038edb0aa"}
Aug 05 17:35:29.654395 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:35:29.654247    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-a0ee2-predictor-69b645cc54-6zzvp" event={"ID":"cf5fb621-292d-454c-88d6-e66edcd9dfb5","Type":"ContainerDied","Data":"868fe003eaf7e2a544236d7c7e9b643b118b49a4051213408b5d449b1dcd9bcc"}
Aug 05 17:35:29.654395 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:35:29.654293    2571 scope.go:117] "RemoveContainer" containerID="7e2dbcef41e03cee3ac11588a096d677760fa135377e473abdcead871d635512"
Aug 05 17:35:29.658554 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:35:29.657836    2571 generic.go:356] "Generic (PLEG): container finished" podID="108727d9-7bf4-4042-9a0f-8b07a5a12f91" containerID="d38aed0ea994ddf62b53cbf2956dca2d006197c2df217c002047e7b081c68613" exitCode=0
Aug 05 17:35:29.658554 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:35:29.658526    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-a0ee2-predictor-7f8d8c965f-kj7sw" event={"ID":"108727d9-7bf4-4042-9a0f-8b07a5a12f91","Type":"ContainerDied","Data":"d38aed0ea994ddf62b53cbf2956dca2d006197c2df217c002047e7b081c68613"}
Aug 05 17:35:29.658844 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:35:29.658806    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-fba55-predictor-77c4dfc47d-j8skd" podUID="a489ce9e-f83f-47f4-8faf-2358a9349294" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused"
Aug 05 17:35:29.672769 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:35:29.672749    2571 scope.go:117] "RemoveContainer" containerID="bbf9e44a35b24a552fa29c1a62006ba0185d457ce4a7df161062333038edb0aa"
Aug 05 17:35:29.680894 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:35:29.680876    2571 scope.go:117] "RemoveContainer" containerID="7e2dbcef41e03cee3ac11588a096d677760fa135377e473abdcead871d635512"
Aug 05 17:35:29.681571 ip-10-0-135-2 kubenswrapper[2571]: E0805 17:35:29.681156    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7e2dbcef41e03cee3ac11588a096d677760fa135377e473abdcead871d635512\": container with ID starting with 7e2dbcef41e03cee3ac11588a096d677760fa135377e473abdcead871d635512 not found: ID does not exist" containerID="7e2dbcef41e03cee3ac11588a096d677760fa135377e473abdcead871d635512"
Aug 05 17:35:29.681571 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:35:29.681190    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7e2dbcef41e03cee3ac11588a096d677760fa135377e473abdcead871d635512"} err="failed to get container status \"7e2dbcef41e03cee3ac11588a096d677760fa135377e473abdcead871d635512\": rpc error: code = NotFound desc = could not find container \"7e2dbcef41e03cee3ac11588a096d677760fa135377e473abdcead871d635512\": container with ID starting with 7e2dbcef41e03cee3ac11588a096d677760fa135377e473abdcead871d635512 not found: ID does not exist"
Aug 05 17:35:29.681571 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:35:29.681212    2571 scope.go:117] "RemoveContainer" containerID="bbf9e44a35b24a552fa29c1a62006ba0185d457ce4a7df161062333038edb0aa"
Aug 05 17:35:29.681794 ip-10-0-135-2 kubenswrapper[2571]: E0805 17:35:29.681612    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bbf9e44a35b24a552fa29c1a62006ba0185d457ce4a7df161062333038edb0aa\": container with ID starting with bbf9e44a35b24a552fa29c1a62006ba0185d457ce4a7df161062333038edb0aa not found: ID does not exist" containerID="bbf9e44a35b24a552fa29c1a62006ba0185d457ce4a7df161062333038edb0aa"
Aug 05 17:35:29.681794 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:35:29.681644    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bbf9e44a35b24a552fa29c1a62006ba0185d457ce4a7df161062333038edb0aa"} err="failed to get container status \"bbf9e44a35b24a552fa29c1a62006ba0185d457ce4a7df161062333038edb0aa\": rpc error: code = NotFound desc = could not find container \"bbf9e44a35b24a552fa29c1a62006ba0185d457ce4a7df161062333038edb0aa\": container with ID starting with bbf9e44a35b24a552fa29c1a62006ba0185d457ce4a7df161062333038edb0aa not found: ID does not exist"
Aug 05 17:35:29.683531 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:35:29.683510    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-a0ee2-predictor-69b645cc54-6zzvp"]
Aug 05 17:35:29.687453 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:35:29.687406    2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-a0ee2-predictor-69b645cc54-6zzvp"]
Aug 05 17:35:29.721789 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:35:29.721771    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-a0ee2-predictor-7f8d8c965f-kj7sw"
Aug 05 17:35:29.838312 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:35:29.838287    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/108727d9-7bf4-4042-9a0f-8b07a5a12f91-proxy-tls\") pod \"108727d9-7bf4-4042-9a0f-8b07a5a12f91\" (UID: \"108727d9-7bf4-4042-9a0f-8b07a5a12f91\") "
Aug 05 17:35:29.838458 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:35:29.838329    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gvv8l\" (UniqueName: \"kubernetes.io/projected/108727d9-7bf4-4042-9a0f-8b07a5a12f91-kube-api-access-gvv8l\") pod \"108727d9-7bf4-4042-9a0f-8b07a5a12f91\" (UID: \"108727d9-7bf4-4042-9a0f-8b07a5a12f91\") "
Aug 05 17:35:29.838458 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:35:29.838366    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/108727d9-7bf4-4042-9a0f-8b07a5a12f91-kube-rbac-proxy-sar-config\") pod \"108727d9-7bf4-4042-9a0f-8b07a5a12f91\" (UID: \"108727d9-7bf4-4042-9a0f-8b07a5a12f91\") "
Aug 05 17:35:29.838766 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:35:29.838740    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/108727d9-7bf4-4042-9a0f-8b07a5a12f91-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "108727d9-7bf4-4042-9a0f-8b07a5a12f91" (UID: "108727d9-7bf4-4042-9a0f-8b07a5a12f91"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 05 17:35:29.840420 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:35:29.840395    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/108727d9-7bf4-4042-9a0f-8b07a5a12f91-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "108727d9-7bf4-4042-9a0f-8b07a5a12f91" (UID: "108727d9-7bf4-4042-9a0f-8b07a5a12f91"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 05 17:35:29.840501 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:35:29.840465    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/108727d9-7bf4-4042-9a0f-8b07a5a12f91-kube-api-access-gvv8l" (OuterVolumeSpecName: "kube-api-access-gvv8l") pod "108727d9-7bf4-4042-9a0f-8b07a5a12f91" (UID: "108727d9-7bf4-4042-9a0f-8b07a5a12f91"). InnerVolumeSpecName "kube-api-access-gvv8l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 05 17:35:29.939821 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:35:29.939771    2571 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/108727d9-7bf4-4042-9a0f-8b07a5a12f91-proxy-tls\") on node \"ip-10-0-135-2.ec2.internal\" DevicePath \"\""
Aug 05 17:35:29.939821 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:35:29.939793    2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gvv8l\" (UniqueName: \"kubernetes.io/projected/108727d9-7bf4-4042-9a0f-8b07a5a12f91-kube-api-access-gvv8l\") on node \"ip-10-0-135-2.ec2.internal\" DevicePath \"\""
Aug 05 17:35:29.939821 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:35:29.939803    2571 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/108727d9-7bf4-4042-9a0f-8b07a5a12f91-kube-rbac-proxy-sar-config\") on node \"ip-10-0-135-2.ec2.internal\" DevicePath \"\""
Aug 05 17:35:30.664117 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:35:30.664039    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-a0ee2-predictor-7f8d8c965f-kj7sw" event={"ID":"108727d9-7bf4-4042-9a0f-8b07a5a12f91","Type":"ContainerDied","Data":"73563aa6b80a115aa05b5a5b80c98e67996572ade320ac6cb8b6cc48f8630648"}
Aug 05 17:35:30.664117 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:35:30.664103    2571 scope.go:117] "RemoveContainer" containerID="039d99edd06cbc2525a5d5cb18022a9f46d5bf67aec4003dcda6d164ed6fee5c"
Aug 05 17:35:30.664609 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:35:30.664055    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-a0ee2-predictor-7f8d8c965f-kj7sw"
Aug 05 17:35:30.672521 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:35:30.672506    2571 scope.go:117] "RemoveContainer" containerID="d38aed0ea994ddf62b53cbf2956dca2d006197c2df217c002047e7b081c68613"
Aug 05 17:35:30.686282 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:35:30.686260    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-a0ee2-predictor-7f8d8c965f-kj7sw"]
Aug 05 17:35:30.690310 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:35:30.690287    2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-a0ee2-predictor-7f8d8c965f-kj7sw"]
Aug 05 17:35:31.137648 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:35:31.137611    2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="108727d9-7bf4-4042-9a0f-8b07a5a12f91" path="/var/lib/kubelet/pods/108727d9-7bf4-4042-9a0f-8b07a5a12f91/volumes"
Aug 05 17:35:31.138034 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:35:31.138022    2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cf5fb621-292d-454c-88d6-e66edcd9dfb5" path="/var/lib/kubelet/pods/cf5fb621-292d-454c-88d6-e66edcd9dfb5/volumes"
Aug 05 17:35:33.653126 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:35:33.653099    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-fba55-predictor-676475b57f-fljvk"
Aug 05 17:35:33.653567 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:35:33.653541    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-fba55-predictor-676475b57f-fljvk" podUID="366121e0-36b3-4704-a224-ee0e872f94e4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.35:8080: connect: connection refused"
Aug 05 17:35:34.663365 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:35:34.663336    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-fba55-predictor-77c4dfc47d-j8skd"
Aug 05 17:35:34.663890 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:35:34.663860    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-fba55-predictor-77c4dfc47d-j8skd" podUID="a489ce9e-f83f-47f4-8faf-2358a9349294" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused"
Aug 05 17:35:43.654000 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:35:43.653958    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-fba55-predictor-676475b57f-fljvk" podUID="366121e0-36b3-4704-a224-ee0e872f94e4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.35:8080: connect: connection refused"
Aug 05 17:35:44.664183 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:35:44.664143    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-fba55-predictor-77c4dfc47d-j8skd" podUID="a489ce9e-f83f-47f4-8faf-2358a9349294" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused"
Aug 05 17:35:53.654292 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:35:53.654253    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-fba55-predictor-676475b57f-fljvk" podUID="366121e0-36b3-4704-a224-ee0e872f94e4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.35:8080: connect: connection refused"
Aug 05 17:35:54.664280 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:35:54.664244    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-fba55-predictor-77c4dfc47d-j8skd" podUID="a489ce9e-f83f-47f4-8faf-2358a9349294" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused"
Aug 05 17:36:03.653576 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:36:03.653535    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-fba55-predictor-676475b57f-fljvk" podUID="366121e0-36b3-4704-a224-ee0e872f94e4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.35:8080: connect: connection refused"
Aug 05 17:36:04.664252 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:36:04.664210    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-fba55-predictor-77c4dfc47d-j8skd" podUID="a489ce9e-f83f-47f4-8faf-2358a9349294" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused"
Aug 05 17:36:13.654535 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:36:13.654502    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-fba55-predictor-676475b57f-fljvk"
Aug 05 17:36:14.665459 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:36:14.665430    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-fba55-predictor-77c4dfc47d-j8skd"
Aug 05 17:36:26.062802 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:36:26.062768    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-d9755-predictor-d8c999f56-sg8lf"]
Aug 05 17:36:26.063160 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:36:26.063023    2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-d9755-predictor-d8c999f56-sg8lf" podUID="37d7cb85-d235-490e-80dc-9e73cc8cfa29" containerName="kserve-container" containerID="cri-o://45dcb43937a43b7375beaf5bb4431800e574349b409baa0252e629af9021c9c3" gracePeriod=30
Aug 05 17:36:26.063160 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:36:26.063035    2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-d9755-predictor-d8c999f56-sg8lf" podUID="37d7cb85-d235-490e-80dc-9e73cc8cfa29" containerName="kube-rbac-proxy" containerID="cri-o://eadae4a9c5e1fd7118df87e82da719a3b59c612a20d54fd0ec8e4f678f46c384" gracePeriod=30
Aug 05 17:36:26.121229 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:36:26.121195    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-d9755-predictor-78b5549d57-wh2st"]
Aug 05 17:36:26.121478 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:36:26.121448    2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-d9755-predictor-78b5549d57-wh2st" podUID="0fe55532-2a2b-41c1-92d4-dc7f0e201ccc" containerName="kserve-container" containerID="cri-o://29331d6ac473f82798132463a470532d5a7d87b5e0d210b81ffbc82b114172dc" gracePeriod=30
Aug 05 17:36:26.121653 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:36:26.121497    2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-d9755-predictor-78b5549d57-wh2st" podUID="0fe55532-2a2b-41c1-92d4-dc7f0e201ccc" containerName="kube-rbac-proxy" containerID="cri-o://f10060501d672f4cbd8ae351765abce050c0e4643ae67970d41545f3de958e3c" gracePeriod=30
Aug 05 17:36:26.154472 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:36:26.154441    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-27640-predictor-55c9498c7d-ll287"]
Aug 05 17:36:26.154870 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:36:26.154852    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="108727d9-7bf4-4042-9a0f-8b07a5a12f91" containerName="kserve-container"
Aug 05 17:36:26.154950 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:36:26.154873    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="108727d9-7bf4-4042-9a0f-8b07a5a12f91" containerName="kserve-container"
Aug 05 17:36:26.154950 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:36:26.154891    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="108727d9-7bf4-4042-9a0f-8b07a5a12f91" containerName="kube-rbac-proxy"
Aug 05 17:36:26.154950 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:36:26.154900    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="108727d9-7bf4-4042-9a0f-8b07a5a12f91" containerName="kube-rbac-proxy"
Aug 05 17:36:26.154950 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:36:26.154921    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cf5fb621-292d-454c-88d6-e66edcd9dfb5" containerName="kserve-container"
Aug 05 17:36:26.154950 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:36:26.154929    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="cf5fb621-292d-454c-88d6-e66edcd9dfb5" containerName="kserve-container"
Aug 05 17:36:26.154950 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:36:26.154940    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cf5fb621-292d-454c-88d6-e66edcd9dfb5" containerName="kube-rbac-proxy"
Aug 05 17:36:26.154950 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:36:26.154949    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="cf5fb621-292d-454c-88d6-e66edcd9dfb5" containerName="kube-rbac-proxy"
Aug 05 17:36:26.155283 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:36:26.155058    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="cf5fb621-292d-454c-88d6-e66edcd9dfb5" containerName="kube-rbac-proxy"
Aug 05 17:36:26.155283 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:36:26.155072    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="108727d9-7bf4-4042-9a0f-8b07a5a12f91" containerName="kserve-container"
Aug 05 17:36:26.155283 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:36:26.155087    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="108727d9-7bf4-4042-9a0f-8b07a5a12f91" containerName="kube-rbac-proxy"
Aug 05 17:36:26.155283 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:36:26.155096    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="cf5fb621-292d-454c-88d6-e66edcd9dfb5" containerName="kserve-container"
Aug 05 17:36:26.158323 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:36:26.158305    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-27640-predictor-55c9498c7d-ll287"
Aug 05 17:36:26.160365 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:36:26.160346    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-27640-kube-rbac-proxy-sar-config\""
Aug 05 17:36:26.160459 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:36:26.160410    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-27640-predictor-serving-cert\""
Aug 05 17:36:26.170082 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:36:26.170061    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-27640-predictor-55c9498c7d-ll287"]
Aug 05 17:36:26.278224 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:36:26.278193    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-27640-predictor-799cfccbdc-v8s6m"]
Aug 05 17:36:26.281843 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:36:26.281826    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-27640-predictor-799cfccbdc-v8s6m"
Aug 05 17:36:26.283900 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:36:26.283875    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-27640-kube-rbac-proxy-sar-config\""
Aug 05 17:36:26.284419 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:36:26.284399    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-27640-predictor-serving-cert\""
Aug 05 17:36:26.293263 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:36:26.293236    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z68z9\" (UniqueName: \"kubernetes.io/projected/aee60f09-ffd8-4733-b965-a572468b235c-kube-api-access-z68z9\") pod \"success-200-isvc-27640-predictor-55c9498c7d-ll287\" (UID: \"aee60f09-ffd8-4733-b965-a572468b235c\") " pod="kserve-ci-e2e-test/success-200-isvc-27640-predictor-55c9498c7d-ll287"
Aug 05 17:36:26.293407 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:36:26.293327    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/aee60f09-ffd8-4733-b965-a572468b235c-proxy-tls\") pod \"success-200-isvc-27640-predictor-55c9498c7d-ll287\" (UID: \"aee60f09-ffd8-4733-b965-a572468b235c\") " pod="kserve-ci-e2e-test/success-200-isvc-27640-predictor-55c9498c7d-ll287"
Aug 05 17:36:26.293783 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:36:26.293465    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/aee60f09-ffd8-4733-b965-a572468b235c-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-27640-predictor-55c9498c7d-ll287\" (UID: \"aee60f09-ffd8-4733-b965-a572468b235c\") " pod="kserve-ci-e2e-test/success-200-isvc-27640-predictor-55c9498c7d-ll287"
Aug 05 17:36:26.295578 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:36:26.295561    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-27640-predictor-799cfccbdc-v8s6m"]
Aug 05 17:36:26.394670 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:36:26.394604    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/aee60f09-ffd8-4733-b965-a572468b235c-proxy-tls\") pod \"success-200-isvc-27640-predictor-55c9498c7d-ll287\" (UID: \"aee60f09-ffd8-4733-b965-a572468b235c\") " pod="kserve-ci-e2e-test/success-200-isvc-27640-predictor-55c9498c7d-ll287"
Aug 05 17:36:26.394670 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:36:26.394651    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wktb4\" (UniqueName: \"kubernetes.io/projected/2dbefbae-0497-4396-ab82-abcd1a08abc2-kube-api-access-wktb4\") pod \"error-404-isvc-27640-predictor-799cfccbdc-v8s6m\" (UID: \"2dbefbae-0497-4396-ab82-abcd1a08abc2\") " pod="kserve-ci-e2e-test/error-404-isvc-27640-predictor-799cfccbdc-v8s6m"
Aug 05 17:36:26.394821 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:36:26.394691    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/aee60f09-ffd8-4733-b965-a572468b235c-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-27640-predictor-55c9498c7d-ll287\" (UID: \"aee60f09-ffd8-4733-b965-a572468b235c\") " pod="kserve-ci-e2e-test/success-200-isvc-27640-predictor-55c9498c7d-ll287"
Aug 05 17:36:26.394821 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:36:26.394729    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2dbefbae-0497-4396-ab82-abcd1a08abc2-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-27640-predictor-799cfccbdc-v8s6m\" (UID: \"2dbefbae-0497-4396-ab82-abcd1a08abc2\") " pod="kserve-ci-e2e-test/error-404-isvc-27640-predictor-799cfccbdc-v8s6m"
Aug 05 17:36:26.394821 ip-10-0-135-2 kubenswrapper[2571]: E0805 17:36:26.394761    2571 secret.go:189] Couldn't get secret kserve-ci-e2e-test/success-200-isvc-27640-predictor-serving-cert: secret "success-200-isvc-27640-predictor-serving-cert" not found
Aug 05 17:36:26.394821 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:36:26.394807    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z68z9\" (UniqueName: \"kubernetes.io/projected/aee60f09-ffd8-4733-b965-a572468b235c-kube-api-access-z68z9\") pod \"success-200-isvc-27640-predictor-55c9498c7d-ll287\" (UID: \"aee60f09-ffd8-4733-b965-a572468b235c\") " pod="kserve-ci-e2e-test/success-200-isvc-27640-predictor-55c9498c7d-ll287"
Aug 05 17:36:26.395004 ip-10-0-135-2 kubenswrapper[2571]: E0805 17:36:26.394822    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/aee60f09-ffd8-4733-b965-a572468b235c-proxy-tls podName:aee60f09-ffd8-4733-b965-a572468b235c nodeName:}" failed. No retries permitted until 2026-08-05 17:36:26.894800933 +0000 UTC m=+1414.262167261 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/aee60f09-ffd8-4733-b965-a572468b235c-proxy-tls") pod "success-200-isvc-27640-predictor-55c9498c7d-ll287" (UID: "aee60f09-ffd8-4733-b965-a572468b235c") : secret "success-200-isvc-27640-predictor-serving-cert" not found
Aug 05 17:36:26.395004 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:36:26.394854    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2dbefbae-0497-4396-ab82-abcd1a08abc2-proxy-tls\") pod \"error-404-isvc-27640-predictor-799cfccbdc-v8s6m\" (UID: \"2dbefbae-0497-4396-ab82-abcd1a08abc2\") " pod="kserve-ci-e2e-test/error-404-isvc-27640-predictor-799cfccbdc-v8s6m"
Aug 05 17:36:26.395356 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:36:26.395336    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/aee60f09-ffd8-4733-b965-a572468b235c-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-27640-predictor-55c9498c7d-ll287\" (UID: \"aee60f09-ffd8-4733-b965-a572468b235c\") " pod="kserve-ci-e2e-test/success-200-isvc-27640-predictor-55c9498c7d-ll287"
Aug 05 17:36:26.408317 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:36:26.408293    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z68z9\" (UniqueName: \"kubernetes.io/projected/aee60f09-ffd8-4733-b965-a572468b235c-kube-api-access-z68z9\") pod \"success-200-isvc-27640-predictor-55c9498c7d-ll287\" (UID: \"aee60f09-ffd8-4733-b965-a572468b235c\") " pod="kserve-ci-e2e-test/success-200-isvc-27640-predictor-55c9498c7d-ll287"
Aug 05 17:36:26.495767 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:36:26.495744    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2dbefbae-0497-4396-ab82-abcd1a08abc2-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-27640-predictor-799cfccbdc-v8s6m\" (UID: \"2dbefbae-0497-4396-ab82-abcd1a08abc2\") " pod="kserve-ci-e2e-test/error-404-isvc-27640-predictor-799cfccbdc-v8s6m"
Aug 05 17:36:26.495945 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:36:26.495788    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2dbefbae-0497-4396-ab82-abcd1a08abc2-proxy-tls\") pod \"error-404-isvc-27640-predictor-799cfccbdc-v8s6m\" (UID: \"2dbefbae-0497-4396-ab82-abcd1a08abc2\") " pod="kserve-ci-e2e-test/error-404-isvc-27640-predictor-799cfccbdc-v8s6m"
Aug 05 17:36:26.496017 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:36:26.495951    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wktb4\" (UniqueName: \"kubernetes.io/projected/2dbefbae-0497-4396-ab82-abcd1a08abc2-kube-api-access-wktb4\") pod \"error-404-isvc-27640-predictor-799cfccbdc-v8s6m\" (UID: \"2dbefbae-0497-4396-ab82-abcd1a08abc2\") " pod="kserve-ci-e2e-test/error-404-isvc-27640-predictor-799cfccbdc-v8s6m"
Aug 05 17:36:26.496442 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:36:26.496423    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2dbefbae-0497-4396-ab82-abcd1a08abc2-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-27640-predictor-799cfccbdc-v8s6m\" (UID: \"2dbefbae-0497-4396-ab82-abcd1a08abc2\") " pod="kserve-ci-e2e-test/error-404-isvc-27640-predictor-799cfccbdc-v8s6m"
Aug 05 17:36:26.497984 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:36:26.497964    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2dbefbae-0497-4396-ab82-abcd1a08abc2-proxy-tls\") pod \"error-404-isvc-27640-predictor-799cfccbdc-v8s6m\" (UID: \"2dbefbae-0497-4396-ab82-abcd1a08abc2\") " pod="kserve-ci-e2e-test/error-404-isvc-27640-predictor-799cfccbdc-v8s6m"
Aug 05 17:36:26.507165 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:36:26.507145    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wktb4\" (UniqueName: \"kubernetes.io/projected/2dbefbae-0497-4396-ab82-abcd1a08abc2-kube-api-access-wktb4\") pod \"error-404-isvc-27640-predictor-799cfccbdc-v8s6m\" (UID: \"2dbefbae-0497-4396-ab82-abcd1a08abc2\") " pod="kserve-ci-e2e-test/error-404-isvc-27640-predictor-799cfccbdc-v8s6m"
Aug 05 17:36:26.591340 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:36:26.591312    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-27640-predictor-799cfccbdc-v8s6m"
Aug 05 17:36:26.726856 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:36:26.726830    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-27640-predictor-799cfccbdc-v8s6m"]
Aug 05 17:36:26.729418 ip-10-0-135-2 kubenswrapper[2571]: W0805 17:36:26.729391    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2dbefbae_0497_4396_ab82_abcd1a08abc2.slice/crio-20c39747a90d6a7f01e63d3a274265da7cd7b0a6f2c4176fcf35a92b2b0f0dbc WatchSource:0}: Error finding container 20c39747a90d6a7f01e63d3a274265da7cd7b0a6f2c4176fcf35a92b2b0f0dbc: Status 404 returned error can't find the container with id 20c39747a90d6a7f01e63d3a274265da7cd7b0a6f2c4176fcf35a92b2b0f0dbc
Aug 05 17:36:26.899450 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:36:26.899422    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/aee60f09-ffd8-4733-b965-a572468b235c-proxy-tls\") pod \"success-200-isvc-27640-predictor-55c9498c7d-ll287\" (UID: \"aee60f09-ffd8-4733-b965-a572468b235c\") " pod="kserve-ci-e2e-test/success-200-isvc-27640-predictor-55c9498c7d-ll287"
Aug 05 17:36:26.901582 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:36:26.901565    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/aee60f09-ffd8-4733-b965-a572468b235c-proxy-tls\") pod \"success-200-isvc-27640-predictor-55c9498c7d-ll287\" (UID: \"aee60f09-ffd8-4733-b965-a572468b235c\") " pod="kserve-ci-e2e-test/success-200-isvc-27640-predictor-55c9498c7d-ll287"
Aug 05 17:36:26.906938 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:36:26.906915    2571 generic.go:356] "Generic (PLEG): container finished" podID="0fe55532-2a2b-41c1-92d4-dc7f0e201ccc" containerID="f10060501d672f4cbd8ae351765abce050c0e4643ae67970d41545f3de958e3c" exitCode=2
Aug 05 17:36:26.907044 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:36:26.906999    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-d9755-predictor-78b5549d57-wh2st" event={"ID":"0fe55532-2a2b-41c1-92d4-dc7f0e201ccc","Type":"ContainerDied","Data":"f10060501d672f4cbd8ae351765abce050c0e4643ae67970d41545f3de958e3c"}
Aug 05 17:36:26.908600 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:36:26.908573    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-27640-predictor-799cfccbdc-v8s6m" event={"ID":"2dbefbae-0497-4396-ab82-abcd1a08abc2","Type":"ContainerStarted","Data":"2c848b6c76864c66558141dbc2395c4c12bcf2746ed906d19172f900267c8e4f"}
Aug 05 17:36:26.908718 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:36:26.908633    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-27640-predictor-799cfccbdc-v8s6m"
Aug 05 17:36:26.908718 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:36:26.908661    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-27640-predictor-799cfccbdc-v8s6m" event={"ID":"2dbefbae-0497-4396-ab82-abcd1a08abc2","Type":"ContainerStarted","Data":"0a29512ff22422e76bb11238207fd052b40e3a4f81855c0e6d50a9b299d7ecbc"}
Aug 05 17:36:26.908718 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:36:26.908691    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-27640-predictor-799cfccbdc-v8s6m" event={"ID":"2dbefbae-0497-4396-ab82-abcd1a08abc2","Type":"ContainerStarted","Data":"20c39747a90d6a7f01e63d3a274265da7cd7b0a6f2c4176fcf35a92b2b0f0dbc"}
Aug 05 17:36:26.908870 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:36:26.908721    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-27640-predictor-799cfccbdc-v8s6m"
Aug 05 17:36:26.910157 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:36:26.910131    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-27640-predictor-799cfccbdc-v8s6m" podUID="2dbefbae-0497-4396-ab82-abcd1a08abc2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.37:8080: connect: connection refused"
Aug 05 17:36:26.910562 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:36:26.910543    2571 generic.go:356] "Generic (PLEG): container finished" podID="37d7cb85-d235-490e-80dc-9e73cc8cfa29" containerID="eadae4a9c5e1fd7118df87e82da719a3b59c612a20d54fd0ec8e4f678f46c384" exitCode=2
Aug 05 17:36:26.910644 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:36:26.910611    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-d9755-predictor-d8c999f56-sg8lf" event={"ID":"37d7cb85-d235-490e-80dc-9e73cc8cfa29","Type":"ContainerDied","Data":"eadae4a9c5e1fd7118df87e82da719a3b59c612a20d54fd0ec8e4f678f46c384"}
Aug 05 17:36:26.930403 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:36:26.930320    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/error-404-isvc-27640-predictor-799cfccbdc-v8s6m" podStartSLOduration=0.93030662 podStartE2EDuration="930.30662ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-05 17:36:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-05 17:36:26.928564458 +0000 UTC m=+1414.295930810" watchObservedRunningTime="2026-08-05 17:36:26.93030662 +0000 UTC m=+1414.297672963"
Aug 05 17:36:27.068703 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:36:27.068664    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-27640-predictor-55c9498c7d-ll287"
Aug 05 17:36:27.201058 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:36:27.201029    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-27640-predictor-55c9498c7d-ll287"]
Aug 05 17:36:27.202778 ip-10-0-135-2 kubenswrapper[2571]: W0805 17:36:27.202753    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaee60f09_ffd8_4733_b965_a572468b235c.slice/crio-f8ed8cdc1109642e433eeab077d2bf53f570626302ec645bf9e54a33ac570feb WatchSource:0}: Error finding container f8ed8cdc1109642e433eeab077d2bf53f570626302ec645bf9e54a33ac570feb: Status 404 returned error can't find the container with id f8ed8cdc1109642e433eeab077d2bf53f570626302ec645bf9e54a33ac570feb
Aug 05 17:36:27.922924 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:36:27.922884    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-27640-predictor-55c9498c7d-ll287" event={"ID":"aee60f09-ffd8-4733-b965-a572468b235c","Type":"ContainerStarted","Data":"b8bfc1cb8de64bfece4617bc1b66e4849f167b8a38bb04d12474ce54085d1421"}
Aug 05 17:36:27.923126 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:36:27.922985    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-27640-predictor-55c9498c7d-ll287" event={"ID":"aee60f09-ffd8-4733-b965-a572468b235c","Type":"ContainerStarted","Data":"1660bb41322293a4b82463349bc8c4b99cef00b0ba1ac6558a574ee1a91fc6db"}
Aug 05 17:36:27.923199 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:36:27.923184    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-27640-predictor-55c9498c7d-ll287"
Aug 05 17:36:27.923324 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:36:27.923227    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-27640-predictor-55c9498c7d-ll287"
Aug 05 17:36:27.923324 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:36:27.923256    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-27640-predictor-55c9498c7d-ll287" event={"ID":"aee60f09-ffd8-4733-b965-a572468b235c","Type":"ContainerStarted","Data":"f8ed8cdc1109642e433eeab077d2bf53f570626302ec645bf9e54a33ac570feb"}
Aug 05 17:36:27.923449 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:36:27.923408    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-27640-predictor-799cfccbdc-v8s6m" podUID="2dbefbae-0497-4396-ab82-abcd1a08abc2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.37:8080: connect: connection refused"
Aug 05 17:36:27.924579 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:36:27.924557    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-27640-predictor-55c9498c7d-ll287" podUID="aee60f09-ffd8-4733-b965-a572468b235c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.36:8080: connect: connection refused"
Aug 05 17:36:27.943115 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:36:27.943066    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/success-200-isvc-27640-predictor-55c9498c7d-ll287" podStartSLOduration=1.943052673 podStartE2EDuration="1.943052673s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-05 17:36:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-05 17:36:27.940807557 +0000 UTC m=+1415.308173900" watchObservedRunningTime="2026-08-05 17:36:27.943052673 +0000 UTC m=+1415.310419017"
Aug 05 17:36:28.927490 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:36:28.927445    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-27640-predictor-55c9498c7d-ll287" podUID="aee60f09-ffd8-4733-b965-a572468b235c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.36:8080: connect: connection refused"
Aug 05 17:36:29.222293 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:36:29.222271    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-d9755-predictor-d8c999f56-sg8lf"
Aug 05 17:36:29.319716 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:36:29.319684    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gwl9l\" (UniqueName: \"kubernetes.io/projected/37d7cb85-d235-490e-80dc-9e73cc8cfa29-kube-api-access-gwl9l\") pod \"37d7cb85-d235-490e-80dc-9e73cc8cfa29\" (UID: \"37d7cb85-d235-490e-80dc-9e73cc8cfa29\") "
Aug 05 17:36:29.319903 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:36:29.319728    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/37d7cb85-d235-490e-80dc-9e73cc8cfa29-kube-rbac-proxy-sar-config\") pod \"37d7cb85-d235-490e-80dc-9e73cc8cfa29\" (UID: \"37d7cb85-d235-490e-80dc-9e73cc8cfa29\") "
Aug 05 17:36:29.319903 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:36:29.319760    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/37d7cb85-d235-490e-80dc-9e73cc8cfa29-proxy-tls\") pod \"37d7cb85-d235-490e-80dc-9e73cc8cfa29\" (UID: \"37d7cb85-d235-490e-80dc-9e73cc8cfa29\") "
Aug 05 17:36:29.320091 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:36:29.320067    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/37d7cb85-d235-490e-80dc-9e73cc8cfa29-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "37d7cb85-d235-490e-80dc-9e73cc8cfa29" (UID: "37d7cb85-d235-490e-80dc-9e73cc8cfa29"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 05 17:36:29.321826 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:36:29.321788    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/37d7cb85-d235-490e-80dc-9e73cc8cfa29-kube-api-access-gwl9l" (OuterVolumeSpecName: "kube-api-access-gwl9l") pod "37d7cb85-d235-490e-80dc-9e73cc8cfa29" (UID: "37d7cb85-d235-490e-80dc-9e73cc8cfa29"). InnerVolumeSpecName "kube-api-access-gwl9l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 05 17:36:29.321918 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:36:29.321880    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/37d7cb85-d235-490e-80dc-9e73cc8cfa29-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "37d7cb85-d235-490e-80dc-9e73cc8cfa29" (UID: "37d7cb85-d235-490e-80dc-9e73cc8cfa29"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 05 17:36:29.421183 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:36:29.421154    2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gwl9l\" (UniqueName: \"kubernetes.io/projected/37d7cb85-d235-490e-80dc-9e73cc8cfa29-kube-api-access-gwl9l\") on node \"ip-10-0-135-2.ec2.internal\" DevicePath \"\""
Aug 05 17:36:29.421183 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:36:29.421179    2571 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/37d7cb85-d235-490e-80dc-9e73cc8cfa29-kube-rbac-proxy-sar-config\") on node \"ip-10-0-135-2.ec2.internal\" DevicePath \"\""
Aug 05 17:36:29.421349 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:36:29.421190    2571 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/37d7cb85-d235-490e-80dc-9e73cc8cfa29-proxy-tls\") on node \"ip-10-0-135-2.ec2.internal\" DevicePath \"\""
Aug 05 17:36:29.601180 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:36:29.601146    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-d9755-predictor-78b5549d57-wh2st" podUID="0fe55532-2a2b-41c1-92d4-dc7f0e201ccc" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.33:8643/healthz\": dial tcp 10.134.0.33:8643: connect: connection refused"
Aug 05 17:36:29.606313 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:36:29.606288    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-d9755-predictor-78b5549d57-wh2st" podUID="0fe55532-2a2b-41c1-92d4-dc7f0e201ccc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused"
Aug 05 17:36:29.866040 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:36:29.866015    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-d9755-predictor-78b5549d57-wh2st"
Aug 05 17:36:29.933064 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:36:29.933033    2571 generic.go:356] "Generic (PLEG): container finished" podID="37d7cb85-d235-490e-80dc-9e73cc8cfa29" containerID="45dcb43937a43b7375beaf5bb4431800e574349b409baa0252e629af9021c9c3" exitCode=0
Aug 05 17:36:29.933460 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:36:29.933100    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-d9755-predictor-d8c999f56-sg8lf"
Aug 05 17:36:29.933460 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:36:29.933118    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-d9755-predictor-d8c999f56-sg8lf" event={"ID":"37d7cb85-d235-490e-80dc-9e73cc8cfa29","Type":"ContainerDied","Data":"45dcb43937a43b7375beaf5bb4431800e574349b409baa0252e629af9021c9c3"}
Aug 05 17:36:29.933460 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:36:29.933179    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-d9755-predictor-d8c999f56-sg8lf" event={"ID":"37d7cb85-d235-490e-80dc-9e73cc8cfa29","Type":"ContainerDied","Data":"a735d79c24d914a8bcb26dcf43ebaa1127210f8915e1b44c5f76c49a406f2229"}
Aug 05 17:36:29.933460 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:36:29.933218    2571 scope.go:117] "RemoveContainer" containerID="eadae4a9c5e1fd7118df87e82da719a3b59c612a20d54fd0ec8e4f678f46c384"
Aug 05 17:36:29.934943 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:36:29.934903    2571 generic.go:356] "Generic (PLEG): container finished" podID="0fe55532-2a2b-41c1-92d4-dc7f0e201ccc" containerID="29331d6ac473f82798132463a470532d5a7d87b5e0d210b81ffbc82b114172dc" exitCode=0
Aug 05 17:36:29.934943 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:36:29.934934    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-d9755-predictor-78b5549d57-wh2st" event={"ID":"0fe55532-2a2b-41c1-92d4-dc7f0e201ccc","Type":"ContainerDied","Data":"29331d6ac473f82798132463a470532d5a7d87b5e0d210b81ffbc82b114172dc"}
Aug 05 17:36:29.935140 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:36:29.934946    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-d9755-predictor-78b5549d57-wh2st"
Aug 05 17:36:29.935140 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:36:29.934986    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-d9755-predictor-78b5549d57-wh2st" event={"ID":"0fe55532-2a2b-41c1-92d4-dc7f0e201ccc","Type":"ContainerDied","Data":"bb1f769057dec9aab1d63b5acce65178ce9fbb94f7243f2376db548bf4b0505a"}
Aug 05 17:36:29.941867 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:36:29.941654    2571 scope.go:117] "RemoveContainer" containerID="45dcb43937a43b7375beaf5bb4431800e574349b409baa0252e629af9021c9c3"
Aug 05 17:36:29.949370 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:36:29.949343    2571 scope.go:117] "RemoveContainer" containerID="eadae4a9c5e1fd7118df87e82da719a3b59c612a20d54fd0ec8e4f678f46c384"
Aug 05 17:36:29.949641 ip-10-0-135-2 kubenswrapper[2571]: E0805 17:36:29.949622    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eadae4a9c5e1fd7118df87e82da719a3b59c612a20d54fd0ec8e4f678f46c384\": container with ID starting with eadae4a9c5e1fd7118df87e82da719a3b59c612a20d54fd0ec8e4f678f46c384 not found: ID does not exist" containerID="eadae4a9c5e1fd7118df87e82da719a3b59c612a20d54fd0ec8e4f678f46c384"
Aug 05 17:36:29.949701 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:36:29.949649    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eadae4a9c5e1fd7118df87e82da719a3b59c612a20d54fd0ec8e4f678f46c384"} err="failed to get container status \"eadae4a9c5e1fd7118df87e82da719a3b59c612a20d54fd0ec8e4f678f46c384\": rpc error: code = NotFound desc = could not find container \"eadae4a9c5e1fd7118df87e82da719a3b59c612a20d54fd0ec8e4f678f46c384\": container with ID starting with eadae4a9c5e1fd7118df87e82da719a3b59c612a20d54fd0ec8e4f678f46c384 not found: ID does not exist"
Aug 05 17:36:29.949701 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:36:29.949665    2571 scope.go:117] "RemoveContainer" containerID="45dcb43937a43b7375beaf5bb4431800e574349b409baa0252e629af9021c9c3"
Aug 05 17:36:29.949953 ip-10-0-135-2 kubenswrapper[2571]: E0805 17:36:29.949935    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"45dcb43937a43b7375beaf5bb4431800e574349b409baa0252e629af9021c9c3\": container with ID starting with 45dcb43937a43b7375beaf5bb4431800e574349b409baa0252e629af9021c9c3 not found: ID does not exist" containerID="45dcb43937a43b7375beaf5bb4431800e574349b409baa0252e629af9021c9c3"
Aug 05 17:36:29.950011 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:36:29.949960    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"45dcb43937a43b7375beaf5bb4431800e574349b409baa0252e629af9021c9c3"} err="failed to get container status \"45dcb43937a43b7375beaf5bb4431800e574349b409baa0252e629af9021c9c3\": rpc error: code = NotFound desc = could not find container \"45dcb43937a43b7375beaf5bb4431800e574349b409baa0252e629af9021c9c3\": container with ID starting with 45dcb43937a43b7375beaf5bb4431800e574349b409baa0252e629af9021c9c3 not found: ID does not exist"
Aug 05 17:36:29.950011 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:36:29.949975    2571 scope.go:117] "RemoveContainer" containerID="f10060501d672f4cbd8ae351765abce050c0e4643ae67970d41545f3de958e3c"
Aug 05 17:36:29.957029 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:36:29.957013    2571 scope.go:117] "RemoveContainer" containerID="29331d6ac473f82798132463a470532d5a7d87b5e0d210b81ffbc82b114172dc"
Aug 05 17:36:29.963612 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:36:29.963594    2571 scope.go:117] "RemoveContainer" containerID="f10060501d672f4cbd8ae351765abce050c0e4643ae67970d41545f3de958e3c"
Aug 05 17:36:29.963825 ip-10-0-135-2 kubenswrapper[2571]: E0805 17:36:29.963807    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f10060501d672f4cbd8ae351765abce050c0e4643ae67970d41545f3de958e3c\": container with ID starting with f10060501d672f4cbd8ae351765abce050c0e4643ae67970d41545f3de958e3c not found: ID does not exist" containerID="f10060501d672f4cbd8ae351765abce050c0e4643ae67970d41545f3de958e3c"
Aug 05 17:36:29.963870 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:36:29.963832    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f10060501d672f4cbd8ae351765abce050c0e4643ae67970d41545f3de958e3c"} err="failed to get container status \"f10060501d672f4cbd8ae351765abce050c0e4643ae67970d41545f3de958e3c\": rpc error: code = NotFound desc = could not find container \"f10060501d672f4cbd8ae351765abce050c0e4643ae67970d41545f3de958e3c\": container with ID starting with f10060501d672f4cbd8ae351765abce050c0e4643ae67970d41545f3de958e3c not found: ID does not exist"
Aug 05 17:36:29.963870 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:36:29.963854    2571 scope.go:117] "RemoveContainer" containerID="29331d6ac473f82798132463a470532d5a7d87b5e0d210b81ffbc82b114172dc"
Aug 05 17:36:29.964070 ip-10-0-135-2 kubenswrapper[2571]: E0805 17:36:29.964053    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"29331d6ac473f82798132463a470532d5a7d87b5e0d210b81ffbc82b114172dc\": container with ID starting with 29331d6ac473f82798132463a470532d5a7d87b5e0d210b81ffbc82b114172dc not found: ID does not exist" containerID="29331d6ac473f82798132463a470532d5a7d87b5e0d210b81ffbc82b114172dc"
Aug 05 17:36:29.964115 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:36:29.964074    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"29331d6ac473f82798132463a470532d5a7d87b5e0d210b81ffbc82b114172dc"} err="failed to get container status \"29331d6ac473f82798132463a470532d5a7d87b5e0d210b81ffbc82b114172dc\": rpc error: code = NotFound desc = could not find container \"29331d6ac473f82798132463a470532d5a7d87b5e0d210b81ffbc82b114172dc\": container with ID starting with 29331d6ac473f82798132463a470532d5a7d87b5e0d210b81ffbc82b114172dc not found: ID does not exist"
Aug 05 17:36:29.979063 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:36:29.979039    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-d9755-predictor-d8c999f56-sg8lf"]
Aug 05 17:36:29.983227 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:36:29.983208    2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-d9755-predictor-d8c999f56-sg8lf"]
Aug 05 17:36:30.025690 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:36:30.025667    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-v2wtg\" (UniqueName: \"kubernetes.io/projected/0fe55532-2a2b-41c1-92d4-dc7f0e201ccc-kube-api-access-v2wtg\") pod \"0fe55532-2a2b-41c1-92d4-dc7f0e201ccc\" (UID: \"0fe55532-2a2b-41c1-92d4-dc7f0e201ccc\") "
Aug 05 17:36:30.025785 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:36:30.025725    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0fe55532-2a2b-41c1-92d4-dc7f0e201ccc-proxy-tls\") pod \"0fe55532-2a2b-41c1-92d4-dc7f0e201ccc\" (UID: \"0fe55532-2a2b-41c1-92d4-dc7f0e201ccc\") "
Aug 05 17:36:30.025785 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:36:30.025747    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0fe55532-2a2b-41c1-92d4-dc7f0e201ccc-kube-rbac-proxy-sar-config\") pod \"0fe55532-2a2b-41c1-92d4-dc7f0e201ccc\" (UID: \"0fe55532-2a2b-41c1-92d4-dc7f0e201ccc\") "
Aug 05 17:36:30.026161 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:36:30.026141    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0fe55532-2a2b-41c1-92d4-dc7f0e201ccc-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "0fe55532-2a2b-41c1-92d4-dc7f0e201ccc" (UID: "0fe55532-2a2b-41c1-92d4-dc7f0e201ccc"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 05 17:36:30.027650 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:36:30.027621    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0fe55532-2a2b-41c1-92d4-dc7f0e201ccc-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "0fe55532-2a2b-41c1-92d4-dc7f0e201ccc" (UID: "0fe55532-2a2b-41c1-92d4-dc7f0e201ccc"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 05 17:36:30.027722 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:36:30.027650    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0fe55532-2a2b-41c1-92d4-dc7f0e201ccc-kube-api-access-v2wtg" (OuterVolumeSpecName: "kube-api-access-v2wtg") pod "0fe55532-2a2b-41c1-92d4-dc7f0e201ccc" (UID: "0fe55532-2a2b-41c1-92d4-dc7f0e201ccc"). InnerVolumeSpecName "kube-api-access-v2wtg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 05 17:36:30.127313 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:36:30.127229    2571 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0fe55532-2a2b-41c1-92d4-dc7f0e201ccc-proxy-tls\") on node \"ip-10-0-135-2.ec2.internal\" DevicePath \"\""
Aug 05 17:36:30.127313 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:36:30.127259    2571 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0fe55532-2a2b-41c1-92d4-dc7f0e201ccc-kube-rbac-proxy-sar-config\") on node \"ip-10-0-135-2.ec2.internal\" DevicePath \"\""
Aug 05 17:36:30.127313 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:36:30.127272    2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v2wtg\" (UniqueName: \"kubernetes.io/projected/0fe55532-2a2b-41c1-92d4-dc7f0e201ccc-kube-api-access-v2wtg\") on node \"ip-10-0-135-2.ec2.internal\" DevicePath \"\""
Aug 05 17:36:30.259842 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:36:30.259815    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-d9755-predictor-78b5549d57-wh2st"]
Aug 05 17:36:30.263253 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:36:30.263226    2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-d9755-predictor-78b5549d57-wh2st"]
Aug 05 17:36:31.136553 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:36:31.136522    2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0fe55532-2a2b-41c1-92d4-dc7f0e201ccc" path="/var/lib/kubelet/pods/0fe55532-2a2b-41c1-92d4-dc7f0e201ccc/volumes"
Aug 05 17:36:31.136917 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:36:31.136909    2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="37d7cb85-d235-490e-80dc-9e73cc8cfa29" path="/var/lib/kubelet/pods/37d7cb85-d235-490e-80dc-9e73cc8cfa29/volumes"
Aug 05 17:36:32.927254 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:36:32.927223    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-27640-predictor-799cfccbdc-v8s6m"
Aug 05 17:36:32.927691 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:36:32.927662    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-27640-predictor-799cfccbdc-v8s6m" podUID="2dbefbae-0497-4396-ab82-abcd1a08abc2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.37:8080: connect: connection refused"
Aug 05 17:36:33.932468 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:36:33.932440    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-27640-predictor-55c9498c7d-ll287"
Aug 05 17:36:33.932989 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:36:33.932961    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-27640-predictor-55c9498c7d-ll287" podUID="aee60f09-ffd8-4733-b965-a572468b235c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.36:8080: connect: connection refused"
Aug 05 17:36:36.408898 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:36:36.408863    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-fba55-predictor-77c4dfc47d-j8skd"]
Aug 05 17:36:36.409433 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:36:36.409155    2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-fba55-predictor-77c4dfc47d-j8skd" podUID="a489ce9e-f83f-47f4-8faf-2358a9349294" containerName="kserve-container" containerID="cri-o://520b1b020911d8435020fb2e94c66b0c29e0f32c33b92653ba9ea7d2c7c9103d" gracePeriod=30
Aug 05 17:36:36.409433 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:36:36.409345    2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-fba55-predictor-77c4dfc47d-j8skd" podUID="a489ce9e-f83f-47f4-8faf-2358a9349294" containerName="kube-rbac-proxy" containerID="cri-o://f611045c7e6a80b8faa32f5a458b7330ab0274b615a572a70be008960c02ad2e" gracePeriod=30
Aug 05 17:36:36.490529 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:36:36.490498    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-8dddc-predictor-66fd6cdf4-znnzz"]
Aug 05 17:36:36.490936 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:36:36.490919    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0fe55532-2a2b-41c1-92d4-dc7f0e201ccc" containerName="kube-rbac-proxy"
Aug 05 17:36:36.491027 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:36:36.490939    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="0fe55532-2a2b-41c1-92d4-dc7f0e201ccc" containerName="kube-rbac-proxy"
Aug 05 17:36:36.491027 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:36:36.490956    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0fe55532-2a2b-41c1-92d4-dc7f0e201ccc" containerName="kserve-container"
Aug 05 17:36:36.491027 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:36:36.490963    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="0fe55532-2a2b-41c1-92d4-dc7f0e201ccc" containerName="kserve-container"
Aug 05 17:36:36.491027 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:36:36.491000    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="37d7cb85-d235-490e-80dc-9e73cc8cfa29" containerName="kube-rbac-proxy"
Aug 05 17:36:36.491027 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:36:36.491009    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="37d7cb85-d235-490e-80dc-9e73cc8cfa29" containerName="kube-rbac-proxy"
Aug 05 17:36:36.491027 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:36:36.491020    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="37d7cb85-d235-490e-80dc-9e73cc8cfa29" containerName="kserve-container"
Aug 05 17:36:36.491027 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:36:36.491029    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="37d7cb85-d235-490e-80dc-9e73cc8cfa29" containerName="kserve-container"
Aug 05 17:36:36.491408 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:36:36.491111    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="37d7cb85-d235-490e-80dc-9e73cc8cfa29" containerName="kserve-container"
Aug 05 17:36:36.491408 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:36:36.491124    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="0fe55532-2a2b-41c1-92d4-dc7f0e201ccc" containerName="kserve-container"
Aug 05 17:36:36.491408 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:36:36.491137    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="0fe55532-2a2b-41c1-92d4-dc7f0e201ccc" containerName="kube-rbac-proxy"
Aug 05 17:36:36.491408 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:36:36.491148    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="37d7cb85-d235-490e-80dc-9e73cc8cfa29" containerName="kube-rbac-proxy"
Aug 05 17:36:36.495459 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:36:36.495429    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-8dddc-predictor-66fd6cdf4-znnzz"
Aug 05 17:36:36.497800 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:36:36.497777    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-8dddc-predictor-serving-cert\""
Aug 05 17:36:36.497908 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:36:36.497818    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-8dddc-kube-rbac-proxy-sar-config\""
Aug 05 17:36:36.505811 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:36:36.505792    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-8dddc-predictor-66fd6cdf4-znnzz"]
Aug 05 17:36:36.547684 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:36:36.547653    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-fba55-predictor-676475b57f-fljvk"]
Aug 05 17:36:36.547938 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:36:36.547912    2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-fba55-predictor-676475b57f-fljvk" podUID="366121e0-36b3-4704-a224-ee0e872f94e4" containerName="kserve-container" containerID="cri-o://c08f54086eca55ccab5f173c65e9745cda91215666caf391c6e83a19be2a89fc" gracePeriod=30
Aug 05 17:36:36.548032 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:36:36.548001    2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-fba55-predictor-676475b57f-fljvk" podUID="366121e0-36b3-4704-a224-ee0e872f94e4" containerName="kube-rbac-proxy" containerID="cri-o://c07e1e66686daa816d2114bef7210b4e4abda360266cca9d86672933983ba610" gracePeriod=30
Aug 05 17:36:36.678604 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:36:36.678574    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fbe776c3-891d-4fbf-acf1-6f4b72abda6c-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-8dddc-predictor-66fd6cdf4-znnzz\" (UID: \"fbe776c3-891d-4fbf-acf1-6f4b72abda6c\") " pod="kserve-ci-e2e-test/success-200-isvc-8dddc-predictor-66fd6cdf4-znnzz"
Aug 05 17:36:36.678752 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:36:36.678618    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fbe776c3-891d-4fbf-acf1-6f4b72abda6c-proxy-tls\") pod \"success-200-isvc-8dddc-predictor-66fd6cdf4-znnzz\" (UID: \"fbe776c3-891d-4fbf-acf1-6f4b72abda6c\") " pod="kserve-ci-e2e-test/success-200-isvc-8dddc-predictor-66fd6cdf4-znnzz"
Aug 05 17:36:36.678752 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:36:36.678664    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s7h77\" (UniqueName: \"kubernetes.io/projected/fbe776c3-891d-4fbf-acf1-6f4b72abda6c-kube-api-access-s7h77\") pod \"success-200-isvc-8dddc-predictor-66fd6cdf4-znnzz\" (UID: \"fbe776c3-891d-4fbf-acf1-6f4b72abda6c\") " pod="kserve-ci-e2e-test/success-200-isvc-8dddc-predictor-66fd6cdf4-znnzz"
Aug 05 17:36:36.710798 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:36:36.710772    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-8dddc-predictor-5558b5787-9qrcb"]
Aug 05 17:36:36.714287 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:36:36.714270    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-8dddc-predictor-5558b5787-9qrcb"
Aug 05 17:36:36.717348 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:36:36.717329    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-8dddc-predictor-serving-cert\""
Aug 05 17:36:36.717483 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:36:36.717466    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-8dddc-kube-rbac-proxy-sar-config\""
Aug 05 17:36:36.731645 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:36:36.731627    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-8dddc-predictor-5558b5787-9qrcb"]
Aug 05 17:36:36.780038 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:36:36.780011    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fbe776c3-891d-4fbf-acf1-6f4b72abda6c-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-8dddc-predictor-66fd6cdf4-znnzz\" (UID: \"fbe776c3-891d-4fbf-acf1-6f4b72abda6c\") " pod="kserve-ci-e2e-test/success-200-isvc-8dddc-predictor-66fd6cdf4-znnzz"
Aug 05 17:36:36.780184 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:36:36.780058    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fbe776c3-891d-4fbf-acf1-6f4b72abda6c-proxy-tls\") pod \"success-200-isvc-8dddc-predictor-66fd6cdf4-znnzz\" (UID: \"fbe776c3-891d-4fbf-acf1-6f4b72abda6c\") " pod="kserve-ci-e2e-test/success-200-isvc-8dddc-predictor-66fd6cdf4-znnzz"
Aug 05 17:36:36.780184 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:36:36.780112    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s7h77\" (UniqueName: \"kubernetes.io/projected/fbe776c3-891d-4fbf-acf1-6f4b72abda6c-kube-api-access-s7h77\") pod \"success-200-isvc-8dddc-predictor-66fd6cdf4-znnzz\" (UID: \"fbe776c3-891d-4fbf-acf1-6f4b72abda6c\") " pod="kserve-ci-e2e-test/success-200-isvc-8dddc-predictor-66fd6cdf4-znnzz"
Aug 05 17:36:36.780740 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:36:36.780717    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fbe776c3-891d-4fbf-acf1-6f4b72abda6c-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-8dddc-predictor-66fd6cdf4-znnzz\" (UID: \"fbe776c3-891d-4fbf-acf1-6f4b72abda6c\") " pod="kserve-ci-e2e-test/success-200-isvc-8dddc-predictor-66fd6cdf4-znnzz"
Aug 05 17:36:36.782485 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:36:36.782464    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fbe776c3-891d-4fbf-acf1-6f4b72abda6c-proxy-tls\") pod \"success-200-isvc-8dddc-predictor-66fd6cdf4-znnzz\" (UID: \"fbe776c3-891d-4fbf-acf1-6f4b72abda6c\") " pod="kserve-ci-e2e-test/success-200-isvc-8dddc-predictor-66fd6cdf4-znnzz"
Aug 05 17:36:36.789533 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:36:36.789510    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s7h77\" (UniqueName: \"kubernetes.io/projected/fbe776c3-891d-4fbf-acf1-6f4b72abda6c-kube-api-access-s7h77\") pod \"success-200-isvc-8dddc-predictor-66fd6cdf4-znnzz\" (UID: \"fbe776c3-891d-4fbf-acf1-6f4b72abda6c\") " pod="kserve-ci-e2e-test/success-200-isvc-8dddc-predictor-66fd6cdf4-znnzz"
Aug 05 17:36:36.805435 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:36:36.805415    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-8dddc-predictor-66fd6cdf4-znnzz"
Aug 05 17:36:36.880981 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:36:36.880949    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e8b2fd82-37b5-457e-ae17-47fadcce2a5e-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-8dddc-predictor-5558b5787-9qrcb\" (UID: \"e8b2fd82-37b5-457e-ae17-47fadcce2a5e\") " pod="kserve-ci-e2e-test/error-404-isvc-8dddc-predictor-5558b5787-9qrcb"
Aug 05 17:36:36.881121 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:36:36.881008    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e8b2fd82-37b5-457e-ae17-47fadcce2a5e-proxy-tls\") pod \"error-404-isvc-8dddc-predictor-5558b5787-9qrcb\" (UID: \"e8b2fd82-37b5-457e-ae17-47fadcce2a5e\") " pod="kserve-ci-e2e-test/error-404-isvc-8dddc-predictor-5558b5787-9qrcb"
Aug 05 17:36:36.881121 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:36:36.881043    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x6x7k\" (UniqueName: \"kubernetes.io/projected/e8b2fd82-37b5-457e-ae17-47fadcce2a5e-kube-api-access-x6x7k\") pod \"error-404-isvc-8dddc-predictor-5558b5787-9qrcb\" (UID: \"e8b2fd82-37b5-457e-ae17-47fadcce2a5e\") " pod="kserve-ci-e2e-test/error-404-isvc-8dddc-predictor-5558b5787-9qrcb"
Aug 05 17:36:36.935774 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:36:36.935751    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-8dddc-predictor-66fd6cdf4-znnzz"]
Aug 05 17:36:36.937980 ip-10-0-135-2 kubenswrapper[2571]: W0805 17:36:36.937951    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfbe776c3_891d_4fbf_acf1_6f4b72abda6c.slice/crio-102d2e33667cd815bacb069a3193c3a6de8a25e09e56d0dfe0d529b8ef9c2a07 WatchSource:0}: Error finding container 102d2e33667cd815bacb069a3193c3a6de8a25e09e56d0dfe0d529b8ef9c2a07: Status 404 returned error can't find the container with id 102d2e33667cd815bacb069a3193c3a6de8a25e09e56d0dfe0d529b8ef9c2a07
Aug 05 17:36:36.970168 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:36:36.970139    2571 generic.go:356] "Generic (PLEG): container finished" podID="a489ce9e-f83f-47f4-8faf-2358a9349294" containerID="f611045c7e6a80b8faa32f5a458b7330ab0274b615a572a70be008960c02ad2e" exitCode=2
Aug 05 17:36:36.970278 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:36:36.970206    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-fba55-predictor-77c4dfc47d-j8skd" event={"ID":"a489ce9e-f83f-47f4-8faf-2358a9349294","Type":"ContainerDied","Data":"f611045c7e6a80b8faa32f5a458b7330ab0274b615a572a70be008960c02ad2e"}
Aug 05 17:36:36.971396 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:36:36.971351    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-8dddc-predictor-66fd6cdf4-znnzz" event={"ID":"fbe776c3-891d-4fbf-acf1-6f4b72abda6c","Type":"ContainerStarted","Data":"102d2e33667cd815bacb069a3193c3a6de8a25e09e56d0dfe0d529b8ef9c2a07"}
Aug 05 17:36:36.973090 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:36:36.973064    2571 generic.go:356] "Generic (PLEG): container finished" podID="366121e0-36b3-4704-a224-ee0e872f94e4" containerID="c07e1e66686daa816d2114bef7210b4e4abda360266cca9d86672933983ba610" exitCode=2
Aug 05 17:36:36.973181 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:36:36.973087    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-fba55-predictor-676475b57f-fljvk" event={"ID":"366121e0-36b3-4704-a224-ee0e872f94e4","Type":"ContainerDied","Data":"c07e1e66686daa816d2114bef7210b4e4abda360266cca9d86672933983ba610"}
Aug 05 17:36:36.983171 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:36:36.983144    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e8b2fd82-37b5-457e-ae17-47fadcce2a5e-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-8dddc-predictor-5558b5787-9qrcb\" (UID: \"e8b2fd82-37b5-457e-ae17-47fadcce2a5e\") " pod="kserve-ci-e2e-test/error-404-isvc-8dddc-predictor-5558b5787-9qrcb"
Aug 05 17:36:36.983258 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:36:36.983223    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e8b2fd82-37b5-457e-ae17-47fadcce2a5e-proxy-tls\") pod \"error-404-isvc-8dddc-predictor-5558b5787-9qrcb\" (UID: \"e8b2fd82-37b5-457e-ae17-47fadcce2a5e\") " pod="kserve-ci-e2e-test/error-404-isvc-8dddc-predictor-5558b5787-9qrcb"
Aug 05 17:36:36.983312 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:36:36.983265    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x6x7k\" (UniqueName: \"kubernetes.io/projected/e8b2fd82-37b5-457e-ae17-47fadcce2a5e-kube-api-access-x6x7k\") pod \"error-404-isvc-8dddc-predictor-5558b5787-9qrcb\" (UID: \"e8b2fd82-37b5-457e-ae17-47fadcce2a5e\") " pod="kserve-ci-e2e-test/error-404-isvc-8dddc-predictor-5558b5787-9qrcb"
Aug 05 17:36:36.986943 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:36:36.986914    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e8b2fd82-37b5-457e-ae17-47fadcce2a5e-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-8dddc-predictor-5558b5787-9qrcb\" (UID: \"e8b2fd82-37b5-457e-ae17-47fadcce2a5e\") " pod="kserve-ci-e2e-test/error-404-isvc-8dddc-predictor-5558b5787-9qrcb"
Aug 05 17:36:36.989475 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:36:36.989454    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e8b2fd82-37b5-457e-ae17-47fadcce2a5e-proxy-tls\") pod \"error-404-isvc-8dddc-predictor-5558b5787-9qrcb\" (UID: \"e8b2fd82-37b5-457e-ae17-47fadcce2a5e\") " pod="kserve-ci-e2e-test/error-404-isvc-8dddc-predictor-5558b5787-9qrcb"
Aug 05 17:36:36.991678 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:36:36.991659    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x6x7k\" (UniqueName: \"kubernetes.io/projected/e8b2fd82-37b5-457e-ae17-47fadcce2a5e-kube-api-access-x6x7k\") pod \"error-404-isvc-8dddc-predictor-5558b5787-9qrcb\" (UID: \"e8b2fd82-37b5-457e-ae17-47fadcce2a5e\") " pod="kserve-ci-e2e-test/error-404-isvc-8dddc-predictor-5558b5787-9qrcb"
Aug 05 17:36:37.023947 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:36:37.023928    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-8dddc-predictor-5558b5787-9qrcb"
Aug 05 17:36:37.164265 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:36:37.164238    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-8dddc-predictor-5558b5787-9qrcb"]
Aug 05 17:36:37.166193 ip-10-0-135-2 kubenswrapper[2571]: W0805 17:36:37.166167    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode8b2fd82_37b5_457e_ae17_47fadcce2a5e.slice/crio-930c1ede5d90de21f97194769abe658b580e03337b0f5f8acf3613c0b45f47fe WatchSource:0}: Error finding container 930c1ede5d90de21f97194769abe658b580e03337b0f5f8acf3613c0b45f47fe: Status 404 returned error can't find the container with id 930c1ede5d90de21f97194769abe658b580e03337b0f5f8acf3613c0b45f47fe
Aug 05 17:36:37.979539 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:36:37.979491    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-8dddc-predictor-66fd6cdf4-znnzz" event={"ID":"fbe776c3-891d-4fbf-acf1-6f4b72abda6c","Type":"ContainerStarted","Data":"dbcab6cedb0bc16ca8d5c649355fa7c0f97ba79cab24a764b3330d51f5f57f9e"}
Aug 05 17:36:37.980020 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:36:37.979603    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-8dddc-predictor-66fd6cdf4-znnzz" event={"ID":"fbe776c3-891d-4fbf-acf1-6f4b72abda6c","Type":"ContainerStarted","Data":"e8d9d961405a7d16ebafea8a944e62ba9f391abbd3de5956e28a657d4ef6a6dd"}
Aug 05 17:36:37.980020 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:36:37.979641    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-8dddc-predictor-66fd6cdf4-znnzz"
Aug 05 17:36:37.980020 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:36:37.979676    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-8dddc-predictor-66fd6cdf4-znnzz"
Aug 05 17:36:37.981543 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:36:37.981507    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-8dddc-predictor-66fd6cdf4-znnzz" podUID="fbe776c3-891d-4fbf-acf1-6f4b72abda6c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.38:8080: connect: connection refused"
Aug 05 17:36:37.981687 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:36:37.981512    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-8dddc-predictor-5558b5787-9qrcb" event={"ID":"e8b2fd82-37b5-457e-ae17-47fadcce2a5e","Type":"ContainerStarted","Data":"3e2d115f8eafd58031a32787e6293dc6a80c6873300927324f5c22431429125e"}
Aug 05 17:36:37.981687 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:36:37.981607    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-8dddc-predictor-5558b5787-9qrcb"
Aug 05 17:36:37.981687 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:36:37.981636    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-8dddc-predictor-5558b5787-9qrcb" event={"ID":"e8b2fd82-37b5-457e-ae17-47fadcce2a5e","Type":"ContainerStarted","Data":"b96ad862b6b46a99e41b0dd376e33c4bfee3ee23be666681737e863a6cc04422"}
Aug 05 17:36:37.981687 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:36:37.981657    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-8dddc-predictor-5558b5787-9qrcb" event={"ID":"e8b2fd82-37b5-457e-ae17-47fadcce2a5e","Type":"ContainerStarted","Data":"930c1ede5d90de21f97194769abe658b580e03337b0f5f8acf3613c0b45f47fe"}
Aug 05 17:36:37.981687 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:36:37.981684    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-8dddc-predictor-5558b5787-9qrcb"
Aug 05 17:36:37.982746 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:36:37.982724    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-8dddc-predictor-5558b5787-9qrcb" podUID="e8b2fd82-37b5-457e-ae17-47fadcce2a5e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.39:8080: connect: connection refused"
Aug 05 17:36:38.000563 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:36:38.000525    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/success-200-isvc-8dddc-predictor-66fd6cdf4-znnzz" podStartSLOduration=2.000513352 podStartE2EDuration="2.000513352s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-05 17:36:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-05 17:36:37.99884906 +0000 UTC m=+1425.366215416" watchObservedRunningTime="2026-08-05 17:36:38.000513352 +0000 UTC m=+1425.367879705"
Aug 05 17:36:38.019703 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:36:38.019670    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/error-404-isvc-8dddc-predictor-5558b5787-9qrcb" podStartSLOduration=2.019658436 podStartE2EDuration="2.019658436s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-05 17:36:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-05 17:36:38.017679419 +0000 UTC m=+1425.385045763" watchObservedRunningTime="2026-08-05 17:36:38.019658436 +0000 UTC m=+1425.387024778"
Aug 05 17:36:38.648076 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:36:38.648029    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-fba55-predictor-676475b57f-fljvk" podUID="366121e0-36b3-4704-a224-ee0e872f94e4" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.35:8643/healthz\": dial tcp 10.134.0.35:8643: connect: connection refused"
Aug 05 17:36:38.986411 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:36:38.986353    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-8dddc-predictor-66fd6cdf4-znnzz" podUID="fbe776c3-891d-4fbf-acf1-6f4b72abda6c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.38:8080: connect: connection refused"
Aug 05 17:36:38.986758 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:36:38.986415    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-8dddc-predictor-5558b5787-9qrcb" podUID="e8b2fd82-37b5-457e-ae17-47fadcce2a5e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.39:8080: connect: connection refused"
Aug 05 17:36:39.659146 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:36:39.659089    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-fba55-predictor-77c4dfc47d-j8skd" podUID="a489ce9e-f83f-47f4-8faf-2358a9349294" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.34:8643/healthz\": dial tcp 10.134.0.34:8643: connect: connection refused"
Aug 05 17:36:39.764004 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:36:39.763981    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-fba55-predictor-77c4dfc47d-j8skd"
Aug 05 17:36:39.808129 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:36:39.808068    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a489ce9e-f83f-47f4-8faf-2358a9349294-kube-rbac-proxy-sar-config\") pod \"a489ce9e-f83f-47f4-8faf-2358a9349294\" (UID: \"a489ce9e-f83f-47f4-8faf-2358a9349294\") "
Aug 05 17:36:39.808252 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:36:39.808144    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a489ce9e-f83f-47f4-8faf-2358a9349294-proxy-tls\") pod \"a489ce9e-f83f-47f4-8faf-2358a9349294\" (UID: \"a489ce9e-f83f-47f4-8faf-2358a9349294\") "
Aug 05 17:36:39.808293 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:36:39.808252    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ts2wq\" (UniqueName: \"kubernetes.io/projected/a489ce9e-f83f-47f4-8faf-2358a9349294-kube-api-access-ts2wq\") pod \"a489ce9e-f83f-47f4-8faf-2358a9349294\" (UID: \"a489ce9e-f83f-47f4-8faf-2358a9349294\") "
Aug 05 17:36:39.808551 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:36:39.808517    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a489ce9e-f83f-47f4-8faf-2358a9349294-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "a489ce9e-f83f-47f4-8faf-2358a9349294" (UID: "a489ce9e-f83f-47f4-8faf-2358a9349294"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 05 17:36:39.810153 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:36:39.810132    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a489ce9e-f83f-47f4-8faf-2358a9349294-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "a489ce9e-f83f-47f4-8faf-2358a9349294" (UID: "a489ce9e-f83f-47f4-8faf-2358a9349294"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 05 17:36:39.810213 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:36:39.810171    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a489ce9e-f83f-47f4-8faf-2358a9349294-kube-api-access-ts2wq" (OuterVolumeSpecName: "kube-api-access-ts2wq") pod "a489ce9e-f83f-47f4-8faf-2358a9349294" (UID: "a489ce9e-f83f-47f4-8faf-2358a9349294"). InnerVolumeSpecName "kube-api-access-ts2wq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 05 17:36:39.909707 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:36:39.909676    2571 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a489ce9e-f83f-47f4-8faf-2358a9349294-kube-rbac-proxy-sar-config\") on node \"ip-10-0-135-2.ec2.internal\" DevicePath \"\""
Aug 05 17:36:39.909707 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:36:39.909704    2571 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a489ce9e-f83f-47f4-8faf-2358a9349294-proxy-tls\") on node \"ip-10-0-135-2.ec2.internal\" DevicePath \"\""
Aug 05 17:36:39.909707 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:36:39.909715    2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ts2wq\" (UniqueName: \"kubernetes.io/projected/a489ce9e-f83f-47f4-8faf-2358a9349294-kube-api-access-ts2wq\") on node \"ip-10-0-135-2.ec2.internal\" DevicePath \"\""
Aug 05 17:36:39.992299 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:36:39.992265    2571 generic.go:356] "Generic (PLEG): container finished" podID="a489ce9e-f83f-47f4-8faf-2358a9349294" containerID="520b1b020911d8435020fb2e94c66b0c29e0f32c33b92653ba9ea7d2c7c9103d" exitCode=0
Aug 05 17:36:39.992757 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:36:39.992318    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-fba55-predictor-77c4dfc47d-j8skd"
Aug 05 17:36:39.992757 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:36:39.992359    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-fba55-predictor-77c4dfc47d-j8skd" event={"ID":"a489ce9e-f83f-47f4-8faf-2358a9349294","Type":"ContainerDied","Data":"520b1b020911d8435020fb2e94c66b0c29e0f32c33b92653ba9ea7d2c7c9103d"}
Aug 05 17:36:39.992757 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:36:39.992460    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-fba55-predictor-77c4dfc47d-j8skd" event={"ID":"a489ce9e-f83f-47f4-8faf-2358a9349294","Type":"ContainerDied","Data":"87fbc49d7625a4b22baada8f8c3768c5aef5edb86ed6e41b7806915085c050b5"}
Aug 05 17:36:39.992757 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:36:39.992501    2571 scope.go:117] "RemoveContainer" containerID="f611045c7e6a80b8faa32f5a458b7330ab0274b615a572a70be008960c02ad2e"
Aug 05 17:36:40.001093 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:36:40.001077    2571 scope.go:117] "RemoveContainer" containerID="520b1b020911d8435020fb2e94c66b0c29e0f32c33b92653ba9ea7d2c7c9103d"
Aug 05 17:36:40.008326 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:36:40.008310    2571 scope.go:117] "RemoveContainer" containerID="f611045c7e6a80b8faa32f5a458b7330ab0274b615a572a70be008960c02ad2e"
Aug 05 17:36:40.008589 ip-10-0-135-2 kubenswrapper[2571]: E0805 17:36:40.008572    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f611045c7e6a80b8faa32f5a458b7330ab0274b615a572a70be008960c02ad2e\": container with ID starting with f611045c7e6a80b8faa32f5a458b7330ab0274b615a572a70be008960c02ad2e not found: ID does not exist" containerID="f611045c7e6a80b8faa32f5a458b7330ab0274b615a572a70be008960c02ad2e"
Aug 05 17:36:40.008637 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:36:40.008597    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f611045c7e6a80b8faa32f5a458b7330ab0274b615a572a70be008960c02ad2e"} err="failed to get container status \"f611045c7e6a80b8faa32f5a458b7330ab0274b615a572a70be008960c02ad2e\": rpc error: code = NotFound desc = could not find container \"f611045c7e6a80b8faa32f5a458b7330ab0274b615a572a70be008960c02ad2e\": container with ID starting with f611045c7e6a80b8faa32f5a458b7330ab0274b615a572a70be008960c02ad2e not found: ID does not exist"
Aug 05 17:36:40.008637 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:36:40.008613    2571 scope.go:117] "RemoveContainer" containerID="520b1b020911d8435020fb2e94c66b0c29e0f32c33b92653ba9ea7d2c7c9103d"
Aug 05 17:36:40.008806 ip-10-0-135-2 kubenswrapper[2571]: E0805 17:36:40.008792    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"520b1b020911d8435020fb2e94c66b0c29e0f32c33b92653ba9ea7d2c7c9103d\": container with ID starting with 520b1b020911d8435020fb2e94c66b0c29e0f32c33b92653ba9ea7d2c7c9103d not found: ID does not exist" containerID="520b1b020911d8435020fb2e94c66b0c29e0f32c33b92653ba9ea7d2c7c9103d"
Aug 05 17:36:40.008843 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:36:40.008809    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"520b1b020911d8435020fb2e94c66b0c29e0f32c33b92653ba9ea7d2c7c9103d"} err="failed to get container status \"520b1b020911d8435020fb2e94c66b0c29e0f32c33b92653ba9ea7d2c7c9103d\": rpc error: code = NotFound desc = could not find container \"520b1b020911d8435020fb2e94c66b0c29e0f32c33b92653ba9ea7d2c7c9103d\": container with ID starting with 520b1b020911d8435020fb2e94c66b0c29e0f32c33b92653ba9ea7d2c7c9103d not found: ID does not exist"
Aug 05 17:36:40.016805 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:36:40.016783    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-fba55-predictor-77c4dfc47d-j8skd"]
Aug 05 17:36:40.019967 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:36:40.019947    2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-fba55-predictor-77c4dfc47d-j8skd"]
Aug 05 17:36:40.279906 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:36:40.279884    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-fba55-predictor-676475b57f-fljvk"
Aug 05 17:36:40.311620 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:36:40.311596    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/366121e0-36b3-4704-a224-ee0e872f94e4-proxy-tls\") pod \"366121e0-36b3-4704-a224-ee0e872f94e4\" (UID: \"366121e0-36b3-4704-a224-ee0e872f94e4\") "
Aug 05 17:36:40.311744 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:36:40.311648    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/366121e0-36b3-4704-a224-ee0e872f94e4-kube-rbac-proxy-sar-config\") pod \"366121e0-36b3-4704-a224-ee0e872f94e4\" (UID: \"366121e0-36b3-4704-a224-ee0e872f94e4\") "
Aug 05 17:36:40.311744 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:36:40.311719    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-knb6q\" (UniqueName: \"kubernetes.io/projected/366121e0-36b3-4704-a224-ee0e872f94e4-kube-api-access-knb6q\") pod \"366121e0-36b3-4704-a224-ee0e872f94e4\" (UID: \"366121e0-36b3-4704-a224-ee0e872f94e4\") "
Aug 05 17:36:40.312067 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:36:40.312039    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/366121e0-36b3-4704-a224-ee0e872f94e4-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "366121e0-36b3-4704-a224-ee0e872f94e4" (UID: "366121e0-36b3-4704-a224-ee0e872f94e4"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 05 17:36:40.313619 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:36:40.313597    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/366121e0-36b3-4704-a224-ee0e872f94e4-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "366121e0-36b3-4704-a224-ee0e872f94e4" (UID: "366121e0-36b3-4704-a224-ee0e872f94e4"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 05 17:36:40.313774 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:36:40.313757    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/366121e0-36b3-4704-a224-ee0e872f94e4-kube-api-access-knb6q" (OuterVolumeSpecName: "kube-api-access-knb6q") pod "366121e0-36b3-4704-a224-ee0e872f94e4" (UID: "366121e0-36b3-4704-a224-ee0e872f94e4"). InnerVolumeSpecName "kube-api-access-knb6q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 05 17:36:40.413191 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:36:40.413119    2571 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/366121e0-36b3-4704-a224-ee0e872f94e4-kube-rbac-proxy-sar-config\") on node \"ip-10-0-135-2.ec2.internal\" DevicePath \"\""
Aug 05 17:36:40.413191 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:36:40.413144    2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-knb6q\" (UniqueName: \"kubernetes.io/projected/366121e0-36b3-4704-a224-ee0e872f94e4-kube-api-access-knb6q\") on node \"ip-10-0-135-2.ec2.internal\" DevicePath \"\""
Aug 05 17:36:40.413191 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:36:40.413158    2571 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/366121e0-36b3-4704-a224-ee0e872f94e4-proxy-tls\") on node \"ip-10-0-135-2.ec2.internal\" DevicePath \"\""
Aug 05 17:36:40.998962 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:36:40.998925    2571 generic.go:356] "Generic (PLEG): container finished" podID="366121e0-36b3-4704-a224-ee0e872f94e4" containerID="c08f54086eca55ccab5f173c65e9745cda91215666caf391c6e83a19be2a89fc" exitCode=0
Aug 05 17:36:40.999348 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:36:40.998983    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-fba55-predictor-676475b57f-fljvk"
Aug 05 17:36:40.999348 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:36:40.998995    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-fba55-predictor-676475b57f-fljvk" event={"ID":"366121e0-36b3-4704-a224-ee0e872f94e4","Type":"ContainerDied","Data":"c08f54086eca55ccab5f173c65e9745cda91215666caf391c6e83a19be2a89fc"}
Aug 05 17:36:40.999348 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:36:40.999057    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-fba55-predictor-676475b57f-fljvk" event={"ID":"366121e0-36b3-4704-a224-ee0e872f94e4","Type":"ContainerDied","Data":"2512c28dd4312702a0bf20ed0f6c158445cb4b91d3500e2faa4922e8e8d11aeb"}
Aug 05 17:36:40.999348 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:36:40.999085    2571 scope.go:117] "RemoveContainer" containerID="c07e1e66686daa816d2114bef7210b4e4abda360266cca9d86672933983ba610"
Aug 05 17:36:41.008290 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:36:41.008272    2571 scope.go:117] "RemoveContainer" containerID="c08f54086eca55ccab5f173c65e9745cda91215666caf391c6e83a19be2a89fc"
Aug 05 17:36:41.015429 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:36:41.015417    2571 scope.go:117] "RemoveContainer" containerID="c07e1e66686daa816d2114bef7210b4e4abda360266cca9d86672933983ba610"
Aug 05 17:36:41.015683 ip-10-0-135-2 kubenswrapper[2571]: E0805 17:36:41.015661    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c07e1e66686daa816d2114bef7210b4e4abda360266cca9d86672933983ba610\": container with ID starting with c07e1e66686daa816d2114bef7210b4e4abda360266cca9d86672933983ba610 not found: ID does not exist" containerID="c07e1e66686daa816d2114bef7210b4e4abda360266cca9d86672933983ba610"
Aug 05 17:36:41.015747 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:36:41.015690    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c07e1e66686daa816d2114bef7210b4e4abda360266cca9d86672933983ba610"} err="failed to get container status \"c07e1e66686daa816d2114bef7210b4e4abda360266cca9d86672933983ba610\": rpc error: code = NotFound desc = could not find container \"c07e1e66686daa816d2114bef7210b4e4abda360266cca9d86672933983ba610\": container with ID starting with c07e1e66686daa816d2114bef7210b4e4abda360266cca9d86672933983ba610 not found: ID does not exist"
Aug 05 17:36:41.015747 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:36:41.015706    2571 scope.go:117] "RemoveContainer" containerID="c08f54086eca55ccab5f173c65e9745cda91215666caf391c6e83a19be2a89fc"
Aug 05 17:36:41.015927 ip-10-0-135-2 kubenswrapper[2571]: E0805 17:36:41.015909    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c08f54086eca55ccab5f173c65e9745cda91215666caf391c6e83a19be2a89fc\": container with ID starting with c08f54086eca55ccab5f173c65e9745cda91215666caf391c6e83a19be2a89fc not found: ID does not exist" containerID="c08f54086eca55ccab5f173c65e9745cda91215666caf391c6e83a19be2a89fc"
Aug 05 17:36:41.015968 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:36:41.015933    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c08f54086eca55ccab5f173c65e9745cda91215666caf391c6e83a19be2a89fc"} err="failed to get container status \"c08f54086eca55ccab5f173c65e9745cda91215666caf391c6e83a19be2a89fc\": rpc error: code = NotFound desc = could not find container \"c08f54086eca55ccab5f173c65e9745cda91215666caf391c6e83a19be2a89fc\": container with ID starting with c08f54086eca55ccab5f173c65e9745cda91215666caf391c6e83a19be2a89fc not found: ID does not exist"
Aug 05 17:36:41.021880 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:36:41.021861    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-fba55-predictor-676475b57f-fljvk"]
Aug 05 17:36:41.026604 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:36:41.026585    2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-fba55-predictor-676475b57f-fljvk"]
Aug 05 17:36:41.136689 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:36:41.136654    2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="366121e0-36b3-4704-a224-ee0e872f94e4" path="/var/lib/kubelet/pods/366121e0-36b3-4704-a224-ee0e872f94e4/volumes"
Aug 05 17:36:41.137098 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:36:41.137085    2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a489ce9e-f83f-47f4-8faf-2358a9349294" path="/var/lib/kubelet/pods/a489ce9e-f83f-47f4-8faf-2358a9349294/volumes"
Aug 05 17:36:42.928609 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:36:42.928564    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-27640-predictor-799cfccbdc-v8s6m" podUID="2dbefbae-0497-4396-ab82-abcd1a08abc2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.37:8080: connect: connection refused"
Aug 05 17:36:43.933697 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:36:43.933659    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-27640-predictor-55c9498c7d-ll287" podUID="aee60f09-ffd8-4733-b965-a572468b235c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.36:8080: connect: connection refused"
Aug 05 17:36:43.990423 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:36:43.990396    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-8dddc-predictor-5558b5787-9qrcb"
Aug 05 17:36:43.990739 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:36:43.990720    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-8dddc-predictor-66fd6cdf4-znnzz"
Aug 05 17:36:43.990973 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:36:43.990949    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-8dddc-predictor-5558b5787-9qrcb" podUID="e8b2fd82-37b5-457e-ae17-47fadcce2a5e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.39:8080: connect: connection refused"
Aug 05 17:36:43.991350 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:36:43.991326    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-8dddc-predictor-66fd6cdf4-znnzz" podUID="fbe776c3-891d-4fbf-acf1-6f4b72abda6c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.38:8080: connect: connection refused"
Aug 05 17:36:52.927993 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:36:52.927914    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-27640-predictor-799cfccbdc-v8s6m" podUID="2dbefbae-0497-4396-ab82-abcd1a08abc2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.37:8080: connect: connection refused"
Aug 05 17:36:53.933777 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:36:53.933739    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-27640-predictor-55c9498c7d-ll287" podUID="aee60f09-ffd8-4733-b965-a572468b235c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.36:8080: connect: connection refused"
Aug 05 17:36:53.990884 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:36:53.990850    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-8dddc-predictor-5558b5787-9qrcb" podUID="e8b2fd82-37b5-457e-ae17-47fadcce2a5e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.39:8080: connect: connection refused"
Aug 05 17:36:53.992077 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:36:53.992052    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-8dddc-predictor-66fd6cdf4-znnzz" podUID="fbe776c3-891d-4fbf-acf1-6f4b72abda6c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.38:8080: connect: connection refused"
Aug 05 17:37:02.927852 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:37:02.927808    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-27640-predictor-799cfccbdc-v8s6m" podUID="2dbefbae-0497-4396-ab82-abcd1a08abc2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.37:8080: connect: connection refused"
Aug 05 17:37:03.933931 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:37:03.933890    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-27640-predictor-55c9498c7d-ll287" podUID="aee60f09-ffd8-4733-b965-a572468b235c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.36:8080: connect: connection refused"
Aug 05 17:37:03.991024 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:37:03.990984    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-8dddc-predictor-5558b5787-9qrcb" podUID="e8b2fd82-37b5-457e-ae17-47fadcce2a5e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.39:8080: connect: connection refused"
Aug 05 17:37:03.991357 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:37:03.991333    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-8dddc-predictor-66fd6cdf4-znnzz" podUID="fbe776c3-891d-4fbf-acf1-6f4b72abda6c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.38:8080: connect: connection refused"
Aug 05 17:37:12.928515 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:37:12.928468    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-27640-predictor-799cfccbdc-v8s6m" podUID="2dbefbae-0497-4396-ab82-abcd1a08abc2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.37:8080: connect: connection refused"
Aug 05 17:37:13.934284 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:37:13.934255    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-27640-predictor-55c9498c7d-ll287"
Aug 05 17:37:13.991418 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:37:13.991362    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-8dddc-predictor-5558b5787-9qrcb" podUID="e8b2fd82-37b5-457e-ae17-47fadcce2a5e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.39:8080: connect: connection refused"
Aug 05 17:37:13.991596 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:37:13.991410    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-8dddc-predictor-66fd6cdf4-znnzz" podUID="fbe776c3-891d-4fbf-acf1-6f4b72abda6c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.38:8080: connect: connection refused"
Aug 05 17:37:22.929479 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:37:22.929449    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-27640-predictor-799cfccbdc-v8s6m"
Aug 05 17:37:23.990917 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:37:23.990879    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-8dddc-predictor-5558b5787-9qrcb" podUID="e8b2fd82-37b5-457e-ae17-47fadcce2a5e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.39:8080: connect: connection refused"
Aug 05 17:37:23.992100 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:37:23.992080    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-8dddc-predictor-66fd6cdf4-znnzz"
Aug 05 17:37:33.992427 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:37:33.992397    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-8dddc-predictor-5558b5787-9qrcb"
Aug 05 17:37:36.466325 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:37:36.466283    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-27640-predictor-55c9498c7d-ll287"]
Aug 05 17:37:36.466748 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:37:36.466615    2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-27640-predictor-55c9498c7d-ll287" podUID="aee60f09-ffd8-4733-b965-a572468b235c" containerName="kserve-container" containerID="cri-o://1660bb41322293a4b82463349bc8c4b99cef00b0ba1ac6558a574ee1a91fc6db" gracePeriod=30
Aug 05 17:37:36.466748 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:37:36.466639    2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-27640-predictor-55c9498c7d-ll287" podUID="aee60f09-ffd8-4733-b965-a572468b235c" containerName="kube-rbac-proxy" containerID="cri-o://b8bfc1cb8de64bfece4617bc1b66e4849f167b8a38bb04d12474ce54085d1421" gracePeriod=30
Aug 05 17:37:36.484328 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:37:36.484304    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-e149b-predictor-5f5fbfc7cd-vq5fz"]
Aug 05 17:37:36.484687 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:37:36.484674    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a489ce9e-f83f-47f4-8faf-2358a9349294" containerName="kserve-container"
Aug 05 17:37:36.484736 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:37:36.484690    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="a489ce9e-f83f-47f4-8faf-2358a9349294" containerName="kserve-container"
Aug 05 17:37:36.484736 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:37:36.484709    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="366121e0-36b3-4704-a224-ee0e872f94e4" containerName="kserve-container"
Aug 05 17:37:36.484736 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:37:36.484714    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="366121e0-36b3-4704-a224-ee0e872f94e4" containerName="kserve-container"
Aug 05 17:37:36.484736 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:37:36.484724    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a489ce9e-f83f-47f4-8faf-2358a9349294" containerName="kube-rbac-proxy"
Aug 05 17:37:36.484736 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:37:36.484729    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="a489ce9e-f83f-47f4-8faf-2358a9349294" containerName="kube-rbac-proxy"
Aug 05 17:37:36.484894 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:37:36.484741    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="366121e0-36b3-4704-a224-ee0e872f94e4" containerName="kube-rbac-proxy"
Aug 05 17:37:36.484894 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:37:36.484747    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="366121e0-36b3-4704-a224-ee0e872f94e4" containerName="kube-rbac-proxy"
Aug 05 17:37:36.484894 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:37:36.484795    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="366121e0-36b3-4704-a224-ee0e872f94e4" containerName="kube-rbac-proxy"
Aug 05 17:37:36.484894 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:37:36.484805    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="a489ce9e-f83f-47f4-8faf-2358a9349294" containerName="kube-rbac-proxy"
Aug 05 17:37:36.484894 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:37:36.484810    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="a489ce9e-f83f-47f4-8faf-2358a9349294" containerName="kserve-container"
Aug 05 17:37:36.484894 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:37:36.484816    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="366121e0-36b3-4704-a224-ee0e872f94e4" containerName="kserve-container"
Aug 05 17:37:36.487997 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:37:36.487982    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-e149b-predictor-5f5fbfc7cd-vq5fz"
Aug 05 17:37:36.490410 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:37:36.490368    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-e149b-predictor-serving-cert\""
Aug 05 17:37:36.490410 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:37:36.490404    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-e149b-kube-rbac-proxy-sar-config\""
Aug 05 17:37:36.502624 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:37:36.502605    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-e149b-predictor-5f5fbfc7cd-vq5fz"]
Aug 05 17:37:36.575628 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:37:36.575600    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h57sq\" (UniqueName: \"kubernetes.io/projected/4d68887a-7de6-4f30-a190-e447efb44fbb-kube-api-access-h57sq\") pod \"success-200-isvc-e149b-predictor-5f5fbfc7cd-vq5fz\" (UID: \"4d68887a-7de6-4f30-a190-e447efb44fbb\") " pod="kserve-ci-e2e-test/success-200-isvc-e149b-predictor-5f5fbfc7cd-vq5fz"
Aug 05 17:37:36.575780 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:37:36.575652    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4d68887a-7de6-4f30-a190-e447efb44fbb-proxy-tls\") pod \"success-200-isvc-e149b-predictor-5f5fbfc7cd-vq5fz\" (UID: \"4d68887a-7de6-4f30-a190-e447efb44fbb\") " pod="kserve-ci-e2e-test/success-200-isvc-e149b-predictor-5f5fbfc7cd-vq5fz"
Aug 05 17:37:36.575780 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:37:36.575754    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4d68887a-7de6-4f30-a190-e447efb44fbb-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-e149b-predictor-5f5fbfc7cd-vq5fz\" (UID: \"4d68887a-7de6-4f30-a190-e447efb44fbb\") " pod="kserve-ci-e2e-test/success-200-isvc-e149b-predictor-5f5fbfc7cd-vq5fz"
Aug 05 17:37:36.588988 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:37:36.588959    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-27640-predictor-799cfccbdc-v8s6m"]
Aug 05 17:37:36.589245 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:37:36.589205    2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-27640-predictor-799cfccbdc-v8s6m" podUID="2dbefbae-0497-4396-ab82-abcd1a08abc2" containerName="kserve-container" containerID="cri-o://0a29512ff22422e76bb11238207fd052b40e3a4f81855c0e6d50a9b299d7ecbc" gracePeriod=30
Aug 05 17:37:36.589365 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:37:36.589224    2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-27640-predictor-799cfccbdc-v8s6m" podUID="2dbefbae-0497-4396-ab82-abcd1a08abc2" containerName="kube-rbac-proxy" containerID="cri-o://2c848b6c76864c66558141dbc2395c4c12bcf2746ed906d19172f900267c8e4f" gracePeriod=30
Aug 05 17:37:36.613570 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:37:36.613545    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-e149b-predictor-6cc79c7d67-7r5pf"]
Aug 05 17:37:36.617054 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:37:36.617039    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-e149b-predictor-6cc79c7d67-7r5pf"
Aug 05 17:37:36.619562 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:37:36.619543    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-e149b-predictor-serving-cert\""
Aug 05 17:37:36.619668 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:37:36.619549    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-e149b-kube-rbac-proxy-sar-config\""
Aug 05 17:37:36.627700 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:37:36.627675    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-e149b-predictor-6cc79c7d67-7r5pf"]
Aug 05 17:37:36.677183 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:37:36.677152    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h57sq\" (UniqueName: \"kubernetes.io/projected/4d68887a-7de6-4f30-a190-e447efb44fbb-kube-api-access-h57sq\") pod \"success-200-isvc-e149b-predictor-5f5fbfc7cd-vq5fz\" (UID: \"4d68887a-7de6-4f30-a190-e447efb44fbb\") " pod="kserve-ci-e2e-test/success-200-isvc-e149b-predictor-5f5fbfc7cd-vq5fz"
Aug 05 17:37:36.677307 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:37:36.677190    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4d68887a-7de6-4f30-a190-e447efb44fbb-proxy-tls\") pod \"success-200-isvc-e149b-predictor-5f5fbfc7cd-vq5fz\" (UID: \"4d68887a-7de6-4f30-a190-e447efb44fbb\") " pod="kserve-ci-e2e-test/success-200-isvc-e149b-predictor-5f5fbfc7cd-vq5fz"
Aug 05 17:37:36.677307 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:37:36.677241    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b528315e-f99c-4518-bdf1-87a2863c69a6-proxy-tls\") pod \"error-404-isvc-e149b-predictor-6cc79c7d67-7r5pf\" (UID: \"b528315e-f99c-4518-bdf1-87a2863c69a6\") " pod="kserve-ci-e2e-test/error-404-isvc-e149b-predictor-6cc79c7d67-7r5pf"
Aug 05 17:37:36.677307 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:37:36.677266    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b528315e-f99c-4518-bdf1-87a2863c69a6-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-e149b-predictor-6cc79c7d67-7r5pf\" (UID: \"b528315e-f99c-4518-bdf1-87a2863c69a6\") " pod="kserve-ci-e2e-test/error-404-isvc-e149b-predictor-6cc79c7d67-7r5pf"
Aug 05 17:37:36.677307 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:37:36.677296    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4d68887a-7de6-4f30-a190-e447efb44fbb-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-e149b-predictor-5f5fbfc7cd-vq5fz\" (UID: \"4d68887a-7de6-4f30-a190-e447efb44fbb\") " pod="kserve-ci-e2e-test/success-200-isvc-e149b-predictor-5f5fbfc7cd-vq5fz"
Aug 05 17:37:36.677587 ip-10-0-135-2 kubenswrapper[2571]: E0805 17:37:36.677404    2571 secret.go:189] Couldn't get secret kserve-ci-e2e-test/success-200-isvc-e149b-predictor-serving-cert: secret "success-200-isvc-e149b-predictor-serving-cert" not found
Aug 05 17:37:36.677587 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:37:36.677411    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bxnlt\" (UniqueName: \"kubernetes.io/projected/b528315e-f99c-4518-bdf1-87a2863c69a6-kube-api-access-bxnlt\") pod \"error-404-isvc-e149b-predictor-6cc79c7d67-7r5pf\" (UID: \"b528315e-f99c-4518-bdf1-87a2863c69a6\") " pod="kserve-ci-e2e-test/error-404-isvc-e149b-predictor-6cc79c7d67-7r5pf"
Aug 05 17:37:36.677587 ip-10-0-135-2 kubenswrapper[2571]: E0805 17:37:36.677472    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4d68887a-7de6-4f30-a190-e447efb44fbb-proxy-tls podName:4d68887a-7de6-4f30-a190-e447efb44fbb nodeName:}" failed. No retries permitted until 2026-08-05 17:37:37.177453615 +0000 UTC m=+1484.544819938 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/4d68887a-7de6-4f30-a190-e447efb44fbb-proxy-tls") pod "success-200-isvc-e149b-predictor-5f5fbfc7cd-vq5fz" (UID: "4d68887a-7de6-4f30-a190-e447efb44fbb") : secret "success-200-isvc-e149b-predictor-serving-cert" not found
Aug 05 17:37:36.677975 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:37:36.677954    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4d68887a-7de6-4f30-a190-e447efb44fbb-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-e149b-predictor-5f5fbfc7cd-vq5fz\" (UID: \"4d68887a-7de6-4f30-a190-e447efb44fbb\") " pod="kserve-ci-e2e-test/success-200-isvc-e149b-predictor-5f5fbfc7cd-vq5fz"
Aug 05 17:37:36.686188 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:37:36.686166    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h57sq\" (UniqueName: \"kubernetes.io/projected/4d68887a-7de6-4f30-a190-e447efb44fbb-kube-api-access-h57sq\") pod \"success-200-isvc-e149b-predictor-5f5fbfc7cd-vq5fz\" (UID: \"4d68887a-7de6-4f30-a190-e447efb44fbb\") " pod="kserve-ci-e2e-test/success-200-isvc-e149b-predictor-5f5fbfc7cd-vq5fz"
Aug 05 17:37:36.778186 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:37:36.778159    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bxnlt\" (UniqueName: \"kubernetes.io/projected/b528315e-f99c-4518-bdf1-87a2863c69a6-kube-api-access-bxnlt\") pod \"error-404-isvc-e149b-predictor-6cc79c7d67-7r5pf\" (UID: \"b528315e-f99c-4518-bdf1-87a2863c69a6\") " pod="kserve-ci-e2e-test/error-404-isvc-e149b-predictor-6cc79c7d67-7r5pf"
Aug 05 17:37:36.778311 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:37:36.778241    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b528315e-f99c-4518-bdf1-87a2863c69a6-proxy-tls\") pod \"error-404-isvc-e149b-predictor-6cc79c7d67-7r5pf\" (UID: \"b528315e-f99c-4518-bdf1-87a2863c69a6\") " pod="kserve-ci-e2e-test/error-404-isvc-e149b-predictor-6cc79c7d67-7r5pf"
Aug 05 17:37:36.778311 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:37:36.778260    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b528315e-f99c-4518-bdf1-87a2863c69a6-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-e149b-predictor-6cc79c7d67-7r5pf\" (UID: \"b528315e-f99c-4518-bdf1-87a2863c69a6\") " pod="kserve-ci-e2e-test/error-404-isvc-e149b-predictor-6cc79c7d67-7r5pf"
Aug 05 17:37:36.778462 ip-10-0-135-2 kubenswrapper[2571]: E0805 17:37:36.778398    2571 secret.go:189] Couldn't get secret kserve-ci-e2e-test/error-404-isvc-e149b-predictor-serving-cert: secret "error-404-isvc-e149b-predictor-serving-cert" not found
Aug 05 17:37:36.778462 ip-10-0-135-2 kubenswrapper[2571]: E0805 17:37:36.778459    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b528315e-f99c-4518-bdf1-87a2863c69a6-proxy-tls podName:b528315e-f99c-4518-bdf1-87a2863c69a6 nodeName:}" failed. No retries permitted until 2026-08-05 17:37:37.278445541 +0000 UTC m=+1484.645811861 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/b528315e-f99c-4518-bdf1-87a2863c69a6-proxy-tls") pod "error-404-isvc-e149b-predictor-6cc79c7d67-7r5pf" (UID: "b528315e-f99c-4518-bdf1-87a2863c69a6") : secret "error-404-isvc-e149b-predictor-serving-cert" not found
Aug 05 17:37:36.778861 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:37:36.778845    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b528315e-f99c-4518-bdf1-87a2863c69a6-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-e149b-predictor-6cc79c7d67-7r5pf\" (UID: \"b528315e-f99c-4518-bdf1-87a2863c69a6\") " pod="kserve-ci-e2e-test/error-404-isvc-e149b-predictor-6cc79c7d67-7r5pf"
Aug 05 17:37:36.787107 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:37:36.787085    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bxnlt\" (UniqueName: \"kubernetes.io/projected/b528315e-f99c-4518-bdf1-87a2863c69a6-kube-api-access-bxnlt\") pod \"error-404-isvc-e149b-predictor-6cc79c7d67-7r5pf\" (UID: \"b528315e-f99c-4518-bdf1-87a2863c69a6\") " pod="kserve-ci-e2e-test/error-404-isvc-e149b-predictor-6cc79c7d67-7r5pf"
Aug 05 17:37:37.181875 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:37:37.181790    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4d68887a-7de6-4f30-a190-e447efb44fbb-proxy-tls\") pod \"success-200-isvc-e149b-predictor-5f5fbfc7cd-vq5fz\" (UID: \"4d68887a-7de6-4f30-a190-e447efb44fbb\") " pod="kserve-ci-e2e-test/success-200-isvc-e149b-predictor-5f5fbfc7cd-vq5fz"
Aug 05 17:37:37.184216 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:37:37.184189    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4d68887a-7de6-4f30-a190-e447efb44fbb-proxy-tls\") pod \"success-200-isvc-e149b-predictor-5f5fbfc7cd-vq5fz\" (UID: \"4d68887a-7de6-4f30-a190-e447efb44fbb\") " pod="kserve-ci-e2e-test/success-200-isvc-e149b-predictor-5f5fbfc7cd-vq5fz"
Aug 05 17:37:37.241501 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:37:37.241460    2571 generic.go:356] "Generic (PLEG): container finished" podID="aee60f09-ffd8-4733-b965-a572468b235c" containerID="b8bfc1cb8de64bfece4617bc1b66e4849f167b8a38bb04d12474ce54085d1421" exitCode=2
Aug 05 17:37:37.241730 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:37:37.241501    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-27640-predictor-55c9498c7d-ll287" event={"ID":"aee60f09-ffd8-4733-b965-a572468b235c","Type":"ContainerDied","Data":"b8bfc1cb8de64bfece4617bc1b66e4849f167b8a38bb04d12474ce54085d1421"}
Aug 05 17:37:37.243433 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:37:37.243408    2571 generic.go:356] "Generic (PLEG): container finished" podID="2dbefbae-0497-4396-ab82-abcd1a08abc2" containerID="2c848b6c76864c66558141dbc2395c4c12bcf2746ed906d19172f900267c8e4f" exitCode=2
Aug 05 17:37:37.243548 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:37:37.243474    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-27640-predictor-799cfccbdc-v8s6m" event={"ID":"2dbefbae-0497-4396-ab82-abcd1a08abc2","Type":"ContainerDied","Data":"2c848b6c76864c66558141dbc2395c4c12bcf2746ed906d19172f900267c8e4f"}
Aug 05 17:37:37.283148 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:37:37.283118    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b528315e-f99c-4518-bdf1-87a2863c69a6-proxy-tls\") pod \"error-404-isvc-e149b-predictor-6cc79c7d67-7r5pf\" (UID: \"b528315e-f99c-4518-bdf1-87a2863c69a6\") " pod="kserve-ci-e2e-test/error-404-isvc-e149b-predictor-6cc79c7d67-7r5pf"
Aug 05 17:37:37.285543 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:37:37.285528    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b528315e-f99c-4518-bdf1-87a2863c69a6-proxy-tls\") pod \"error-404-isvc-e149b-predictor-6cc79c7d67-7r5pf\" (UID: \"b528315e-f99c-4518-bdf1-87a2863c69a6\") " pod="kserve-ci-e2e-test/error-404-isvc-e149b-predictor-6cc79c7d67-7r5pf"
Aug 05 17:37:37.397563 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:37:37.397525    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-e149b-predictor-5f5fbfc7cd-vq5fz"
Aug 05 17:37:37.527637 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:37:37.527612    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-e149b-predictor-6cc79c7d67-7r5pf"
Aug 05 17:37:37.535939 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:37:37.535807    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-e149b-predictor-5f5fbfc7cd-vq5fz"]
Aug 05 17:37:37.539428 ip-10-0-135-2 kubenswrapper[2571]: W0805 17:37:37.539396    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4d68887a_7de6_4f30_a190_e447efb44fbb.slice/crio-b5989bad421304c90c05a21646be2cc38ed321fb4ca0877f7f025841f27f4606 WatchSource:0}: Error finding container b5989bad421304c90c05a21646be2cc38ed321fb4ca0877f7f025841f27f4606: Status 404 returned error can't find the container with id b5989bad421304c90c05a21646be2cc38ed321fb4ca0877f7f025841f27f4606
Aug 05 17:37:37.676116 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:37:37.676095    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-e149b-predictor-6cc79c7d67-7r5pf"]
Aug 05 17:37:37.677590 ip-10-0-135-2 kubenswrapper[2571]: W0805 17:37:37.677569    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb528315e_f99c_4518_bdf1_87a2863c69a6.slice/crio-6e15b089a39e4390f1e7b57053005f37e0ab7b31ba265a9deed24a5bca529080 WatchSource:0}: Error finding container 6e15b089a39e4390f1e7b57053005f37e0ab7b31ba265a9deed24a5bca529080: Status 404 returned error can't find the container with id 6e15b089a39e4390f1e7b57053005f37e0ab7b31ba265a9deed24a5bca529080
Aug 05 17:37:37.923724 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:37:37.923683    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-27640-predictor-799cfccbdc-v8s6m" podUID="2dbefbae-0497-4396-ab82-abcd1a08abc2" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.37:8643/healthz\": dial tcp 10.134.0.37:8643: connect: connection refused"
Aug 05 17:37:38.251672 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:37:38.251642    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-e149b-predictor-5f5fbfc7cd-vq5fz" event={"ID":"4d68887a-7de6-4f30-a190-e447efb44fbb","Type":"ContainerStarted","Data":"8f94407c18f52a938b716af90bfd6fd64eacb4faffe02684f83be83f0291082b"}
Aug 05 17:37:38.251858 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:37:38.251698    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-e149b-predictor-5f5fbfc7cd-vq5fz" event={"ID":"4d68887a-7de6-4f30-a190-e447efb44fbb","Type":"ContainerStarted","Data":"7b2d132afcdd21c1bbdda244cfc1ecdbeccda0d22aed4eb60aa76220c6b2cb7e"}
Aug 05 17:37:38.251858 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:37:38.251733    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-e149b-predictor-5f5fbfc7cd-vq5fz" event={"ID":"4d68887a-7de6-4f30-a190-e447efb44fbb","Type":"ContainerStarted","Data":"b5989bad421304c90c05a21646be2cc38ed321fb4ca0877f7f025841f27f4606"}
Aug 05 17:37:38.251858 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:37:38.251765    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-e149b-predictor-5f5fbfc7cd-vq5fz"
Aug 05 17:37:38.253436 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:37:38.253409    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-e149b-predictor-6cc79c7d67-7r5pf" event={"ID":"b528315e-f99c-4518-bdf1-87a2863c69a6","Type":"ContainerStarted","Data":"51761bed037f05547f7d368956e335f28a0ea8a4ed69c93169a3c25ce90168d9"}
Aug 05 17:37:38.253576 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:37:38.253464    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-e149b-predictor-6cc79c7d67-7r5pf" event={"ID":"b528315e-f99c-4518-bdf1-87a2863c69a6","Type":"ContainerStarted","Data":"018eccd20db267e475a715d33621f2c67123455028fe1ce983553d7eb400b911"}
Aug 05 17:37:38.253576 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:37:38.253508    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-e149b-predictor-6cc79c7d67-7r5pf"
Aug 05 17:37:38.253576 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:37:38.253536    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-e149b-predictor-6cc79c7d67-7r5pf"
Aug 05 17:37:38.253576 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:37:38.253560    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-e149b-predictor-6cc79c7d67-7r5pf" event={"ID":"b528315e-f99c-4518-bdf1-87a2863c69a6","Type":"ContainerStarted","Data":"6e15b089a39e4390f1e7b57053005f37e0ab7b31ba265a9deed24a5bca529080"}
Aug 05 17:37:38.254812 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:37:38.254790    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-e149b-predictor-6cc79c7d67-7r5pf" podUID="b528315e-f99c-4518-bdf1-87a2863c69a6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.41:8080: connect: connection refused"
Aug 05 17:37:38.278494 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:37:38.278450    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/success-200-isvc-e149b-predictor-5f5fbfc7cd-vq5fz" podStartSLOduration=2.2784368170000002 podStartE2EDuration="2.278436817s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-05 17:37:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-05 17:37:38.27632537 +0000 UTC m=+1485.643691709" watchObservedRunningTime="2026-08-05 17:37:38.278436817 +0000 UTC m=+1485.645803167"
Aug 05 17:37:38.303588 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:37:38.303538    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/error-404-isvc-e149b-predictor-6cc79c7d67-7r5pf" podStartSLOduration=2.303520987 podStartE2EDuration="2.303520987s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-05 17:37:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-05 17:37:38.303152695 +0000 UTC m=+1485.670519055" watchObservedRunningTime="2026-08-05 17:37:38.303520987 +0000 UTC m=+1485.670887333"
Aug 05 17:37:38.928222 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:37:38.928181    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-27640-predictor-55c9498c7d-ll287" podUID="aee60f09-ffd8-4733-b965-a572468b235c" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.36:8643/healthz\": dial tcp 10.134.0.36:8643: connect: connection refused"
Aug 05 17:37:39.258708 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:37:39.258669    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-e149b-predictor-5f5fbfc7cd-vq5fz"
Aug 05 17:37:39.258906 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:37:39.258721    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-e149b-predictor-6cc79c7d67-7r5pf" podUID="b528315e-f99c-4518-bdf1-87a2863c69a6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.41:8080: connect: connection refused"
Aug 05 17:37:39.260069 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:37:39.260043    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-e149b-predictor-5f5fbfc7cd-vq5fz" podUID="4d68887a-7de6-4f30-a190-e447efb44fbb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.40:8080: connect: connection refused"
Aug 05 17:37:39.620295 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:37:39.620268    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-27640-predictor-55c9498c7d-ll287"
Aug 05 17:37:39.706774 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:37:39.706750    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/aee60f09-ffd8-4733-b965-a572468b235c-kube-rbac-proxy-sar-config\") pod \"aee60f09-ffd8-4733-b965-a572468b235c\" (UID: \"aee60f09-ffd8-4733-b965-a572468b235c\") "
Aug 05 17:37:39.706841 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:37:39.706826    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-z68z9\" (UniqueName: \"kubernetes.io/projected/aee60f09-ffd8-4733-b965-a572468b235c-kube-api-access-z68z9\") pod \"aee60f09-ffd8-4733-b965-a572468b235c\" (UID: \"aee60f09-ffd8-4733-b965-a572468b235c\") "
Aug 05 17:37:39.706897 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:37:39.706884    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/aee60f09-ffd8-4733-b965-a572468b235c-proxy-tls\") pod \"aee60f09-ffd8-4733-b965-a572468b235c\" (UID: \"aee60f09-ffd8-4733-b965-a572468b235c\") "
Aug 05 17:37:39.707255 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:37:39.707213    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aee60f09-ffd8-4733-b965-a572468b235c-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "aee60f09-ffd8-4733-b965-a572468b235c" (UID: "aee60f09-ffd8-4733-b965-a572468b235c"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 05 17:37:39.708750 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:37:39.708724    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aee60f09-ffd8-4733-b965-a572468b235c-kube-api-access-z68z9" (OuterVolumeSpecName: "kube-api-access-z68z9") pod "aee60f09-ffd8-4733-b965-a572468b235c" (UID: "aee60f09-ffd8-4733-b965-a572468b235c"). InnerVolumeSpecName "kube-api-access-z68z9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 05 17:37:39.708915 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:37:39.708890    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/aee60f09-ffd8-4733-b965-a572468b235c-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "aee60f09-ffd8-4733-b965-a572468b235c" (UID: "aee60f09-ffd8-4733-b965-a572468b235c"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 05 17:37:39.808080 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:37:39.808016    2571 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/aee60f09-ffd8-4733-b965-a572468b235c-kube-rbac-proxy-sar-config\") on node \"ip-10-0-135-2.ec2.internal\" DevicePath \"\""
Aug 05 17:37:39.808080 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:37:39.808040    2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z68z9\" (UniqueName: \"kubernetes.io/projected/aee60f09-ffd8-4733-b965-a572468b235c-kube-api-access-z68z9\") on node \"ip-10-0-135-2.ec2.internal\" DevicePath \"\""
Aug 05 17:37:39.808080 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:37:39.808050    2571 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/aee60f09-ffd8-4733-b965-a572468b235c-proxy-tls\") on node \"ip-10-0-135-2.ec2.internal\" DevicePath \"\""
Aug 05 17:37:39.821389 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:37:39.821357    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-27640-predictor-799cfccbdc-v8s6m"
Aug 05 17:37:39.908915 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:37:39.908886    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2dbefbae-0497-4396-ab82-abcd1a08abc2-proxy-tls\") pod \"2dbefbae-0497-4396-ab82-abcd1a08abc2\" (UID: \"2dbefbae-0497-4396-ab82-abcd1a08abc2\") "
Aug 05 17:37:39.909065 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:37:39.908983    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2dbefbae-0497-4396-ab82-abcd1a08abc2-kube-rbac-proxy-sar-config\") pod \"2dbefbae-0497-4396-ab82-abcd1a08abc2\" (UID: \"2dbefbae-0497-4396-ab82-abcd1a08abc2\") "
Aug 05 17:37:39.909065 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:37:39.909009    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wktb4\" (UniqueName: \"kubernetes.io/projected/2dbefbae-0497-4396-ab82-abcd1a08abc2-kube-api-access-wktb4\") pod \"2dbefbae-0497-4396-ab82-abcd1a08abc2\" (UID: \"2dbefbae-0497-4396-ab82-abcd1a08abc2\") "
Aug 05 17:37:39.909357 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:37:39.909334    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2dbefbae-0497-4396-ab82-abcd1a08abc2-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "2dbefbae-0497-4396-ab82-abcd1a08abc2" (UID: "2dbefbae-0497-4396-ab82-abcd1a08abc2"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 05 17:37:39.910975 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:37:39.910958    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2dbefbae-0497-4396-ab82-abcd1a08abc2-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "2dbefbae-0497-4396-ab82-abcd1a08abc2" (UID: "2dbefbae-0497-4396-ab82-abcd1a08abc2"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 05 17:37:39.911033 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:37:39.911001    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2dbefbae-0497-4396-ab82-abcd1a08abc2-kube-api-access-wktb4" (OuterVolumeSpecName: "kube-api-access-wktb4") pod "2dbefbae-0497-4396-ab82-abcd1a08abc2" (UID: "2dbefbae-0497-4396-ab82-abcd1a08abc2"). InnerVolumeSpecName "kube-api-access-wktb4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 05 17:37:40.010371 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:37:40.010342    2571 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2dbefbae-0497-4396-ab82-abcd1a08abc2-proxy-tls\") on node \"ip-10-0-135-2.ec2.internal\" DevicePath \"\""
Aug 05 17:37:40.010371 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:37:40.010366    2571 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2dbefbae-0497-4396-ab82-abcd1a08abc2-kube-rbac-proxy-sar-config\") on node \"ip-10-0-135-2.ec2.internal\" DevicePath \"\""
Aug 05 17:37:40.010371 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:37:40.010393    2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wktb4\" (UniqueName: \"kubernetes.io/projected/2dbefbae-0497-4396-ab82-abcd1a08abc2-kube-api-access-wktb4\") on node \"ip-10-0-135-2.ec2.internal\" DevicePath \"\""
Aug 05 17:37:40.265210 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:37:40.265177    2571 generic.go:356] "Generic (PLEG): container finished" podID="aee60f09-ffd8-4733-b965-a572468b235c" containerID="1660bb41322293a4b82463349bc8c4b99cef00b0ba1ac6558a574ee1a91fc6db" exitCode=0
Aug 05 17:37:40.265430 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:37:40.265232    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-27640-predictor-55c9498c7d-ll287"
Aug 05 17:37:40.265430 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:37:40.265262    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-27640-predictor-55c9498c7d-ll287" event={"ID":"aee60f09-ffd8-4733-b965-a572468b235c","Type":"ContainerDied","Data":"1660bb41322293a4b82463349bc8c4b99cef00b0ba1ac6558a574ee1a91fc6db"}
Aug 05 17:37:40.265430 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:37:40.265336    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-27640-predictor-55c9498c7d-ll287" event={"ID":"aee60f09-ffd8-4733-b965-a572468b235c","Type":"ContainerDied","Data":"f8ed8cdc1109642e433eeab077d2bf53f570626302ec645bf9e54a33ac570feb"}
Aug 05 17:37:40.265430 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:37:40.265393    2571 scope.go:117] "RemoveContainer" containerID="b8bfc1cb8de64bfece4617bc1b66e4849f167b8a38bb04d12474ce54085d1421"
Aug 05 17:37:40.266951 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:37:40.266929    2571 generic.go:356] "Generic (PLEG): container finished" podID="2dbefbae-0497-4396-ab82-abcd1a08abc2" containerID="0a29512ff22422e76bb11238207fd052b40e3a4f81855c0e6d50a9b299d7ecbc" exitCode=0
Aug 05 17:37:40.267075 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:37:40.266988    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-27640-predictor-799cfccbdc-v8s6m"
Aug 05 17:37:40.267075 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:37:40.267014    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-27640-predictor-799cfccbdc-v8s6m" event={"ID":"2dbefbae-0497-4396-ab82-abcd1a08abc2","Type":"ContainerDied","Data":"0a29512ff22422e76bb11238207fd052b40e3a4f81855c0e6d50a9b299d7ecbc"}
Aug 05 17:37:40.267194 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:37:40.267078    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-27640-predictor-799cfccbdc-v8s6m" event={"ID":"2dbefbae-0497-4396-ab82-abcd1a08abc2","Type":"ContainerDied","Data":"20c39747a90d6a7f01e63d3a274265da7cd7b0a6f2c4176fcf35a92b2b0f0dbc"}
Aug 05 17:37:40.267541 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:37:40.267494    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-e149b-predictor-5f5fbfc7cd-vq5fz" podUID="4d68887a-7de6-4f30-a190-e447efb44fbb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.40:8080: connect: connection refused"
Aug 05 17:37:40.277470 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:37:40.277453    2571 scope.go:117] "RemoveContainer" containerID="1660bb41322293a4b82463349bc8c4b99cef00b0ba1ac6558a574ee1a91fc6db"
Aug 05 17:37:40.286079 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:37:40.286045    2571 scope.go:117] "RemoveContainer" containerID="b8bfc1cb8de64bfece4617bc1b66e4849f167b8a38bb04d12474ce54085d1421"
Aug 05 17:37:40.286306 ip-10-0-135-2 kubenswrapper[2571]: E0805 17:37:40.286288    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b8bfc1cb8de64bfece4617bc1b66e4849f167b8a38bb04d12474ce54085d1421\": container with ID starting with b8bfc1cb8de64bfece4617bc1b66e4849f167b8a38bb04d12474ce54085d1421 not found: ID does not exist" containerID="b8bfc1cb8de64bfece4617bc1b66e4849f167b8a38bb04d12474ce54085d1421"
Aug 05 17:37:40.286348 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:37:40.286314    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b8bfc1cb8de64bfece4617bc1b66e4849f167b8a38bb04d12474ce54085d1421"} err="failed to get container status \"b8bfc1cb8de64bfece4617bc1b66e4849f167b8a38bb04d12474ce54085d1421\": rpc error: code = NotFound desc = could not find container \"b8bfc1cb8de64bfece4617bc1b66e4849f167b8a38bb04d12474ce54085d1421\": container with ID starting with b8bfc1cb8de64bfece4617bc1b66e4849f167b8a38bb04d12474ce54085d1421 not found: ID does not exist"
Aug 05 17:37:40.286348 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:37:40.286331    2571 scope.go:117] "RemoveContainer" containerID="1660bb41322293a4b82463349bc8c4b99cef00b0ba1ac6558a574ee1a91fc6db"
Aug 05 17:37:40.286621 ip-10-0-135-2 kubenswrapper[2571]: E0805 17:37:40.286602    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1660bb41322293a4b82463349bc8c4b99cef00b0ba1ac6558a574ee1a91fc6db\": container with ID starting with 1660bb41322293a4b82463349bc8c4b99cef00b0ba1ac6558a574ee1a91fc6db not found: ID does not exist" containerID="1660bb41322293a4b82463349bc8c4b99cef00b0ba1ac6558a574ee1a91fc6db"
Aug 05 17:37:40.286678 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:37:40.286628    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1660bb41322293a4b82463349bc8c4b99cef00b0ba1ac6558a574ee1a91fc6db"} err="failed to get container status \"1660bb41322293a4b82463349bc8c4b99cef00b0ba1ac6558a574ee1a91fc6db\": rpc error: code = NotFound desc = could not find container \"1660bb41322293a4b82463349bc8c4b99cef00b0ba1ac6558a574ee1a91fc6db\": container with ID starting with 1660bb41322293a4b82463349bc8c4b99cef00b0ba1ac6558a574ee1a91fc6db not found: ID does not exist"
Aug 05 17:37:40.286678 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:37:40.286643    2571 scope.go:117] "RemoveContainer" containerID="2c848b6c76864c66558141dbc2395c4c12bcf2746ed906d19172f900267c8e4f"
Aug 05 17:37:40.293866 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:37:40.293848    2571 scope.go:117] "RemoveContainer" containerID="0a29512ff22422e76bb11238207fd052b40e3a4f81855c0e6d50a9b299d7ecbc"
Aug 05 17:37:40.297732 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:37:40.297713    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-27640-predictor-55c9498c7d-ll287"]
Aug 05 17:37:40.300707 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:37:40.300688    2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-27640-predictor-55c9498c7d-ll287"]
Aug 05 17:37:40.301428 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:37:40.301412    2571 scope.go:117] "RemoveContainer" containerID="2c848b6c76864c66558141dbc2395c4c12bcf2746ed906d19172f900267c8e4f"
Aug 05 17:37:40.301657 ip-10-0-135-2 kubenswrapper[2571]: E0805 17:37:40.301636    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2c848b6c76864c66558141dbc2395c4c12bcf2746ed906d19172f900267c8e4f\": container with ID starting with 2c848b6c76864c66558141dbc2395c4c12bcf2746ed906d19172f900267c8e4f not found: ID does not exist" containerID="2c848b6c76864c66558141dbc2395c4c12bcf2746ed906d19172f900267c8e4f"
Aug 05 17:37:40.301721 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:37:40.301664    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2c848b6c76864c66558141dbc2395c4c12bcf2746ed906d19172f900267c8e4f"} err="failed to get container status \"2c848b6c76864c66558141dbc2395c4c12bcf2746ed906d19172f900267c8e4f\": rpc error: code = NotFound desc = could not find container \"2c848b6c76864c66558141dbc2395c4c12bcf2746ed906d19172f900267c8e4f\": container with ID starting with 2c848b6c76864c66558141dbc2395c4c12bcf2746ed906d19172f900267c8e4f not found: ID does not exist"
Aug 05 17:37:40.301721 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:37:40.301680    2571 scope.go:117] "RemoveContainer" containerID="0a29512ff22422e76bb11238207fd052b40e3a4f81855c0e6d50a9b299d7ecbc"
Aug 05 17:37:40.301918 ip-10-0-135-2 kubenswrapper[2571]: E0805 17:37:40.301901    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0a29512ff22422e76bb11238207fd052b40e3a4f81855c0e6d50a9b299d7ecbc\": container with ID starting with 0a29512ff22422e76bb11238207fd052b40e3a4f81855c0e6d50a9b299d7ecbc not found: ID does not exist" containerID="0a29512ff22422e76bb11238207fd052b40e3a4f81855c0e6d50a9b299d7ecbc"
Aug 05 17:37:40.301962 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:37:40.301923    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0a29512ff22422e76bb11238207fd052b40e3a4f81855c0e6d50a9b299d7ecbc"} err="failed to get container status \"0a29512ff22422e76bb11238207fd052b40e3a4f81855c0e6d50a9b299d7ecbc\": rpc error: code = NotFound desc = could not find container \"0a29512ff22422e76bb11238207fd052b40e3a4f81855c0e6d50a9b299d7ecbc\": container with ID starting with 0a29512ff22422e76bb11238207fd052b40e3a4f81855c0e6d50a9b299d7ecbc not found: ID does not exist"
Aug 05 17:37:40.318771 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:37:40.318753    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-27640-predictor-799cfccbdc-v8s6m"]
Aug 05 17:37:40.323149 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:37:40.323121    2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-27640-predictor-799cfccbdc-v8s6m"]
Aug 05 17:37:41.136559 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:37:41.136527    2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2dbefbae-0497-4396-ab82-abcd1a08abc2" path="/var/lib/kubelet/pods/2dbefbae-0497-4396-ab82-abcd1a08abc2/volumes"
Aug 05 17:37:41.136950 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:37:41.136936    2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="aee60f09-ffd8-4733-b965-a572468b235c" path="/var/lib/kubelet/pods/aee60f09-ffd8-4733-b965-a572468b235c/volumes"
Aug 05 17:37:44.262968 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:37:44.262941    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-e149b-predictor-6cc79c7d67-7r5pf"
Aug 05 17:37:44.263440 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:37:44.263406    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-e149b-predictor-6cc79c7d67-7r5pf" podUID="b528315e-f99c-4518-bdf1-87a2863c69a6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.41:8080: connect: connection refused"
Aug 05 17:37:45.272272 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:37:45.272241    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-e149b-predictor-5f5fbfc7cd-vq5fz"
Aug 05 17:37:45.272787 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:37:45.272756    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-e149b-predictor-5f5fbfc7cd-vq5fz" podUID="4d68887a-7de6-4f30-a190-e447efb44fbb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.40:8080: connect: connection refused"
Aug 05 17:37:54.263884 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:37:54.263840    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-e149b-predictor-6cc79c7d67-7r5pf" podUID="b528315e-f99c-4518-bdf1-87a2863c69a6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.41:8080: connect: connection refused"
Aug 05 17:37:55.272771 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:37:55.272732    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-e149b-predictor-5f5fbfc7cd-vq5fz" podUID="4d68887a-7de6-4f30-a190-e447efb44fbb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.40:8080: connect: connection refused"
Aug 05 17:38:04.263360 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:38:04.263320    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-e149b-predictor-6cc79c7d67-7r5pf" podUID="b528315e-f99c-4518-bdf1-87a2863c69a6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.41:8080: connect: connection refused"
Aug 05 17:38:05.272793 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:38:05.272755    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-e149b-predictor-5f5fbfc7cd-vq5fz" podUID="4d68887a-7de6-4f30-a190-e447efb44fbb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.40:8080: connect: connection refused"
Aug 05 17:38:14.263457 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:38:14.263414    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-e149b-predictor-6cc79c7d67-7r5pf" podUID="b528315e-f99c-4518-bdf1-87a2863c69a6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.41:8080: connect: connection refused"
Aug 05 17:38:15.273449 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:38:15.273352    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-e149b-predictor-5f5fbfc7cd-vq5fz" podUID="4d68887a-7de6-4f30-a190-e447efb44fbb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.40:8080: connect: connection refused"
Aug 05 17:38:24.264567 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:38:24.264534    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-e149b-predictor-6cc79c7d67-7r5pf"
Aug 05 17:38:25.274111 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:38:25.274083    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-e149b-predictor-5f5fbfc7cd-vq5fz"
Aug 05 17:45:51.313856 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:45:51.313785    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-8dddc-predictor-66fd6cdf4-znnzz"]
Aug 05 17:45:51.316287 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:45:51.314029    2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-8dddc-predictor-66fd6cdf4-znnzz" podUID="fbe776c3-891d-4fbf-acf1-6f4b72abda6c" containerName="kserve-container" containerID="cri-o://e8d9d961405a7d16ebafea8a944e62ba9f391abbd3de5956e28a657d4ef6a6dd" gracePeriod=30
Aug 05 17:45:51.316287 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:45:51.314060    2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-8dddc-predictor-66fd6cdf4-znnzz" podUID="fbe776c3-891d-4fbf-acf1-6f4b72abda6c" containerName="kube-rbac-proxy" containerID="cri-o://dbcab6cedb0bc16ca8d5c649355fa7c0f97ba79cab24a764b3330d51f5f57f9e" gracePeriod=30
Aug 05 17:45:51.340039 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:45:51.340007    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-92074-predictor-79d689c86d-w5pjl"]
Aug 05 17:45:51.340571 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:45:51.340554    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2dbefbae-0497-4396-ab82-abcd1a08abc2" containerName="kube-rbac-proxy"
Aug 05 17:45:51.340619 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:45:51.340575    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="2dbefbae-0497-4396-ab82-abcd1a08abc2" containerName="kube-rbac-proxy"
Aug 05 17:45:51.340619 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:45:51.340601    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="aee60f09-ffd8-4733-b965-a572468b235c" containerName="kserve-container"
Aug 05 17:45:51.340619 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:45:51.340611    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="aee60f09-ffd8-4733-b965-a572468b235c" containerName="kserve-container"
Aug 05 17:45:51.340717 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:45:51.340623    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="aee60f09-ffd8-4733-b965-a572468b235c" containerName="kube-rbac-proxy"
Aug 05 17:45:51.340717 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:45:51.340631    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="aee60f09-ffd8-4733-b965-a572468b235c" containerName="kube-rbac-proxy"
Aug 05 17:45:51.340717 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:45:51.340646    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2dbefbae-0497-4396-ab82-abcd1a08abc2" containerName="kserve-container"
Aug 05 17:45:51.340717 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:45:51.340655    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="2dbefbae-0497-4396-ab82-abcd1a08abc2" containerName="kserve-container"
Aug 05 17:45:51.340836 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:45:51.340762    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="aee60f09-ffd8-4733-b965-a572468b235c" containerName="kserve-container"
Aug 05 17:45:51.340836 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:45:51.340775    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="2dbefbae-0497-4396-ab82-abcd1a08abc2" containerName="kserve-container"
Aug 05 17:45:51.340836 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:45:51.340785    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="2dbefbae-0497-4396-ab82-abcd1a08abc2" containerName="kube-rbac-proxy"
Aug 05 17:45:51.340836 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:45:51.340794    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="aee60f09-ffd8-4733-b965-a572468b235c" containerName="kube-rbac-proxy"
Aug 05 17:45:51.344201 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:45:51.344185    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-92074-predictor-79d689c86d-w5pjl"
Aug 05 17:45:51.347099 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:45:51.347079    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-92074-kube-rbac-proxy-sar-config\""
Aug 05 17:45:51.347184 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:45:51.347080    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-92074-predictor-serving-cert\""
Aug 05 17:45:51.354857 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:45:51.354839    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-92074-predictor-79d689c86d-w5pjl"]
Aug 05 17:45:51.434661 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:45:51.434628    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-8dddc-predictor-5558b5787-9qrcb"]
Aug 05 17:45:51.434879 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:45:51.434856    2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-8dddc-predictor-5558b5787-9qrcb" podUID="e8b2fd82-37b5-457e-ae17-47fadcce2a5e" containerName="kserve-container" containerID="cri-o://b96ad862b6b46a99e41b0dd376e33c4bfee3ee23be666681737e863a6cc04422" gracePeriod=30
Aug 05 17:45:51.434967 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:45:51.434884    2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-8dddc-predictor-5558b5787-9qrcb" podUID="e8b2fd82-37b5-457e-ae17-47fadcce2a5e" containerName="kube-rbac-proxy" containerID="cri-o://3e2d115f8eafd58031a32787e6293dc6a80c6873300927324f5c22431429125e" gracePeriod=30
Aug 05 17:45:51.447917 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:45:51.447896    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-92074-predictor-589dc7f869-8mmn2"]
Aug 05 17:45:51.451317 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:45:51.451300    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-92074-predictor-589dc7f869-8mmn2"
Aug 05 17:45:51.453673 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:45:51.453652    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-92074-kube-rbac-proxy-sar-config\""
Aug 05 17:45:51.453673 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:45:51.453665    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-92074-predictor-serving-cert\""
Aug 05 17:45:51.457504 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:45:51.457481    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9jdgz\" (UniqueName: \"kubernetes.io/projected/337b1675-3909-4f89-a905-35432d0ff577-kube-api-access-9jdgz\") pod \"success-200-isvc-92074-predictor-79d689c86d-w5pjl\" (UID: \"337b1675-3909-4f89-a905-35432d0ff577\") " pod="kserve-ci-e2e-test/success-200-isvc-92074-predictor-79d689c86d-w5pjl"
Aug 05 17:45:51.457604 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:45:51.457546    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/337b1675-3909-4f89-a905-35432d0ff577-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-92074-predictor-79d689c86d-w5pjl\" (UID: \"337b1675-3909-4f89-a905-35432d0ff577\") " pod="kserve-ci-e2e-test/success-200-isvc-92074-predictor-79d689c86d-w5pjl"
Aug 05 17:45:51.457604 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:45:51.457580    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/337b1675-3909-4f89-a905-35432d0ff577-proxy-tls\") pod \"success-200-isvc-92074-predictor-79d689c86d-w5pjl\" (UID: \"337b1675-3909-4f89-a905-35432d0ff577\") " pod="kserve-ci-e2e-test/success-200-isvc-92074-predictor-79d689c86d-w5pjl"
Aug 05 17:45:51.464129 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:45:51.464111    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-92074-predictor-589dc7f869-8mmn2"]
Aug 05 17:45:51.558235 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:45:51.558200    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/337b1675-3909-4f89-a905-35432d0ff577-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-92074-predictor-79d689c86d-w5pjl\" (UID: \"337b1675-3909-4f89-a905-35432d0ff577\") " pod="kserve-ci-e2e-test/success-200-isvc-92074-predictor-79d689c86d-w5pjl"
Aug 05 17:45:51.558343 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:45:51.558248    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/337b1675-3909-4f89-a905-35432d0ff577-proxy-tls\") pod \"success-200-isvc-92074-predictor-79d689c86d-w5pjl\" (UID: \"337b1675-3909-4f89-a905-35432d0ff577\") " pod="kserve-ci-e2e-test/success-200-isvc-92074-predictor-79d689c86d-w5pjl"
Aug 05 17:45:51.558343 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:45:51.558275    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/45df6046-112b-493c-b9bf-b746d05c3a21-proxy-tls\") pod \"error-404-isvc-92074-predictor-589dc7f869-8mmn2\" (UID: \"45df6046-112b-493c-b9bf-b746d05c3a21\") " pod="kserve-ci-e2e-test/error-404-isvc-92074-predictor-589dc7f869-8mmn2"
Aug 05 17:45:51.558343 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:45:51.558303    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-42gj7\" (UniqueName: \"kubernetes.io/projected/45df6046-112b-493c-b9bf-b746d05c3a21-kube-api-access-42gj7\") pod \"error-404-isvc-92074-predictor-589dc7f869-8mmn2\" (UID: \"45df6046-112b-493c-b9bf-b746d05c3a21\") " pod="kserve-ci-e2e-test/error-404-isvc-92074-predictor-589dc7f869-8mmn2"
Aug 05 17:45:51.558537 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:45:51.558350    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/45df6046-112b-493c-b9bf-b746d05c3a21-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-92074-predictor-589dc7f869-8mmn2\" (UID: \"45df6046-112b-493c-b9bf-b746d05c3a21\") " pod="kserve-ci-e2e-test/error-404-isvc-92074-predictor-589dc7f869-8mmn2"
Aug 05 17:45:51.558537 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:45:51.558424    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9jdgz\" (UniqueName: \"kubernetes.io/projected/337b1675-3909-4f89-a905-35432d0ff577-kube-api-access-9jdgz\") pod \"success-200-isvc-92074-predictor-79d689c86d-w5pjl\" (UID: \"337b1675-3909-4f89-a905-35432d0ff577\") " pod="kserve-ci-e2e-test/success-200-isvc-92074-predictor-79d689c86d-w5pjl"
Aug 05 17:45:51.558834 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:45:51.558814    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/337b1675-3909-4f89-a905-35432d0ff577-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-92074-predictor-79d689c86d-w5pjl\" (UID: \"337b1675-3909-4f89-a905-35432d0ff577\") " pod="kserve-ci-e2e-test/success-200-isvc-92074-predictor-79d689c86d-w5pjl"
Aug 05 17:45:51.560814 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:45:51.560785    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/337b1675-3909-4f89-a905-35432d0ff577-proxy-tls\") pod \"success-200-isvc-92074-predictor-79d689c86d-w5pjl\" (UID: \"337b1675-3909-4f89-a905-35432d0ff577\") " pod="kserve-ci-e2e-test/success-200-isvc-92074-predictor-79d689c86d-w5pjl"
Aug 05 17:45:51.571091 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:45:51.571043    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9jdgz\" (UniqueName: \"kubernetes.io/projected/337b1675-3909-4f89-a905-35432d0ff577-kube-api-access-9jdgz\") pod \"success-200-isvc-92074-predictor-79d689c86d-w5pjl\" (UID: \"337b1675-3909-4f89-a905-35432d0ff577\") " pod="kserve-ci-e2e-test/success-200-isvc-92074-predictor-79d689c86d-w5pjl"
Aug 05 17:45:51.654599 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:45:51.654568    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-92074-predictor-79d689c86d-w5pjl"
Aug 05 17:45:51.659700 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:45:51.659665    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/45df6046-112b-493c-b9bf-b746d05c3a21-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-92074-predictor-589dc7f869-8mmn2\" (UID: \"45df6046-112b-493c-b9bf-b746d05c3a21\") " pod="kserve-ci-e2e-test/error-404-isvc-92074-predictor-589dc7f869-8mmn2"
Aug 05 17:45:51.659834 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:45:51.659815    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/45df6046-112b-493c-b9bf-b746d05c3a21-proxy-tls\") pod \"error-404-isvc-92074-predictor-589dc7f869-8mmn2\" (UID: \"45df6046-112b-493c-b9bf-b746d05c3a21\") " pod="kserve-ci-e2e-test/error-404-isvc-92074-predictor-589dc7f869-8mmn2"
Aug 05 17:45:51.659954 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:45:51.659848    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-42gj7\" (UniqueName: \"kubernetes.io/projected/45df6046-112b-493c-b9bf-b746d05c3a21-kube-api-access-42gj7\") pod \"error-404-isvc-92074-predictor-589dc7f869-8mmn2\" (UID: \"45df6046-112b-493c-b9bf-b746d05c3a21\") " pod="kserve-ci-e2e-test/error-404-isvc-92074-predictor-589dc7f869-8mmn2"
Aug 05 17:45:51.660013 ip-10-0-135-2 kubenswrapper[2571]: E0805 17:45:51.659960    2571 secret.go:189] Couldn't get secret kserve-ci-e2e-test/error-404-isvc-92074-predictor-serving-cert: secret "error-404-isvc-92074-predictor-serving-cert" not found
Aug 05 17:45:51.660053 ip-10-0-135-2 kubenswrapper[2571]: E0805 17:45:51.660035    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/45df6046-112b-493c-b9bf-b746d05c3a21-proxy-tls podName:45df6046-112b-493c-b9bf-b746d05c3a21 nodeName:}" failed. No retries permitted until 2026-08-05 17:45:52.160017856 +0000 UTC m=+1979.527384182 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/45df6046-112b-493c-b9bf-b746d05c3a21-proxy-tls") pod "error-404-isvc-92074-predictor-589dc7f869-8mmn2" (UID: "45df6046-112b-493c-b9bf-b746d05c3a21") : secret "error-404-isvc-92074-predictor-serving-cert" not found
Aug 05 17:45:51.660279 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:45:51.660259    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/45df6046-112b-493c-b9bf-b746d05c3a21-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-92074-predictor-589dc7f869-8mmn2\" (UID: \"45df6046-112b-493c-b9bf-b746d05c3a21\") " pod="kserve-ci-e2e-test/error-404-isvc-92074-predictor-589dc7f869-8mmn2"
Aug 05 17:45:51.669136 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:45:51.669112    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-42gj7\" (UniqueName: \"kubernetes.io/projected/45df6046-112b-493c-b9bf-b746d05c3a21-kube-api-access-42gj7\") pod \"error-404-isvc-92074-predictor-589dc7f869-8mmn2\" (UID: \"45df6046-112b-493c-b9bf-b746d05c3a21\") " pod="kserve-ci-e2e-test/error-404-isvc-92074-predictor-589dc7f869-8mmn2"
Aug 05 17:45:51.788747 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:45:51.788570    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-92074-predictor-79d689c86d-w5pjl"]
Aug 05 17:45:51.791316 ip-10-0-135-2 kubenswrapper[2571]: W0805 17:45:51.791288    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod337b1675_3909_4f89_a905_35432d0ff577.slice/crio-95280739e6f2b2afdc05003d8de9614f9c3d78d0b7edce5cf1ca9f32a4bfc416 WatchSource:0}: Error finding container 95280739e6f2b2afdc05003d8de9614f9c3d78d0b7edce5cf1ca9f32a4bfc416: Status 404 returned error can't find the container with id 95280739e6f2b2afdc05003d8de9614f9c3d78d0b7edce5cf1ca9f32a4bfc416
Aug 05 17:45:52.164744 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:45:52.164666    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/45df6046-112b-493c-b9bf-b746d05c3a21-proxy-tls\") pod \"error-404-isvc-92074-predictor-589dc7f869-8mmn2\" (UID: \"45df6046-112b-493c-b9bf-b746d05c3a21\") " pod="kserve-ci-e2e-test/error-404-isvc-92074-predictor-589dc7f869-8mmn2"
Aug 05 17:45:52.166990 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:45:52.166967    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/45df6046-112b-493c-b9bf-b746d05c3a21-proxy-tls\") pod \"error-404-isvc-92074-predictor-589dc7f869-8mmn2\" (UID: \"45df6046-112b-493c-b9bf-b746d05c3a21\") " pod="kserve-ci-e2e-test/error-404-isvc-92074-predictor-589dc7f869-8mmn2"
Aug 05 17:45:52.312909 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:45:52.312880    2571 generic.go:356] "Generic (PLEG): container finished" podID="fbe776c3-891d-4fbf-acf1-6f4b72abda6c" containerID="dbcab6cedb0bc16ca8d5c649355fa7c0f97ba79cab24a764b3330d51f5f57f9e" exitCode=2
Aug 05 17:45:52.313094 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:45:52.312950    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-8dddc-predictor-66fd6cdf4-znnzz" event={"ID":"fbe776c3-891d-4fbf-acf1-6f4b72abda6c","Type":"ContainerDied","Data":"dbcab6cedb0bc16ca8d5c649355fa7c0f97ba79cab24a764b3330d51f5f57f9e"}
Aug 05 17:45:52.314616 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:45:52.314584    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-92074-predictor-79d689c86d-w5pjl" event={"ID":"337b1675-3909-4f89-a905-35432d0ff577","Type":"ContainerStarted","Data":"258ceef332b42dcbf23de9c286b2734731bd77a950a560fbd71a5a65bf135e8f"}
Aug 05 17:45:52.314942 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:45:52.314631    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-92074-predictor-79d689c86d-w5pjl" event={"ID":"337b1675-3909-4f89-a905-35432d0ff577","Type":"ContainerStarted","Data":"7a8b5c3ce5bb204b97598752cca1de3327c9c9f9cc49b6a14468cd8bce114211"}
Aug 05 17:45:52.314942 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:45:52.314662    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-92074-predictor-79d689c86d-w5pjl" event={"ID":"337b1675-3909-4f89-a905-35432d0ff577","Type":"ContainerStarted","Data":"95280739e6f2b2afdc05003d8de9614f9c3d78d0b7edce5cf1ca9f32a4bfc416"}
Aug 05 17:45:52.314942 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:45:52.314769    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-92074-predictor-79d689c86d-w5pjl"
Aug 05 17:45:52.316296 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:45:52.316273    2571 generic.go:356] "Generic (PLEG): container finished" podID="e8b2fd82-37b5-457e-ae17-47fadcce2a5e" containerID="3e2d115f8eafd58031a32787e6293dc6a80c6873300927324f5c22431429125e" exitCode=2
Aug 05 17:45:52.316400 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:45:52.316348    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-8dddc-predictor-5558b5787-9qrcb" event={"ID":"e8b2fd82-37b5-457e-ae17-47fadcce2a5e","Type":"ContainerDied","Data":"3e2d115f8eafd58031a32787e6293dc6a80c6873300927324f5c22431429125e"}
Aug 05 17:45:52.337496 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:45:52.337450    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/success-200-isvc-92074-predictor-79d689c86d-w5pjl" podStartSLOduration=1.33743673 podStartE2EDuration="1.33743673s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-05 17:45:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-05 17:45:52.335709333 +0000 UTC m=+1979.703075676" watchObservedRunningTime="2026-08-05 17:45:52.33743673 +0000 UTC m=+1979.704803073"
Aug 05 17:45:52.361893 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:45:52.361867    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-92074-predictor-589dc7f869-8mmn2"
Aug 05 17:45:52.503104 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:45:52.503076    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-92074-predictor-589dc7f869-8mmn2"]
Aug 05 17:45:52.504636 ip-10-0-135-2 kubenswrapper[2571]: W0805 17:45:52.504611    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod45df6046_112b_493c_b9bf_b746d05c3a21.slice/crio-4a7e8877fca8479efe7eaf62ba6050ed28f5fabccf461739ea104b0c568b9cd3 WatchSource:0}: Error finding container 4a7e8877fca8479efe7eaf62ba6050ed28f5fabccf461739ea104b0c568b9cd3: Status 404 returned error can't find the container with id 4a7e8877fca8479efe7eaf62ba6050ed28f5fabccf461739ea104b0c568b9cd3
Aug 05 17:45:53.322881 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:45:53.322842    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-92074-predictor-589dc7f869-8mmn2" event={"ID":"45df6046-112b-493c-b9bf-b746d05c3a21","Type":"ContainerStarted","Data":"d8aba550390f736c0547b246d73c65c9492807538fbf2aec54ecaa0781d46b6f"}
Aug 05 17:45:53.323303 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:45:53.323216    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-92074-predictor-79d689c86d-w5pjl"
Aug 05 17:45:53.323303 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:45:53.323250    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-92074-predictor-589dc7f869-8mmn2" event={"ID":"45df6046-112b-493c-b9bf-b746d05c3a21","Type":"ContainerStarted","Data":"fd4a5054b179d69f81577871b6c0d6fb709a755368fd464952d0de9fbd7412ed"}
Aug 05 17:45:53.323303 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:45:53.323278    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-92074-predictor-589dc7f869-8mmn2" event={"ID":"45df6046-112b-493c-b9bf-b746d05c3a21","Type":"ContainerStarted","Data":"4a7e8877fca8479efe7eaf62ba6050ed28f5fabccf461739ea104b0c568b9cd3"}
Aug 05 17:45:53.324689 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:45:53.324661    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-92074-predictor-79d689c86d-w5pjl" podUID="337b1675-3909-4f89-a905-35432d0ff577" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.42:8080: connect: connection refused"
Aug 05 17:45:53.344327 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:45:53.344269    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/error-404-isvc-92074-predictor-589dc7f869-8mmn2" podStartSLOduration=2.344249823 podStartE2EDuration="2.344249823s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-05 17:45:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-05 17:45:53.342466696 +0000 UTC m=+1980.709833033" watchObservedRunningTime="2026-08-05 17:45:53.344249823 +0000 UTC m=+1980.711616168"
Aug 05 17:45:53.987025 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:45:53.986988    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-8dddc-predictor-5558b5787-9qrcb" podUID="e8b2fd82-37b5-457e-ae17-47fadcce2a5e" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.39:8643/healthz\": dial tcp 10.134.0.39:8643: connect: connection refused"
Aug 05 17:45:53.987025 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:45:53.987007    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-8dddc-predictor-66fd6cdf4-znnzz" podUID="fbe776c3-891d-4fbf-acf1-6f4b72abda6c" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.38:8643/healthz\": dial tcp 10.134.0.38:8643: connect: connection refused"
Aug 05 17:45:53.991417 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:45:53.991398    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-8dddc-predictor-5558b5787-9qrcb" podUID="e8b2fd82-37b5-457e-ae17-47fadcce2a5e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.39:8080: connect: connection refused"
Aug 05 17:45:53.991417 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:45:53.991406    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-8dddc-predictor-66fd6cdf4-znnzz" podUID="fbe776c3-891d-4fbf-acf1-6f4b72abda6c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.38:8080: connect: connection refused"
Aug 05 17:45:54.327508 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:45:54.327412    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-92074-predictor-589dc7f869-8mmn2"
Aug 05 17:45:54.327508 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:45:54.327427    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-92074-predictor-79d689c86d-w5pjl" podUID="337b1675-3909-4f89-a905-35432d0ff577" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.42:8080: connect: connection refused"
Aug 05 17:45:54.327884 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:45:54.327526    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-92074-predictor-589dc7f869-8mmn2"
Aug 05 17:45:54.328889 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:45:54.328862    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-92074-predictor-589dc7f869-8mmn2" podUID="45df6046-112b-493c-b9bf-b746d05c3a21" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.43:8080: connect: connection refused"
Aug 05 17:45:54.563622 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:45:54.563600    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-8dddc-predictor-66fd6cdf4-znnzz"
Aug 05 17:45:54.687178 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:45:54.687153    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-8dddc-predictor-5558b5787-9qrcb"
Aug 05 17:45:54.687306 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:45:54.687207    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-s7h77\" (UniqueName: \"kubernetes.io/projected/fbe776c3-891d-4fbf-acf1-6f4b72abda6c-kube-api-access-s7h77\") pod \"fbe776c3-891d-4fbf-acf1-6f4b72abda6c\" (UID: \"fbe776c3-891d-4fbf-acf1-6f4b72abda6c\") "
Aug 05 17:45:54.687370 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:45:54.687302    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fbe776c3-891d-4fbf-acf1-6f4b72abda6c-kube-rbac-proxy-sar-config\") pod \"fbe776c3-891d-4fbf-acf1-6f4b72abda6c\" (UID: \"fbe776c3-891d-4fbf-acf1-6f4b72abda6c\") "
Aug 05 17:45:54.687370 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:45:54.687335    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fbe776c3-891d-4fbf-acf1-6f4b72abda6c-proxy-tls\") pod \"fbe776c3-891d-4fbf-acf1-6f4b72abda6c\" (UID: \"fbe776c3-891d-4fbf-acf1-6f4b72abda6c\") "
Aug 05 17:45:54.687708 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:45:54.687683    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fbe776c3-891d-4fbf-acf1-6f4b72abda6c-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "fbe776c3-891d-4fbf-acf1-6f4b72abda6c" (UID: "fbe776c3-891d-4fbf-acf1-6f4b72abda6c"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 05 17:45:54.689086 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:45:54.689060    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fbe776c3-891d-4fbf-acf1-6f4b72abda6c-kube-api-access-s7h77" (OuterVolumeSpecName: "kube-api-access-s7h77") pod "fbe776c3-891d-4fbf-acf1-6f4b72abda6c" (UID: "fbe776c3-891d-4fbf-acf1-6f4b72abda6c"). InnerVolumeSpecName "kube-api-access-s7h77". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 05 17:45:54.689256 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:45:54.689239    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fbe776c3-891d-4fbf-acf1-6f4b72abda6c-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "fbe776c3-891d-4fbf-acf1-6f4b72abda6c" (UID: "fbe776c3-891d-4fbf-acf1-6f4b72abda6c"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 05 17:45:54.788585 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:45:54.788559    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-x6x7k\" (UniqueName: \"kubernetes.io/projected/e8b2fd82-37b5-457e-ae17-47fadcce2a5e-kube-api-access-x6x7k\") pod \"e8b2fd82-37b5-457e-ae17-47fadcce2a5e\" (UID: \"e8b2fd82-37b5-457e-ae17-47fadcce2a5e\") "
Aug 05 17:45:54.788768 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:45:54.788601    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e8b2fd82-37b5-457e-ae17-47fadcce2a5e-kube-rbac-proxy-sar-config\") pod \"e8b2fd82-37b5-457e-ae17-47fadcce2a5e\" (UID: \"e8b2fd82-37b5-457e-ae17-47fadcce2a5e\") "
Aug 05 17:45:54.788768 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:45:54.788625    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e8b2fd82-37b5-457e-ae17-47fadcce2a5e-proxy-tls\") pod \"e8b2fd82-37b5-457e-ae17-47fadcce2a5e\" (UID: \"e8b2fd82-37b5-457e-ae17-47fadcce2a5e\") "
Aug 05 17:45:54.788889 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:45:54.788833    2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s7h77\" (UniqueName: \"kubernetes.io/projected/fbe776c3-891d-4fbf-acf1-6f4b72abda6c-kube-api-access-s7h77\") on node \"ip-10-0-135-2.ec2.internal\" DevicePath \"\""
Aug 05 17:45:54.788889 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:45:54.788851    2571 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fbe776c3-891d-4fbf-acf1-6f4b72abda6c-kube-rbac-proxy-sar-config\") on node \"ip-10-0-135-2.ec2.internal\" DevicePath \"\""
Aug 05 17:45:54.788889 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:45:54.788864    2571 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fbe776c3-891d-4fbf-acf1-6f4b72abda6c-proxy-tls\") on node \"ip-10-0-135-2.ec2.internal\" DevicePath \"\""
Aug 05 17:45:54.789026 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:45:54.788922    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e8b2fd82-37b5-457e-ae17-47fadcce2a5e-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "e8b2fd82-37b5-457e-ae17-47fadcce2a5e" (UID: "e8b2fd82-37b5-457e-ae17-47fadcce2a5e"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 05 17:45:54.790681 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:45:54.790662    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e8b2fd82-37b5-457e-ae17-47fadcce2a5e-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "e8b2fd82-37b5-457e-ae17-47fadcce2a5e" (UID: "e8b2fd82-37b5-457e-ae17-47fadcce2a5e"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 05 17:45:54.790814 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:45:54.790794    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e8b2fd82-37b5-457e-ae17-47fadcce2a5e-kube-api-access-x6x7k" (OuterVolumeSpecName: "kube-api-access-x6x7k") pod "e8b2fd82-37b5-457e-ae17-47fadcce2a5e" (UID: "e8b2fd82-37b5-457e-ae17-47fadcce2a5e"). InnerVolumeSpecName "kube-api-access-x6x7k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 05 17:45:54.889863 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:45:54.889799    2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x6x7k\" (UniqueName: \"kubernetes.io/projected/e8b2fd82-37b5-457e-ae17-47fadcce2a5e-kube-api-access-x6x7k\") on node \"ip-10-0-135-2.ec2.internal\" DevicePath \"\""
Aug 05 17:45:54.889863 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:45:54.889821    2571 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e8b2fd82-37b5-457e-ae17-47fadcce2a5e-kube-rbac-proxy-sar-config\") on node \"ip-10-0-135-2.ec2.internal\" DevicePath \"\""
Aug 05 17:45:54.889863 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:45:54.889830    2571 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e8b2fd82-37b5-457e-ae17-47fadcce2a5e-proxy-tls\") on node \"ip-10-0-135-2.ec2.internal\" DevicePath \"\""
Aug 05 17:45:55.334254 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:45:55.334223    2571 generic.go:356] "Generic (PLEG): container finished" podID="e8b2fd82-37b5-457e-ae17-47fadcce2a5e" containerID="b96ad862b6b46a99e41b0dd376e33c4bfee3ee23be666681737e863a6cc04422" exitCode=0
Aug 05 17:45:55.334646 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:45:55.334273    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-8dddc-predictor-5558b5787-9qrcb"
Aug 05 17:45:55.334646 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:45:55.334307    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-8dddc-predictor-5558b5787-9qrcb" event={"ID":"e8b2fd82-37b5-457e-ae17-47fadcce2a5e","Type":"ContainerDied","Data":"b96ad862b6b46a99e41b0dd376e33c4bfee3ee23be666681737e863a6cc04422"}
Aug 05 17:45:55.334646 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:45:55.334366    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-8dddc-predictor-5558b5787-9qrcb" event={"ID":"e8b2fd82-37b5-457e-ae17-47fadcce2a5e","Type":"ContainerDied","Data":"930c1ede5d90de21f97194769abe658b580e03337b0f5f8acf3613c0b45f47fe"}
Aug 05 17:45:55.334646 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:45:55.334426    2571 scope.go:117] "RemoveContainer" containerID="3e2d115f8eafd58031a32787e6293dc6a80c6873300927324f5c22431429125e"
Aug 05 17:45:55.336145 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:45:55.336119    2571 generic.go:356] "Generic (PLEG): container finished" podID="fbe776c3-891d-4fbf-acf1-6f4b72abda6c" containerID="e8d9d961405a7d16ebafea8a944e62ba9f391abbd3de5956e28a657d4ef6a6dd" exitCode=0
Aug 05 17:45:55.336265 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:45:55.336174    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-8dddc-predictor-66fd6cdf4-znnzz"
Aug 05 17:45:55.336265 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:45:55.336198    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-8dddc-predictor-66fd6cdf4-znnzz" event={"ID":"fbe776c3-891d-4fbf-acf1-6f4b72abda6c","Type":"ContainerDied","Data":"e8d9d961405a7d16ebafea8a944e62ba9f391abbd3de5956e28a657d4ef6a6dd"}
Aug 05 17:45:55.336265 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:45:55.336250    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-8dddc-predictor-66fd6cdf4-znnzz" event={"ID":"fbe776c3-891d-4fbf-acf1-6f4b72abda6c","Type":"ContainerDied","Data":"102d2e33667cd815bacb069a3193c3a6de8a25e09e56d0dfe0d529b8ef9c2a07"}
Aug 05 17:45:55.336739 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:45:55.336610    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-92074-predictor-589dc7f869-8mmn2" podUID="45df6046-112b-493c-b9bf-b746d05c3a21" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.43:8080: connect: connection refused"
Aug 05 17:45:55.343986 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:45:55.343947    2571 scope.go:117] "RemoveContainer" containerID="b96ad862b6b46a99e41b0dd376e33c4bfee3ee23be666681737e863a6cc04422"
Aug 05 17:45:55.354566 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:45:55.354545    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-8dddc-predictor-5558b5787-9qrcb"]
Aug 05 17:45:55.357671 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:45:55.357640    2571 scope.go:117] "RemoveContainer" containerID="3e2d115f8eafd58031a32787e6293dc6a80c6873300927324f5c22431429125e"
Aug 05 17:45:55.358179 ip-10-0-135-2 kubenswrapper[2571]: E0805 17:45:55.358158    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3e2d115f8eafd58031a32787e6293dc6a80c6873300927324f5c22431429125e\": container with ID starting with 3e2d115f8eafd58031a32787e6293dc6a80c6873300927324f5c22431429125e not found: ID does not exist" containerID="3e2d115f8eafd58031a32787e6293dc6a80c6873300927324f5c22431429125e"
Aug 05 17:45:55.358271 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:45:55.358191    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3e2d115f8eafd58031a32787e6293dc6a80c6873300927324f5c22431429125e"} err="failed to get container status \"3e2d115f8eafd58031a32787e6293dc6a80c6873300927324f5c22431429125e\": rpc error: code = NotFound desc = could not find container \"3e2d115f8eafd58031a32787e6293dc6a80c6873300927324f5c22431429125e\": container with ID starting with 3e2d115f8eafd58031a32787e6293dc6a80c6873300927324f5c22431429125e not found: ID does not exist"
Aug 05 17:45:55.358271 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:45:55.358217    2571 scope.go:117] "RemoveContainer" containerID="b96ad862b6b46a99e41b0dd376e33c4bfee3ee23be666681737e863a6cc04422"
Aug 05 17:45:55.358550 ip-10-0-135-2 kubenswrapper[2571]: E0805 17:45:55.358533    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b96ad862b6b46a99e41b0dd376e33c4bfee3ee23be666681737e863a6cc04422\": container with ID starting with b96ad862b6b46a99e41b0dd376e33c4bfee3ee23be666681737e863a6cc04422 not found: ID does not exist" containerID="b96ad862b6b46a99e41b0dd376e33c4bfee3ee23be666681737e863a6cc04422"
Aug 05 17:45:55.358616 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:45:55.358556    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b96ad862b6b46a99e41b0dd376e33c4bfee3ee23be666681737e863a6cc04422"} err="failed to get container status \"b96ad862b6b46a99e41b0dd376e33c4bfee3ee23be666681737e863a6cc04422\": rpc error: code = NotFound desc = could not find container \"b96ad862b6b46a99e41b0dd376e33c4bfee3ee23be666681737e863a6cc04422\": container with ID starting with b96ad862b6b46a99e41b0dd376e33c4bfee3ee23be666681737e863a6cc04422 not found: ID does not exist"
Aug 05 17:45:55.358616 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:45:55.358573    2571 scope.go:117] "RemoveContainer" containerID="dbcab6cedb0bc16ca8d5c649355fa7c0f97ba79cab24a764b3330d51f5f57f9e"
Aug 05 17:45:55.359743 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:45:55.359722    2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-8dddc-predictor-5558b5787-9qrcb"]
Aug 05 17:45:55.365885 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:45:55.365870    2571 scope.go:117] "RemoveContainer" containerID="e8d9d961405a7d16ebafea8a944e62ba9f391abbd3de5956e28a657d4ef6a6dd"
Aug 05 17:45:55.371648 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:45:55.371628    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-8dddc-predictor-66fd6cdf4-znnzz"]
Aug 05 17:45:55.373088 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:45:55.373074    2571 scope.go:117] "RemoveContainer" containerID="dbcab6cedb0bc16ca8d5c649355fa7c0f97ba79cab24a764b3330d51f5f57f9e"
Aug 05 17:45:55.373337 ip-10-0-135-2 kubenswrapper[2571]: E0805 17:45:55.373318    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dbcab6cedb0bc16ca8d5c649355fa7c0f97ba79cab24a764b3330d51f5f57f9e\": container with ID starting with dbcab6cedb0bc16ca8d5c649355fa7c0f97ba79cab24a764b3330d51f5f57f9e not found: ID does not exist" containerID="dbcab6cedb0bc16ca8d5c649355fa7c0f97ba79cab24a764b3330d51f5f57f9e"
Aug 05 17:45:55.373413 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:45:55.373346    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dbcab6cedb0bc16ca8d5c649355fa7c0f97ba79cab24a764b3330d51f5f57f9e"} err="failed to get container status \"dbcab6cedb0bc16ca8d5c649355fa7c0f97ba79cab24a764b3330d51f5f57f9e\": rpc error: code = NotFound desc = could not find container \"dbcab6cedb0bc16ca8d5c649355fa7c0f97ba79cab24a764b3330d51f5f57f9e\": container with ID starting with dbcab6cedb0bc16ca8d5c649355fa7c0f97ba79cab24a764b3330d51f5f57f9e not found: ID does not exist"
Aug 05 17:45:55.373413 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:45:55.373362    2571 scope.go:117] "RemoveContainer" containerID="e8d9d961405a7d16ebafea8a944e62ba9f391abbd3de5956e28a657d4ef6a6dd"
Aug 05 17:45:55.373610 ip-10-0-135-2 kubenswrapper[2571]: E0805 17:45:55.373593    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e8d9d961405a7d16ebafea8a944e62ba9f391abbd3de5956e28a657d4ef6a6dd\": container with ID starting with e8d9d961405a7d16ebafea8a944e62ba9f391abbd3de5956e28a657d4ef6a6dd not found: ID does not exist" containerID="e8d9d961405a7d16ebafea8a944e62ba9f391abbd3de5956e28a657d4ef6a6dd"
Aug 05 17:45:55.373657 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:45:55.373616    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e8d9d961405a7d16ebafea8a944e62ba9f391abbd3de5956e28a657d4ef6a6dd"} err="failed to get container status \"e8d9d961405a7d16ebafea8a944e62ba9f391abbd3de5956e28a657d4ef6a6dd\": rpc error: code = NotFound desc = could not find container \"e8d9d961405a7d16ebafea8a944e62ba9f391abbd3de5956e28a657d4ef6a6dd\": container with ID starting with e8d9d961405a7d16ebafea8a944e62ba9f391abbd3de5956e28a657d4ef6a6dd not found: ID does not exist"
Aug 05 17:45:55.376828 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:45:55.376808    2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-8dddc-predictor-66fd6cdf4-znnzz"]
Aug 05 17:45:57.136709 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:45:57.136678    2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e8b2fd82-37b5-457e-ae17-47fadcce2a5e" path="/var/lib/kubelet/pods/e8b2fd82-37b5-457e-ae17-47fadcce2a5e/volumes"
Aug 05 17:45:57.137102 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:45:57.137088    2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fbe776c3-891d-4fbf-acf1-6f4b72abda6c" path="/var/lib/kubelet/pods/fbe776c3-891d-4fbf-acf1-6f4b72abda6c/volumes"
Aug 05 17:45:59.331965 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:45:59.331935    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-92074-predictor-79d689c86d-w5pjl"
Aug 05 17:45:59.332948 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:45:59.332885    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-92074-predictor-79d689c86d-w5pjl" podUID="337b1675-3909-4f89-a905-35432d0ff577" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.42:8080: connect: connection refused"
Aug 05 17:46:00.340520 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:46:00.340494    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-92074-predictor-589dc7f869-8mmn2"
Aug 05 17:46:00.340989 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:46:00.340891    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-92074-predictor-589dc7f869-8mmn2" podUID="45df6046-112b-493c-b9bf-b746d05c3a21" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.43:8080: connect: connection refused"
Aug 05 17:46:09.332360 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:46:09.332319    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-92074-predictor-79d689c86d-w5pjl" podUID="337b1675-3909-4f89-a905-35432d0ff577" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.42:8080: connect: connection refused"
Aug 05 17:46:10.341748 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:46:10.341708    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-92074-predictor-589dc7f869-8mmn2" podUID="45df6046-112b-493c-b9bf-b746d05c3a21" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.43:8080: connect: connection refused"
Aug 05 17:46:19.333390 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:46:19.333336    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-92074-predictor-79d689c86d-w5pjl" podUID="337b1675-3909-4f89-a905-35432d0ff577" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.42:8080: connect: connection refused"
Aug 05 17:46:20.341103 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:46:20.341066    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-92074-predictor-589dc7f869-8mmn2" podUID="45df6046-112b-493c-b9bf-b746d05c3a21" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.43:8080: connect: connection refused"
Aug 05 17:46:29.332392 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:46:29.332342    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-92074-predictor-79d689c86d-w5pjl" podUID="337b1675-3909-4f89-a905-35432d0ff577" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.42:8080: connect: connection refused"
Aug 05 17:46:30.340957 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:46:30.340911    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-92074-predictor-589dc7f869-8mmn2" podUID="45df6046-112b-493c-b9bf-b746d05c3a21" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.43:8080: connect: connection refused"
Aug 05 17:46:39.333264 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:46:39.333235    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-92074-predictor-79d689c86d-w5pjl"
Aug 05 17:46:40.342229 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:46:40.342201    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-92074-predictor-589dc7f869-8mmn2"
Aug 05 17:46:51.218503 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:46:51.218468    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-e149b-predictor-5f5fbfc7cd-vq5fz"]
Aug 05 17:46:51.218950 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:46:51.218664    2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-e149b-predictor-5f5fbfc7cd-vq5fz" podUID="4d68887a-7de6-4f30-a190-e447efb44fbb" containerName="kserve-container" containerID="cri-o://7b2d132afcdd21c1bbdda244cfc1ecdbeccda0d22aed4eb60aa76220c6b2cb7e" gracePeriod=30
Aug 05 17:46:51.218950 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:46:51.218726    2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-e149b-predictor-5f5fbfc7cd-vq5fz" podUID="4d68887a-7de6-4f30-a190-e447efb44fbb" containerName="kube-rbac-proxy" containerID="cri-o://8f94407c18f52a938b716af90bfd6fd64eacb4faffe02684f83be83f0291082b" gracePeriod=30
Aug 05 17:46:51.297798 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:46:51.297767    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-e149b-predictor-6cc79c7d67-7r5pf"]
Aug 05 17:46:51.298071 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:46:51.298044    2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-e149b-predictor-6cc79c7d67-7r5pf" podUID="b528315e-f99c-4518-bdf1-87a2863c69a6" containerName="kserve-container" containerID="cri-o://018eccd20db267e475a715d33621f2c67123455028fe1ce983553d7eb400b911" gracePeriod=30
Aug 05 17:46:51.298158 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:46:51.298110    2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-e149b-predictor-6cc79c7d67-7r5pf" podUID="b528315e-f99c-4518-bdf1-87a2863c69a6" containerName="kube-rbac-proxy" containerID="cri-o://51761bed037f05547f7d368956e335f28a0ea8a4ed69c93169a3c25ce90168d9" gracePeriod=30
Aug 05 17:46:51.336098 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:46:51.336066    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-741a0-predictor-77484d674-xt592"]
Aug 05 17:46:51.336724 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:46:51.336701    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fbe776c3-891d-4fbf-acf1-6f4b72abda6c" containerName="kube-rbac-proxy"
Aug 05 17:46:51.336817 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:46:51.336727    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="fbe776c3-891d-4fbf-acf1-6f4b72abda6c" containerName="kube-rbac-proxy"
Aug 05 17:46:51.336817 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:46:51.336749    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e8b2fd82-37b5-457e-ae17-47fadcce2a5e" containerName="kserve-container"
Aug 05 17:46:51.336817 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:46:51.336758    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="e8b2fd82-37b5-457e-ae17-47fadcce2a5e" containerName="kserve-container"
Aug 05 17:46:51.336817 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:46:51.336786    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fbe776c3-891d-4fbf-acf1-6f4b72abda6c" containerName="kserve-container"
Aug 05 17:46:51.336817 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:46:51.336795    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="fbe776c3-891d-4fbf-acf1-6f4b72abda6c" containerName="kserve-container"
Aug 05 17:46:51.336817 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:46:51.336817    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e8b2fd82-37b5-457e-ae17-47fadcce2a5e" containerName="kube-rbac-proxy"
Aug 05 17:46:51.337135 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:46:51.336826    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="e8b2fd82-37b5-457e-ae17-47fadcce2a5e" containerName="kube-rbac-proxy"
Aug 05 17:46:51.337135 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:46:51.336911    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="e8b2fd82-37b5-457e-ae17-47fadcce2a5e" containerName="kserve-container"
Aug 05 17:46:51.337135 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:46:51.336922    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="fbe776c3-891d-4fbf-acf1-6f4b72abda6c" containerName="kube-rbac-proxy"
Aug 05 17:46:51.337135 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:46:51.336934    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="e8b2fd82-37b5-457e-ae17-47fadcce2a5e" containerName="kube-rbac-proxy"
Aug 05 17:46:51.337135 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:46:51.336951    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="fbe776c3-891d-4fbf-acf1-6f4b72abda6c" containerName="kserve-container"
Aug 05 17:46:51.340417 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:46:51.340397    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-741a0-predictor-77484d674-xt592"
Aug 05 17:46:51.342986 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:46:51.342965    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-741a0-predictor-serving-cert\""
Aug 05 17:46:51.343563 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:46:51.343541    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-741a0-kube-rbac-proxy-sar-config\""
Aug 05 17:46:51.351577 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:46:51.351557    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-741a0-predictor-77484d674-xt592"]
Aug 05 17:46:51.435170 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:46:51.435144    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/30ca866f-36b3-4229-9126-d138dd30da9f-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-741a0-predictor-77484d674-xt592\" (UID: \"30ca866f-36b3-4229-9126-d138dd30da9f\") " pod="kserve-ci-e2e-test/success-200-isvc-741a0-predictor-77484d674-xt592"
Aug 05 17:46:51.435274 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:46:51.435180    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/30ca866f-36b3-4229-9126-d138dd30da9f-proxy-tls\") pod \"success-200-isvc-741a0-predictor-77484d674-xt592\" (UID: \"30ca866f-36b3-4229-9126-d138dd30da9f\") " pod="kserve-ci-e2e-test/success-200-isvc-741a0-predictor-77484d674-xt592"
Aug 05 17:46:51.435274 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:46:51.435257    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d77p8\" (UniqueName: \"kubernetes.io/projected/30ca866f-36b3-4229-9126-d138dd30da9f-kube-api-access-d77p8\") pod \"success-200-isvc-741a0-predictor-77484d674-xt592\" (UID: \"30ca866f-36b3-4229-9126-d138dd30da9f\") " pod="kserve-ci-e2e-test/success-200-isvc-741a0-predictor-77484d674-xt592"
Aug 05 17:46:51.449133 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:46:51.449108    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-741a0-predictor-6c788cfb87-2t5vl"]
Aug 05 17:46:51.452954 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:46:51.452937    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-741a0-predictor-6c788cfb87-2t5vl"
Aug 05 17:46:51.455450 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:46:51.455427    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-741a0-predictor-serving-cert\""
Aug 05 17:46:51.455532 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:46:51.455430    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-741a0-kube-rbac-proxy-sar-config\""
Aug 05 17:46:51.463495 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:46:51.463471    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-741a0-predictor-6c788cfb87-2t5vl"]
Aug 05 17:46:51.535980 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:46:51.535949    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gp9l2\" (UniqueName: \"kubernetes.io/projected/04ea2238-8ad7-4bc1-9021-e17922673b58-kube-api-access-gp9l2\") pod \"error-404-isvc-741a0-predictor-6c788cfb87-2t5vl\" (UID: \"04ea2238-8ad7-4bc1-9021-e17922673b58\") " pod="kserve-ci-e2e-test/error-404-isvc-741a0-predictor-6c788cfb87-2t5vl"
Aug 05 17:46:51.536140 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:46:51.535986    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d77p8\" (UniqueName: \"kubernetes.io/projected/30ca866f-36b3-4229-9126-d138dd30da9f-kube-api-access-d77p8\") pod \"success-200-isvc-741a0-predictor-77484d674-xt592\" (UID: \"30ca866f-36b3-4229-9126-d138dd30da9f\") " pod="kserve-ci-e2e-test/success-200-isvc-741a0-predictor-77484d674-xt592"
Aug 05 17:46:51.536140 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:46:51.536066    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/30ca866f-36b3-4229-9126-d138dd30da9f-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-741a0-predictor-77484d674-xt592\" (UID: \"30ca866f-36b3-4229-9126-d138dd30da9f\") " pod="kserve-ci-e2e-test/success-200-isvc-741a0-predictor-77484d674-xt592"
Aug 05 17:46:51.536140 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:46:51.536095    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/30ca866f-36b3-4229-9126-d138dd30da9f-proxy-tls\") pod \"success-200-isvc-741a0-predictor-77484d674-xt592\" (UID: \"30ca866f-36b3-4229-9126-d138dd30da9f\") " pod="kserve-ci-e2e-test/success-200-isvc-741a0-predictor-77484d674-xt592"
Aug 05 17:46:51.536140 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:46:51.536120    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/04ea2238-8ad7-4bc1-9021-e17922673b58-proxy-tls\") pod \"error-404-isvc-741a0-predictor-6c788cfb87-2t5vl\" (UID: \"04ea2238-8ad7-4bc1-9021-e17922673b58\") " pod="kserve-ci-e2e-test/error-404-isvc-741a0-predictor-6c788cfb87-2t5vl"
Aug 05 17:46:51.536331 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:46:51.536162    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/04ea2238-8ad7-4bc1-9021-e17922673b58-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-741a0-predictor-6c788cfb87-2t5vl\" (UID: \"04ea2238-8ad7-4bc1-9021-e17922673b58\") " pod="kserve-ci-e2e-test/error-404-isvc-741a0-predictor-6c788cfb87-2t5vl"
Aug 05 17:46:51.536815 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:46:51.536780    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/30ca866f-36b3-4229-9126-d138dd30da9f-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-741a0-predictor-77484d674-xt592\" (UID: \"30ca866f-36b3-4229-9126-d138dd30da9f\") " pod="kserve-ci-e2e-test/success-200-isvc-741a0-predictor-77484d674-xt592"
Aug 05 17:46:51.538670 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:46:51.538652    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/30ca866f-36b3-4229-9126-d138dd30da9f-proxy-tls\") pod \"success-200-isvc-741a0-predictor-77484d674-xt592\" (UID: \"30ca866f-36b3-4229-9126-d138dd30da9f\") " pod="kserve-ci-e2e-test/success-200-isvc-741a0-predictor-77484d674-xt592"
Aug 05 17:46:51.544342 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:46:51.544319    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d77p8\" (UniqueName: \"kubernetes.io/projected/30ca866f-36b3-4229-9126-d138dd30da9f-kube-api-access-d77p8\") pod \"success-200-isvc-741a0-predictor-77484d674-xt592\" (UID: \"30ca866f-36b3-4229-9126-d138dd30da9f\") " pod="kserve-ci-e2e-test/success-200-isvc-741a0-predictor-77484d674-xt592"
Aug 05 17:46:51.601611 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:46:51.601586    2571 generic.go:356] "Generic (PLEG): container finished" podID="b528315e-f99c-4518-bdf1-87a2863c69a6" containerID="51761bed037f05547f7d368956e335f28a0ea8a4ed69c93169a3c25ce90168d9" exitCode=2
Aug 05 17:46:51.601707 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:46:51.601661    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-e149b-predictor-6cc79c7d67-7r5pf" event={"ID":"b528315e-f99c-4518-bdf1-87a2863c69a6","Type":"ContainerDied","Data":"51761bed037f05547f7d368956e335f28a0ea8a4ed69c93169a3c25ce90168d9"}
Aug 05 17:46:51.603403 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:46:51.603368    2571 generic.go:356] "Generic (PLEG): container finished" podID="4d68887a-7de6-4f30-a190-e447efb44fbb" containerID="8f94407c18f52a938b716af90bfd6fd64eacb4faffe02684f83be83f0291082b" exitCode=2
Aug 05 17:46:51.603484 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:46:51.603398    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-e149b-predictor-5f5fbfc7cd-vq5fz" event={"ID":"4d68887a-7de6-4f30-a190-e447efb44fbb","Type":"ContainerDied","Data":"8f94407c18f52a938b716af90bfd6fd64eacb4faffe02684f83be83f0291082b"}
Aug 05 17:46:51.643350 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:46:51.637778    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/04ea2238-8ad7-4bc1-9021-e17922673b58-proxy-tls\") pod \"error-404-isvc-741a0-predictor-6c788cfb87-2t5vl\" (UID: \"04ea2238-8ad7-4bc1-9021-e17922673b58\") " pod="kserve-ci-e2e-test/error-404-isvc-741a0-predictor-6c788cfb87-2t5vl"
Aug 05 17:46:51.643350 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:46:51.637899    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/04ea2238-8ad7-4bc1-9021-e17922673b58-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-741a0-predictor-6c788cfb87-2t5vl\" (UID: \"04ea2238-8ad7-4bc1-9021-e17922673b58\") " pod="kserve-ci-e2e-test/error-404-isvc-741a0-predictor-6c788cfb87-2t5vl"
Aug 05 17:46:51.643350 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:46:51.638044    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gp9l2\" (UniqueName: \"kubernetes.io/projected/04ea2238-8ad7-4bc1-9021-e17922673b58-kube-api-access-gp9l2\") pod \"error-404-isvc-741a0-predictor-6c788cfb87-2t5vl\" (UID: \"04ea2238-8ad7-4bc1-9021-e17922673b58\") " pod="kserve-ci-e2e-test/error-404-isvc-741a0-predictor-6c788cfb87-2t5vl"
Aug 05 17:46:51.643350 ip-10-0-135-2 kubenswrapper[2571]: E0805 17:46:51.638557    2571 secret.go:189] Couldn't get secret kserve-ci-e2e-test/error-404-isvc-741a0-predictor-serving-cert: secret "error-404-isvc-741a0-predictor-serving-cert" not found
Aug 05 17:46:51.643350 ip-10-0-135-2 kubenswrapper[2571]: E0805 17:46:51.638633    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/04ea2238-8ad7-4bc1-9021-e17922673b58-proxy-tls podName:04ea2238-8ad7-4bc1-9021-e17922673b58 nodeName:}" failed. No retries permitted until 2026-08-05 17:46:52.138612372 +0000 UTC m=+2039.505978692 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/04ea2238-8ad7-4bc1-9021-e17922673b58-proxy-tls") pod "error-404-isvc-741a0-predictor-6c788cfb87-2t5vl" (UID: "04ea2238-8ad7-4bc1-9021-e17922673b58") : secret "error-404-isvc-741a0-predictor-serving-cert" not found
Aug 05 17:46:51.643350 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:46:51.639242    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/04ea2238-8ad7-4bc1-9021-e17922673b58-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-741a0-predictor-6c788cfb87-2t5vl\" (UID: \"04ea2238-8ad7-4bc1-9021-e17922673b58\") " pod="kserve-ci-e2e-test/error-404-isvc-741a0-predictor-6c788cfb87-2t5vl"
Aug 05 17:46:51.649644 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:46:51.649620    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gp9l2\" (UniqueName: \"kubernetes.io/projected/04ea2238-8ad7-4bc1-9021-e17922673b58-kube-api-access-gp9l2\") pod \"error-404-isvc-741a0-predictor-6c788cfb87-2t5vl\" (UID: \"04ea2238-8ad7-4bc1-9021-e17922673b58\") " pod="kserve-ci-e2e-test/error-404-isvc-741a0-predictor-6c788cfb87-2t5vl"
Aug 05 17:46:51.651471 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:46:51.651441    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-741a0-predictor-77484d674-xt592"
Aug 05 17:46:51.783875 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:46:51.783851    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-741a0-predictor-77484d674-xt592"]
Aug 05 17:46:51.786339 ip-10-0-135-2 kubenswrapper[2571]: W0805 17:46:51.786283    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod30ca866f_36b3_4229_9126_d138dd30da9f.slice/crio-1fcf45720e03332b42399961aca9785b9d1cca0495a427a86d895400b0a9d6c3 WatchSource:0}: Error finding container 1fcf45720e03332b42399961aca9785b9d1cca0495a427a86d895400b0a9d6c3: Status 404 returned error can't find the container with id 1fcf45720e03332b42399961aca9785b9d1cca0495a427a86d895400b0a9d6c3
Aug 05 17:46:52.141948 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:46:52.141853    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/04ea2238-8ad7-4bc1-9021-e17922673b58-proxy-tls\") pod \"error-404-isvc-741a0-predictor-6c788cfb87-2t5vl\" (UID: \"04ea2238-8ad7-4bc1-9021-e17922673b58\") " pod="kserve-ci-e2e-test/error-404-isvc-741a0-predictor-6c788cfb87-2t5vl"
Aug 05 17:46:52.144334 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:46:52.144306    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/04ea2238-8ad7-4bc1-9021-e17922673b58-proxy-tls\") pod \"error-404-isvc-741a0-predictor-6c788cfb87-2t5vl\" (UID: \"04ea2238-8ad7-4bc1-9021-e17922673b58\") " pod="kserve-ci-e2e-test/error-404-isvc-741a0-predictor-6c788cfb87-2t5vl"
Aug 05 17:46:52.363332 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:46:52.363306    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-741a0-predictor-6c788cfb87-2t5vl"
Aug 05 17:46:52.496092 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:46:52.496066    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-741a0-predictor-6c788cfb87-2t5vl"]
Aug 05 17:46:52.498750 ip-10-0-135-2 kubenswrapper[2571]: W0805 17:46:52.498726    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod04ea2238_8ad7_4bc1_9021_e17922673b58.slice/crio-dcd8d3d9653df8db7a013d300d067e66be93f5306ae0e17d7ea23cb034089045 WatchSource:0}: Error finding container dcd8d3d9653df8db7a013d300d067e66be93f5306ae0e17d7ea23cb034089045: Status 404 returned error can't find the container with id dcd8d3d9653df8db7a013d300d067e66be93f5306ae0e17d7ea23cb034089045
Aug 05 17:46:52.610537 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:46:52.610510    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-741a0-predictor-6c788cfb87-2t5vl" event={"ID":"04ea2238-8ad7-4bc1-9021-e17922673b58","Type":"ContainerStarted","Data":"e4f99ff07f2672d0b6e789b7e306335817c6aa20c0d3a64f8cc6f4b31453be03"}
Aug 05 17:46:52.610639 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:46:52.610569    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-741a0-predictor-6c788cfb87-2t5vl" event={"ID":"04ea2238-8ad7-4bc1-9021-e17922673b58","Type":"ContainerStarted","Data":"dcd8d3d9653df8db7a013d300d067e66be93f5306ae0e17d7ea23cb034089045"}
Aug 05 17:46:52.612176 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:46:52.612145    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-741a0-predictor-77484d674-xt592" event={"ID":"30ca866f-36b3-4229-9126-d138dd30da9f","Type":"ContainerStarted","Data":"142fa6953c9d9653a5008612e1b9bdf57cc8d62d3d42fb91fa2364ecf4bdf441"}
Aug 05 17:46:52.612294 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:46:52.612203    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-741a0-predictor-77484d674-xt592" event={"ID":"30ca866f-36b3-4229-9126-d138dd30da9f","Type":"ContainerStarted","Data":"becf095cf95ec3d57d9c36da8df12a1db8e814a989c511d41a41f47e0a1630ac"}
Aug 05 17:46:52.612294 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:46:52.612250    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-741a0-predictor-77484d674-xt592" event={"ID":"30ca866f-36b3-4229-9126-d138dd30da9f","Type":"ContainerStarted","Data":"1fcf45720e03332b42399961aca9785b9d1cca0495a427a86d895400b0a9d6c3"}
Aug 05 17:46:52.612294 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:46:52.612287    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-741a0-predictor-77484d674-xt592"
Aug 05 17:46:52.612473 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:46:52.612335    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-741a0-predictor-77484d674-xt592"
Aug 05 17:46:52.613305 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:46:52.613279    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-741a0-predictor-77484d674-xt592" podUID="30ca866f-36b3-4229-9126-d138dd30da9f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.44:8080: connect: connection refused"
Aug 05 17:46:52.633149 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:46:52.633105    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/success-200-isvc-741a0-predictor-77484d674-xt592" podStartSLOduration=1.633090886 podStartE2EDuration="1.633090886s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-05 17:46:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-05 17:46:52.632034129 +0000 UTC m=+2039.999400471" watchObservedRunningTime="2026-08-05 17:46:52.633090886 +0000 UTC m=+2040.000457230"
Aug 05 17:46:53.618699 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:46:53.618658    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-741a0-predictor-6c788cfb87-2t5vl" event={"ID":"04ea2238-8ad7-4bc1-9021-e17922673b58","Type":"ContainerStarted","Data":"cef1a82358db665d62f49227fce1b8d19be3f73f80fb8f2c0640311cbd05ff50"}
Aug 05 17:46:53.619156 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:46:53.619076    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-741a0-predictor-77484d674-xt592" podUID="30ca866f-36b3-4229-9126-d138dd30da9f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.44:8080: connect: connection refused"
Aug 05 17:46:53.639671 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:46:53.639620    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/error-404-isvc-741a0-predictor-6c788cfb87-2t5vl" podStartSLOduration=2.639604307 podStartE2EDuration="2.639604307s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-05 17:46:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-05 17:46:53.638096626 +0000 UTC m=+2041.005462969" watchObservedRunningTime="2026-08-05 17:46:53.639604307 +0000 UTC m=+2041.006970649"
Aug 05 17:46:54.259205 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:46:54.259172    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-e149b-predictor-6cc79c7d67-7r5pf" podUID="b528315e-f99c-4518-bdf1-87a2863c69a6" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.41:8643/healthz\": dial tcp 10.134.0.41:8643: connect: connection refused"
Aug 05 17:46:54.263609 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:46:54.263585    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-e149b-predictor-6cc79c7d67-7r5pf" podUID="b528315e-f99c-4518-bdf1-87a2863c69a6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.41:8080: connect: connection refused"
Aug 05 17:46:54.521547 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:46:54.521526    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-e149b-predictor-5f5fbfc7cd-vq5fz"
Aug 05 17:46:54.624056 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:46:54.624025    2571 generic.go:356] "Generic (PLEG): container finished" podID="4d68887a-7de6-4f30-a190-e447efb44fbb" containerID="7b2d132afcdd21c1bbdda244cfc1ecdbeccda0d22aed4eb60aa76220c6b2cb7e" exitCode=0
Aug 05 17:46:54.624475 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:46:54.624088    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-e149b-predictor-5f5fbfc7cd-vq5fz"
Aug 05 17:46:54.624475 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:46:54.624109    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-e149b-predictor-5f5fbfc7cd-vq5fz" event={"ID":"4d68887a-7de6-4f30-a190-e447efb44fbb","Type":"ContainerDied","Data":"7b2d132afcdd21c1bbdda244cfc1ecdbeccda0d22aed4eb60aa76220c6b2cb7e"}
Aug 05 17:46:54.624475 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:46:54.624174    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-e149b-predictor-5f5fbfc7cd-vq5fz" event={"ID":"4d68887a-7de6-4f30-a190-e447efb44fbb","Type":"ContainerDied","Data":"b5989bad421304c90c05a21646be2cc38ed321fb4ca0877f7f025841f27f4606"}
Aug 05 17:46:54.624475 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:46:54.624211    2571 scope.go:117] "RemoveContainer" containerID="8f94407c18f52a938b716af90bfd6fd64eacb4faffe02684f83be83f0291082b"
Aug 05 17:46:54.626174 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:46:54.626142    2571 generic.go:356] "Generic (PLEG): container finished" podID="b528315e-f99c-4518-bdf1-87a2863c69a6" containerID="018eccd20db267e475a715d33621f2c67123455028fe1ce983553d7eb400b911" exitCode=0
Aug 05 17:46:54.626278 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:46:54.626193    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-e149b-predictor-6cc79c7d67-7r5pf" event={"ID":"b528315e-f99c-4518-bdf1-87a2863c69a6","Type":"ContainerDied","Data":"018eccd20db267e475a715d33621f2c67123455028fe1ce983553d7eb400b911"}
Aug 05 17:46:54.626590 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:46:54.626573    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-741a0-predictor-6c788cfb87-2t5vl"
Aug 05 17:46:54.626694 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:46:54.626626    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-741a0-predictor-6c788cfb87-2t5vl"
Aug 05 17:46:54.627653 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:46:54.627624    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-741a0-predictor-6c788cfb87-2t5vl" podUID="04ea2238-8ad7-4bc1-9021-e17922673b58" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.45:8080: connect: connection refused"
Aug 05 17:46:54.632909 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:46:54.632893    2571 scope.go:117] "RemoveContainer" containerID="7b2d132afcdd21c1bbdda244cfc1ecdbeccda0d22aed4eb60aa76220c6b2cb7e"
Aug 05 17:46:54.641847 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:46:54.641825    2571 scope.go:117] "RemoveContainer" containerID="8f94407c18f52a938b716af90bfd6fd64eacb4faffe02684f83be83f0291082b"
Aug 05 17:46:54.642155 ip-10-0-135-2 kubenswrapper[2571]: E0805 17:46:54.642129    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8f94407c18f52a938b716af90bfd6fd64eacb4faffe02684f83be83f0291082b\": container with ID starting with 8f94407c18f52a938b716af90bfd6fd64eacb4faffe02684f83be83f0291082b not found: ID does not exist" containerID="8f94407c18f52a938b716af90bfd6fd64eacb4faffe02684f83be83f0291082b"
Aug 05 17:46:54.642257 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:46:54.642166    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8f94407c18f52a938b716af90bfd6fd64eacb4faffe02684f83be83f0291082b"} err="failed to get container status \"8f94407c18f52a938b716af90bfd6fd64eacb4faffe02684f83be83f0291082b\": rpc error: code = NotFound desc = could not find container \"8f94407c18f52a938b716af90bfd6fd64eacb4faffe02684f83be83f0291082b\": container with ID starting with 8f94407c18f52a938b716af90bfd6fd64eacb4faffe02684f83be83f0291082b not found: ID does not exist"
Aug 05 17:46:54.642257 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:46:54.642192    2571 scope.go:117] "RemoveContainer" containerID="7b2d132afcdd21c1bbdda244cfc1ecdbeccda0d22aed4eb60aa76220c6b2cb7e"
Aug 05 17:46:54.642534 ip-10-0-135-2 kubenswrapper[2571]: E0805 17:46:54.642514    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7b2d132afcdd21c1bbdda244cfc1ecdbeccda0d22aed4eb60aa76220c6b2cb7e\": container with ID starting with 7b2d132afcdd21c1bbdda244cfc1ecdbeccda0d22aed4eb60aa76220c6b2cb7e not found: ID does not exist" containerID="7b2d132afcdd21c1bbdda244cfc1ecdbeccda0d22aed4eb60aa76220c6b2cb7e"
Aug 05 17:46:54.642598 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:46:54.642553    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7b2d132afcdd21c1bbdda244cfc1ecdbeccda0d22aed4eb60aa76220c6b2cb7e"} err="failed to get container status \"7b2d132afcdd21c1bbdda244cfc1ecdbeccda0d22aed4eb60aa76220c6b2cb7e\": rpc error: code = NotFound desc = could not find container \"7b2d132afcdd21c1bbdda244cfc1ecdbeccda0d22aed4eb60aa76220c6b2cb7e\": container with ID starting with 7b2d132afcdd21c1bbdda244cfc1ecdbeccda0d22aed4eb60aa76220c6b2cb7e not found: ID does not exist"
Aug 05 17:46:54.652890 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:46:54.652872    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-e149b-predictor-6cc79c7d67-7r5pf"
Aug 05 17:46:54.664450 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:46:54.664431    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4d68887a-7de6-4f30-a190-e447efb44fbb-kube-rbac-proxy-sar-config\") pod \"4d68887a-7de6-4f30-a190-e447efb44fbb\" (UID: \"4d68887a-7de6-4f30-a190-e447efb44fbb\") "
Aug 05 17:46:54.664557 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:46:54.664544    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-h57sq\" (UniqueName: \"kubernetes.io/projected/4d68887a-7de6-4f30-a190-e447efb44fbb-kube-api-access-h57sq\") pod \"4d68887a-7de6-4f30-a190-e447efb44fbb\" (UID: \"4d68887a-7de6-4f30-a190-e447efb44fbb\") "
Aug 05 17:46:54.664604 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:46:54.664569    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4d68887a-7de6-4f30-a190-e447efb44fbb-proxy-tls\") pod \"4d68887a-7de6-4f30-a190-e447efb44fbb\" (UID: \"4d68887a-7de6-4f30-a190-e447efb44fbb\") "
Aug 05 17:46:54.664727 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:46:54.664709    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4d68887a-7de6-4f30-a190-e447efb44fbb-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "4d68887a-7de6-4f30-a190-e447efb44fbb" (UID: "4d68887a-7de6-4f30-a190-e447efb44fbb"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 05 17:46:54.666605 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:46:54.666583    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4d68887a-7de6-4f30-a190-e447efb44fbb-kube-api-access-h57sq" (OuterVolumeSpecName: "kube-api-access-h57sq") pod "4d68887a-7de6-4f30-a190-e447efb44fbb" (UID: "4d68887a-7de6-4f30-a190-e447efb44fbb"). InnerVolumeSpecName "kube-api-access-h57sq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 05 17:46:54.666694 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:46:54.666613    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4d68887a-7de6-4f30-a190-e447efb44fbb-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "4d68887a-7de6-4f30-a190-e447efb44fbb" (UID: "4d68887a-7de6-4f30-a190-e447efb44fbb"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 05 17:46:54.765259 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:46:54.765233    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b528315e-f99c-4518-bdf1-87a2863c69a6-proxy-tls\") pod \"b528315e-f99c-4518-bdf1-87a2863c69a6\" (UID: \"b528315e-f99c-4518-bdf1-87a2863c69a6\") "
Aug 05 17:46:54.765451 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:46:54.765264    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bxnlt\" (UniqueName: \"kubernetes.io/projected/b528315e-f99c-4518-bdf1-87a2863c69a6-kube-api-access-bxnlt\") pod \"b528315e-f99c-4518-bdf1-87a2863c69a6\" (UID: \"b528315e-f99c-4518-bdf1-87a2863c69a6\") "
Aug 05 17:46:54.765451 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:46:54.765322    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b528315e-f99c-4518-bdf1-87a2863c69a6-kube-rbac-proxy-sar-config\") pod \"b528315e-f99c-4518-bdf1-87a2863c69a6\" (UID: \"b528315e-f99c-4518-bdf1-87a2863c69a6\") "
Aug 05 17:46:54.765679 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:46:54.765656    2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h57sq\" (UniqueName: \"kubernetes.io/projected/4d68887a-7de6-4f30-a190-e447efb44fbb-kube-api-access-h57sq\") on node \"ip-10-0-135-2.ec2.internal\" DevicePath \"\""
Aug 05 17:46:54.765775 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:46:54.765681    2571 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4d68887a-7de6-4f30-a190-e447efb44fbb-proxy-tls\") on node \"ip-10-0-135-2.ec2.internal\" DevicePath \"\""
Aug 05 17:46:54.765775 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:46:54.765695    2571 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4d68887a-7de6-4f30-a190-e447efb44fbb-kube-rbac-proxy-sar-config\") on node \"ip-10-0-135-2.ec2.internal\" DevicePath \"\""
Aug 05 17:46:54.765775 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:46:54.765763    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b528315e-f99c-4518-bdf1-87a2863c69a6-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "b528315e-f99c-4518-bdf1-87a2863c69a6" (UID: "b528315e-f99c-4518-bdf1-87a2863c69a6"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 05 17:46:54.767558 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:46:54.767536    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b528315e-f99c-4518-bdf1-87a2863c69a6-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "b528315e-f99c-4518-bdf1-87a2863c69a6" (UID: "b528315e-f99c-4518-bdf1-87a2863c69a6"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 05 17:46:54.767558 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:46:54.767547    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b528315e-f99c-4518-bdf1-87a2863c69a6-kube-api-access-bxnlt" (OuterVolumeSpecName: "kube-api-access-bxnlt") pod "b528315e-f99c-4518-bdf1-87a2863c69a6" (UID: "b528315e-f99c-4518-bdf1-87a2863c69a6"). InnerVolumeSpecName "kube-api-access-bxnlt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 05 17:46:54.866786 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:46:54.866757    2571 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b528315e-f99c-4518-bdf1-87a2863c69a6-proxy-tls\") on node \"ip-10-0-135-2.ec2.internal\" DevicePath \"\""
Aug 05 17:46:54.866786 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:46:54.866783    2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bxnlt\" (UniqueName: \"kubernetes.io/projected/b528315e-f99c-4518-bdf1-87a2863c69a6-kube-api-access-bxnlt\") on node \"ip-10-0-135-2.ec2.internal\" DevicePath \"\""
Aug 05 17:46:54.866950 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:46:54.866796    2571 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b528315e-f99c-4518-bdf1-87a2863c69a6-kube-rbac-proxy-sar-config\") on node \"ip-10-0-135-2.ec2.internal\" DevicePath \"\""
Aug 05 17:46:54.948254 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:46:54.948211    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-e149b-predictor-5f5fbfc7cd-vq5fz"]
Aug 05 17:46:54.950826 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:46:54.950804    2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-e149b-predictor-5f5fbfc7cd-vq5fz"]
Aug 05 17:46:55.136689 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:46:55.136614    2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4d68887a-7de6-4f30-a190-e447efb44fbb" path="/var/lib/kubelet/pods/4d68887a-7de6-4f30-a190-e447efb44fbb/volumes"
Aug 05 17:46:55.632973 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:46:55.632945    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-e149b-predictor-6cc79c7d67-7r5pf"
Aug 05 17:46:55.633463 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:46:55.632976    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-e149b-predictor-6cc79c7d67-7r5pf" event={"ID":"b528315e-f99c-4518-bdf1-87a2863c69a6","Type":"ContainerDied","Data":"6e15b089a39e4390f1e7b57053005f37e0ab7b31ba265a9deed24a5bca529080"}
Aug 05 17:46:55.633463 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:46:55.633083    2571 scope.go:117] "RemoveContainer" containerID="51761bed037f05547f7d368956e335f28a0ea8a4ed69c93169a3c25ce90168d9"
Aug 05 17:46:55.633463 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:46:55.633362    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-741a0-predictor-6c788cfb87-2t5vl" podUID="04ea2238-8ad7-4bc1-9021-e17922673b58" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.45:8080: connect: connection refused"
Aug 05 17:46:55.641837 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:46:55.641818    2571 scope.go:117] "RemoveContainer" containerID="018eccd20db267e475a715d33621f2c67123455028fe1ce983553d7eb400b911"
Aug 05 17:46:55.654399 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:46:55.654360    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-e149b-predictor-6cc79c7d67-7r5pf"]
Aug 05 17:46:55.661017 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:46:55.660998    2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-e149b-predictor-6cc79c7d67-7r5pf"]
Aug 05 17:46:57.137202 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:46:57.137167    2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b528315e-f99c-4518-bdf1-87a2863c69a6" path="/var/lib/kubelet/pods/b528315e-f99c-4518-bdf1-87a2863c69a6/volumes"
Aug 05 17:46:58.623115 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:46:58.623088    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-741a0-predictor-77484d674-xt592"
Aug 05 17:46:58.623657 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:46:58.623633    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-741a0-predictor-77484d674-xt592" podUID="30ca866f-36b3-4229-9126-d138dd30da9f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.44:8080: connect: connection refused"
Aug 05 17:47:00.637433 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:47:00.637404    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-741a0-predictor-6c788cfb87-2t5vl"
Aug 05 17:47:00.637837 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:47:00.637783    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-741a0-predictor-6c788cfb87-2t5vl" podUID="04ea2238-8ad7-4bc1-9021-e17922673b58" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.45:8080: connect: connection refused"
Aug 05 17:47:01.580176 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:47:01.580141    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-92074-predictor-79d689c86d-w5pjl"]
Aug 05 17:47:01.580522 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:47:01.580485    2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-92074-predictor-79d689c86d-w5pjl" podUID="337b1675-3909-4f89-a905-35432d0ff577" containerName="kserve-container" containerID="cri-o://7a8b5c3ce5bb204b97598752cca1de3327c9c9f9cc49b6a14468cd8bce114211" gracePeriod=30
Aug 05 17:47:01.580669 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:47:01.580527    2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-92074-predictor-79d689c86d-w5pjl" podUID="337b1675-3909-4f89-a905-35432d0ff577" containerName="kube-rbac-proxy" containerID="cri-o://258ceef332b42dcbf23de9c286b2734731bd77a950a560fbd71a5a65bf135e8f" gracePeriod=30
Aug 05 17:47:01.638909 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:47:01.638885    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-db0ba-predictor-5cd4b48554-7xcb6"]
Aug 05 17:47:01.639418 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:47:01.639400    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4d68887a-7de6-4f30-a190-e447efb44fbb" containerName="kube-rbac-proxy"
Aug 05 17:47:01.639509 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:47:01.639420    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="4d68887a-7de6-4f30-a190-e447efb44fbb" containerName="kube-rbac-proxy"
Aug 05 17:47:01.639509 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:47:01.639435    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4d68887a-7de6-4f30-a190-e447efb44fbb" containerName="kserve-container"
Aug 05 17:47:01.639509 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:47:01.639444    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="4d68887a-7de6-4f30-a190-e447efb44fbb" containerName="kserve-container"
Aug 05 17:47:01.639509 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:47:01.639456    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b528315e-f99c-4518-bdf1-87a2863c69a6" containerName="kserve-container"
Aug 05 17:47:01.639509 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:47:01.639464    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="b528315e-f99c-4518-bdf1-87a2863c69a6" containerName="kserve-container"
Aug 05 17:47:01.639509 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:47:01.639508    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b528315e-f99c-4518-bdf1-87a2863c69a6" containerName="kube-rbac-proxy"
Aug 05 17:47:01.639799 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:47:01.639518    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="b528315e-f99c-4518-bdf1-87a2863c69a6" containerName="kube-rbac-proxy"
Aug 05 17:47:01.639799 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:47:01.639609    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="b528315e-f99c-4518-bdf1-87a2863c69a6" containerName="kube-rbac-proxy"
Aug 05 17:47:01.639799 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:47:01.639621    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="4d68887a-7de6-4f30-a190-e447efb44fbb" containerName="kserve-container"
Aug 05 17:47:01.639799 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:47:01.639633    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="4d68887a-7de6-4f30-a190-e447efb44fbb" containerName="kube-rbac-proxy"
Aug 05 17:47:01.639799 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:47:01.639644    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="b528315e-f99c-4518-bdf1-87a2863c69a6" containerName="kserve-container"
Aug 05 17:47:01.643056 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:47:01.643039    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-db0ba-predictor-5cd4b48554-7xcb6"
Aug 05 17:47:01.645345 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:47:01.645327    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-db0ba-predictor-serving-cert\""
Aug 05 17:47:01.645451 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:47:01.645349    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-db0ba-kube-rbac-proxy-sar-config\""
Aug 05 17:47:01.655066 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:47:01.655043    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-db0ba-predictor-5cd4b48554-7xcb6"]
Aug 05 17:47:01.708701 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:47:01.708674    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-92074-predictor-589dc7f869-8mmn2"]
Aug 05 17:47:01.708911 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:47:01.708882    2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-92074-predictor-589dc7f869-8mmn2" podUID="45df6046-112b-493c-b9bf-b746d05c3a21" containerName="kserve-container" containerID="cri-o://fd4a5054b179d69f81577871b6c0d6fb709a755368fd464952d0de9fbd7412ed" gracePeriod=30
Aug 05 17:47:01.709015 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:47:01.708946    2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-92074-predictor-589dc7f869-8mmn2" podUID="45df6046-112b-493c-b9bf-b746d05c3a21" containerName="kube-rbac-proxy" containerID="cri-o://d8aba550390f736c0547b246d73c65c9492807538fbf2aec54ecaa0781d46b6f" gracePeriod=30
Aug 05 17:47:01.723054 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:47:01.723031    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h6fx8\" (UniqueName: \"kubernetes.io/projected/a5dc3454-c0b1-4150-aa89-3e4034c4e681-kube-api-access-h6fx8\") pod \"success-200-isvc-db0ba-predictor-5cd4b48554-7xcb6\" (UID: \"a5dc3454-c0b1-4150-aa89-3e4034c4e681\") " pod="kserve-ci-e2e-test/success-200-isvc-db0ba-predictor-5cd4b48554-7xcb6"
Aug 05 17:47:01.723167 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:47:01.723091    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a5dc3454-c0b1-4150-aa89-3e4034c4e681-proxy-tls\") pod \"success-200-isvc-db0ba-predictor-5cd4b48554-7xcb6\" (UID: \"a5dc3454-c0b1-4150-aa89-3e4034c4e681\") " pod="kserve-ci-e2e-test/success-200-isvc-db0ba-predictor-5cd4b48554-7xcb6"
Aug 05 17:47:01.723232 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:47:01.723168    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a5dc3454-c0b1-4150-aa89-3e4034c4e681-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-db0ba-predictor-5cd4b48554-7xcb6\" (UID: \"a5dc3454-c0b1-4150-aa89-3e4034c4e681\") " pod="kserve-ci-e2e-test/success-200-isvc-db0ba-predictor-5cd4b48554-7xcb6"
Aug 05 17:47:01.776461 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:47:01.776430    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-db0ba-predictor-68c79c6fb7-mpttp"]
Aug 05 17:47:01.779972 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:47:01.779957    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-db0ba-predictor-68c79c6fb7-mpttp"
Aug 05 17:47:01.782251 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:47:01.782232    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-db0ba-predictor-serving-cert\""
Aug 05 17:47:01.782523 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:47:01.782504    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-db0ba-kube-rbac-proxy-sar-config\""
Aug 05 17:47:01.793135 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:47:01.793116    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-db0ba-predictor-68c79c6fb7-mpttp"]
Aug 05 17:47:01.824452 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:47:01.824435    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h6fx8\" (UniqueName: \"kubernetes.io/projected/a5dc3454-c0b1-4150-aa89-3e4034c4e681-kube-api-access-h6fx8\") pod \"success-200-isvc-db0ba-predictor-5cd4b48554-7xcb6\" (UID: \"a5dc3454-c0b1-4150-aa89-3e4034c4e681\") " pod="kserve-ci-e2e-test/success-200-isvc-db0ba-predictor-5cd4b48554-7xcb6"
Aug 05 17:47:01.824524 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:47:01.824485    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a5dc3454-c0b1-4150-aa89-3e4034c4e681-proxy-tls\") pod \"success-200-isvc-db0ba-predictor-5cd4b48554-7xcb6\" (UID: \"a5dc3454-c0b1-4150-aa89-3e4034c4e681\") " pod="kserve-ci-e2e-test/success-200-isvc-db0ba-predictor-5cd4b48554-7xcb6"
Aug 05 17:47:01.824524 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:47:01.824513    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a5dc3454-c0b1-4150-aa89-3e4034c4e681-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-db0ba-predictor-5cd4b48554-7xcb6\" (UID: \"a5dc3454-c0b1-4150-aa89-3e4034c4e681\") " pod="kserve-ci-e2e-test/success-200-isvc-db0ba-predictor-5cd4b48554-7xcb6"
Aug 05 17:47:01.824648 ip-10-0-135-2 kubenswrapper[2571]: E0805 17:47:01.824631    2571 secret.go:189] Couldn't get secret kserve-ci-e2e-test/success-200-isvc-db0ba-predictor-serving-cert: secret "success-200-isvc-db0ba-predictor-serving-cert" not found
Aug 05 17:47:01.824696 ip-10-0-135-2 kubenswrapper[2571]: E0805 17:47:01.824685    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a5dc3454-c0b1-4150-aa89-3e4034c4e681-proxy-tls podName:a5dc3454-c0b1-4150-aa89-3e4034c4e681 nodeName:}" failed. No retries permitted until 2026-08-05 17:47:02.32466959 +0000 UTC m=+2049.692035915 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/a5dc3454-c0b1-4150-aa89-3e4034c4e681-proxy-tls") pod "success-200-isvc-db0ba-predictor-5cd4b48554-7xcb6" (UID: "a5dc3454-c0b1-4150-aa89-3e4034c4e681") : secret "success-200-isvc-db0ba-predictor-serving-cert" not found
Aug 05 17:47:01.825076 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:47:01.825060    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a5dc3454-c0b1-4150-aa89-3e4034c4e681-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-db0ba-predictor-5cd4b48554-7xcb6\" (UID: \"a5dc3454-c0b1-4150-aa89-3e4034c4e681\") " pod="kserve-ci-e2e-test/success-200-isvc-db0ba-predictor-5cd4b48554-7xcb6"
Aug 05 17:47:01.858597 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:47:01.858539    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h6fx8\" (UniqueName: \"kubernetes.io/projected/a5dc3454-c0b1-4150-aa89-3e4034c4e681-kube-api-access-h6fx8\") pod \"success-200-isvc-db0ba-predictor-5cd4b48554-7xcb6\" (UID: \"a5dc3454-c0b1-4150-aa89-3e4034c4e681\") " pod="kserve-ci-e2e-test/success-200-isvc-db0ba-predictor-5cd4b48554-7xcb6"
Aug 05 17:47:01.925885 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:47:01.925858    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8b670052-3f8a-4f70-82a8-d772ab208ea0-proxy-tls\") pod \"error-404-isvc-db0ba-predictor-68c79c6fb7-mpttp\" (UID: \"8b670052-3f8a-4f70-82a8-d772ab208ea0\") " pod="kserve-ci-e2e-test/error-404-isvc-db0ba-predictor-68c79c6fb7-mpttp"
Aug 05 17:47:01.925986 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:47:01.925892    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8b670052-3f8a-4f70-82a8-d772ab208ea0-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-db0ba-predictor-68c79c6fb7-mpttp\" (UID: \"8b670052-3f8a-4f70-82a8-d772ab208ea0\") " pod="kserve-ci-e2e-test/error-404-isvc-db0ba-predictor-68c79c6fb7-mpttp"
Aug 05 17:47:01.925986 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:47:01.925951    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l7kcz\" (UniqueName: \"kubernetes.io/projected/8b670052-3f8a-4f70-82a8-d772ab208ea0-kube-api-access-l7kcz\") pod \"error-404-isvc-db0ba-predictor-68c79c6fb7-mpttp\" (UID: \"8b670052-3f8a-4f70-82a8-d772ab208ea0\") " pod="kserve-ci-e2e-test/error-404-isvc-db0ba-predictor-68c79c6fb7-mpttp"
Aug 05 17:47:02.027062 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:47:02.027024    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8b670052-3f8a-4f70-82a8-d772ab208ea0-proxy-tls\") pod \"error-404-isvc-db0ba-predictor-68c79c6fb7-mpttp\" (UID: \"8b670052-3f8a-4f70-82a8-d772ab208ea0\") " pod="kserve-ci-e2e-test/error-404-isvc-db0ba-predictor-68c79c6fb7-mpttp"
Aug 05 17:47:02.027214 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:47:02.027068    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8b670052-3f8a-4f70-82a8-d772ab208ea0-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-db0ba-predictor-68c79c6fb7-mpttp\" (UID: \"8b670052-3f8a-4f70-82a8-d772ab208ea0\") " pod="kserve-ci-e2e-test/error-404-isvc-db0ba-predictor-68c79c6fb7-mpttp"
Aug 05 17:47:02.027214 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:47:02.027123    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l7kcz\" (UniqueName: \"kubernetes.io/projected/8b670052-3f8a-4f70-82a8-d772ab208ea0-kube-api-access-l7kcz\") pod \"error-404-isvc-db0ba-predictor-68c79c6fb7-mpttp\" (UID: \"8b670052-3f8a-4f70-82a8-d772ab208ea0\") " pod="kserve-ci-e2e-test/error-404-isvc-db0ba-predictor-68c79c6fb7-mpttp"
Aug 05 17:47:02.027214 ip-10-0-135-2 kubenswrapper[2571]: E0805 17:47:02.027168    2571 secret.go:189] Couldn't get secret kserve-ci-e2e-test/error-404-isvc-db0ba-predictor-serving-cert: secret "error-404-isvc-db0ba-predictor-serving-cert" not found
Aug 05 17:47:02.027369 ip-10-0-135-2 kubenswrapper[2571]: E0805 17:47:02.027235    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8b670052-3f8a-4f70-82a8-d772ab208ea0-proxy-tls podName:8b670052-3f8a-4f70-82a8-d772ab208ea0 nodeName:}" failed. No retries permitted until 2026-08-05 17:47:02.527218376 +0000 UTC m=+2049.894584697 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/8b670052-3f8a-4f70-82a8-d772ab208ea0-proxy-tls") pod "error-404-isvc-db0ba-predictor-68c79c6fb7-mpttp" (UID: "8b670052-3f8a-4f70-82a8-d772ab208ea0") : secret "error-404-isvc-db0ba-predictor-serving-cert" not found
Aug 05 17:47:02.027733 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:47:02.027714    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8b670052-3f8a-4f70-82a8-d772ab208ea0-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-db0ba-predictor-68c79c6fb7-mpttp\" (UID: \"8b670052-3f8a-4f70-82a8-d772ab208ea0\") " pod="kserve-ci-e2e-test/error-404-isvc-db0ba-predictor-68c79c6fb7-mpttp"
Aug 05 17:47:02.049838 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:47:02.049811    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l7kcz\" (UniqueName: \"kubernetes.io/projected/8b670052-3f8a-4f70-82a8-d772ab208ea0-kube-api-access-l7kcz\") pod \"error-404-isvc-db0ba-predictor-68c79c6fb7-mpttp\" (UID: \"8b670052-3f8a-4f70-82a8-d772ab208ea0\") " pod="kserve-ci-e2e-test/error-404-isvc-db0ba-predictor-68c79c6fb7-mpttp"
Aug 05 17:47:02.329654 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:47:02.329621    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a5dc3454-c0b1-4150-aa89-3e4034c4e681-proxy-tls\") pod \"success-200-isvc-db0ba-predictor-5cd4b48554-7xcb6\" (UID: \"a5dc3454-c0b1-4150-aa89-3e4034c4e681\") " pod="kserve-ci-e2e-test/success-200-isvc-db0ba-predictor-5cd4b48554-7xcb6"
Aug 05 17:47:02.331926 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:47:02.331905    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a5dc3454-c0b1-4150-aa89-3e4034c4e681-proxy-tls\") pod \"success-200-isvc-db0ba-predictor-5cd4b48554-7xcb6\" (UID: \"a5dc3454-c0b1-4150-aa89-3e4034c4e681\") " pod="kserve-ci-e2e-test/success-200-isvc-db0ba-predictor-5cd4b48554-7xcb6"
Aug 05 17:47:02.531096 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:47:02.531059    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8b670052-3f8a-4f70-82a8-d772ab208ea0-proxy-tls\") pod \"error-404-isvc-db0ba-predictor-68c79c6fb7-mpttp\" (UID: \"8b670052-3f8a-4f70-82a8-d772ab208ea0\") " pod="kserve-ci-e2e-test/error-404-isvc-db0ba-predictor-68c79c6fb7-mpttp"
Aug 05 17:47:02.533514 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:47:02.533490    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8b670052-3f8a-4f70-82a8-d772ab208ea0-proxy-tls\") pod \"error-404-isvc-db0ba-predictor-68c79c6fb7-mpttp\" (UID: \"8b670052-3f8a-4f70-82a8-d772ab208ea0\") " pod="kserve-ci-e2e-test/error-404-isvc-db0ba-predictor-68c79c6fb7-mpttp"
Aug 05 17:47:02.555359 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:47:02.555329    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-db0ba-predictor-5cd4b48554-7xcb6"
Aug 05 17:47:02.671233 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:47:02.671203    2571 generic.go:356] "Generic (PLEG): container finished" podID="45df6046-112b-493c-b9bf-b746d05c3a21" containerID="d8aba550390f736c0547b246d73c65c9492807538fbf2aec54ecaa0781d46b6f" exitCode=2
Aug 05 17:47:02.671641 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:47:02.671269    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-92074-predictor-589dc7f869-8mmn2" event={"ID":"45df6046-112b-493c-b9bf-b746d05c3a21","Type":"ContainerDied","Data":"d8aba550390f736c0547b246d73c65c9492807538fbf2aec54ecaa0781d46b6f"}
Aug 05 17:47:02.673009 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:47:02.672988    2571 generic.go:356] "Generic (PLEG): container finished" podID="337b1675-3909-4f89-a905-35432d0ff577" containerID="258ceef332b42dcbf23de9c286b2734731bd77a950a560fbd71a5a65bf135e8f" exitCode=2
Aug 05 17:47:02.673086 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:47:02.673031    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-92074-predictor-79d689c86d-w5pjl" event={"ID":"337b1675-3909-4f89-a905-35432d0ff577","Type":"ContainerDied","Data":"258ceef332b42dcbf23de9c286b2734731bd77a950a560fbd71a5a65bf135e8f"}
Aug 05 17:47:02.685123 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:47:02.685077    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-db0ba-predictor-5cd4b48554-7xcb6"]
Aug 05 17:47:02.687228 ip-10-0-135-2 kubenswrapper[2571]: W0805 17:47:02.687208    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5dc3454_c0b1_4150_aa89_3e4034c4e681.slice/crio-9c43ae0a9ff2836931f3b9dd6838b11556fcd7fda2b1e6c718b4e21d9ffae08a WatchSource:0}: Error finding container 9c43ae0a9ff2836931f3b9dd6838b11556fcd7fda2b1e6c718b4e21d9ffae08a: Status 404 returned error can't find the container with id 9c43ae0a9ff2836931f3b9dd6838b11556fcd7fda2b1e6c718b4e21d9ffae08a
Aug 05 17:47:02.689615 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:47:02.689599    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-db0ba-predictor-68c79c6fb7-mpttp"
Aug 05 17:47:02.830992 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:47:02.830969    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-db0ba-predictor-68c79c6fb7-mpttp"]
Aug 05 17:47:02.832606 ip-10-0-135-2 kubenswrapper[2571]: W0805 17:47:02.832583    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8b670052_3f8a_4f70_82a8_d772ab208ea0.slice/crio-671d1fbc8188919c77cf003778f42095c8349b6b064c7bb900c4f19b6afa38d1 WatchSource:0}: Error finding container 671d1fbc8188919c77cf003778f42095c8349b6b064c7bb900c4f19b6afa38d1: Status 404 returned error can't find the container with id 671d1fbc8188919c77cf003778f42095c8349b6b064c7bb900c4f19b6afa38d1
Aug 05 17:47:03.678710 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:47:03.678673    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-db0ba-predictor-68c79c6fb7-mpttp" event={"ID":"8b670052-3f8a-4f70-82a8-d772ab208ea0","Type":"ContainerStarted","Data":"d6a8b499dd9fd29c4e05db3675052bc899051e31fe1ea8a3821b2761d0802e0e"}
Aug 05 17:47:03.679130 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:47:03.678744    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-db0ba-predictor-68c79c6fb7-mpttp"
Aug 05 17:47:03.679130 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:47:03.678776    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-db0ba-predictor-68c79c6fb7-mpttp"
Aug 05 17:47:03.679130 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:47:03.678804    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-db0ba-predictor-68c79c6fb7-mpttp" event={"ID":"8b670052-3f8a-4f70-82a8-d772ab208ea0","Type":"ContainerStarted","Data":"1e8c419575f0dfaeeeccaf96ae220bca2a763864c06544bffc3ff3e09b25fdb7"}
Aug 05 17:47:03.679130 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:47:03.678834    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-db0ba-predictor-68c79c6fb7-mpttp" event={"ID":"8b670052-3f8a-4f70-82a8-d772ab208ea0","Type":"ContainerStarted","Data":"671d1fbc8188919c77cf003778f42095c8349b6b064c7bb900c4f19b6afa38d1"}
Aug 05 17:47:03.680536 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:47:03.680495    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-db0ba-predictor-68c79c6fb7-mpttp" podUID="8b670052-3f8a-4f70-82a8-d772ab208ea0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.47:8080: connect: connection refused"
Aug 05 17:47:03.680663 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:47:03.680580    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-db0ba-predictor-5cd4b48554-7xcb6" event={"ID":"a5dc3454-c0b1-4150-aa89-3e4034c4e681","Type":"ContainerStarted","Data":"2be79787d96ef2ea66149008ed22b7e16612f947ff14d9579f4df3d4c840e616"}
Aug 05 17:47:03.680663 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:47:03.680630    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-db0ba-predictor-5cd4b48554-7xcb6" event={"ID":"a5dc3454-c0b1-4150-aa89-3e4034c4e681","Type":"ContainerStarted","Data":"ad3df2a5fb51663f6d5375395e47f0ac451078a6aa07e842a8a72567ff631165"}
Aug 05 17:47:03.680663 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:47:03.680652    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-db0ba-predictor-5cd4b48554-7xcb6" event={"ID":"a5dc3454-c0b1-4150-aa89-3e4034c4e681","Type":"ContainerStarted","Data":"9c43ae0a9ff2836931f3b9dd6838b11556fcd7fda2b1e6c718b4e21d9ffae08a"}
Aug 05 17:47:03.680789 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:47:03.680674    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-db0ba-predictor-5cd4b48554-7xcb6"
Aug 05 17:47:03.680789 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:47:03.680703    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-db0ba-predictor-5cd4b48554-7xcb6"
Aug 05 17:47:03.681762 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:47:03.681742    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-db0ba-predictor-5cd4b48554-7xcb6" podUID="a5dc3454-c0b1-4150-aa89-3e4034c4e681" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.46:8080: connect: connection refused"
Aug 05 17:47:03.702308 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:47:03.702269    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/error-404-isvc-db0ba-predictor-68c79c6fb7-mpttp" podStartSLOduration=2.70225573 podStartE2EDuration="2.70225573s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-05 17:47:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-05 17:47:03.700440552 +0000 UTC m=+2051.067806916" watchObservedRunningTime="2026-08-05 17:47:03.70225573 +0000 UTC m=+2051.069622072"
Aug 05 17:47:03.719485 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:47:03.719451    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/success-200-isvc-db0ba-predictor-5cd4b48554-7xcb6" podStartSLOduration=2.719438679 podStartE2EDuration="2.719438679s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-05 17:47:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-05 17:47:03.719083898 +0000 UTC m=+2051.086450242" watchObservedRunningTime="2026-08-05 17:47:03.719438679 +0000 UTC m=+2051.086805022"
Aug 05 17:47:04.327934 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:47:04.327896    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-92074-predictor-79d689c86d-w5pjl" podUID="337b1675-3909-4f89-a905-35432d0ff577" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.42:8643/healthz\": dial tcp 10.134.0.42:8643: connect: connection refused"
Aug 05 17:47:04.685977 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:47:04.685933    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-db0ba-predictor-68c79c6fb7-mpttp" podUID="8b670052-3f8a-4f70-82a8-d772ab208ea0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.47:8080: connect: connection refused"
Aug 05 17:47:04.686368 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:47:04.685987    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-db0ba-predictor-5cd4b48554-7xcb6" podUID="a5dc3454-c0b1-4150-aa89-3e4034c4e681" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.46:8080: connect: connection refused"
Aug 05 17:47:04.829695 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:47:04.829667    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-92074-predictor-79d689c86d-w5pjl"
Aug 05 17:47:04.952804 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:47:04.952770    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/337b1675-3909-4f89-a905-35432d0ff577-proxy-tls\") pod \"337b1675-3909-4f89-a905-35432d0ff577\" (UID: \"337b1675-3909-4f89-a905-35432d0ff577\") "
Aug 05 17:47:04.952933 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:47:04.952873    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/337b1675-3909-4f89-a905-35432d0ff577-kube-rbac-proxy-sar-config\") pod \"337b1675-3909-4f89-a905-35432d0ff577\" (UID: \"337b1675-3909-4f89-a905-35432d0ff577\") "
Aug 05 17:47:04.952989 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:47:04.952944    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9jdgz\" (UniqueName: \"kubernetes.io/projected/337b1675-3909-4f89-a905-35432d0ff577-kube-api-access-9jdgz\") pod \"337b1675-3909-4f89-a905-35432d0ff577\" (UID: \"337b1675-3909-4f89-a905-35432d0ff577\") "
Aug 05 17:47:04.953250 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:47:04.953222    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/337b1675-3909-4f89-a905-35432d0ff577-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "337b1675-3909-4f89-a905-35432d0ff577" (UID: "337b1675-3909-4f89-a905-35432d0ff577"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 05 17:47:04.954801 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:47:04.954773    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/337b1675-3909-4f89-a905-35432d0ff577-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "337b1675-3909-4f89-a905-35432d0ff577" (UID: "337b1675-3909-4f89-a905-35432d0ff577"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 05 17:47:04.954907 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:47:04.954872    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/337b1675-3909-4f89-a905-35432d0ff577-kube-api-access-9jdgz" (OuterVolumeSpecName: "kube-api-access-9jdgz") pod "337b1675-3909-4f89-a905-35432d0ff577" (UID: "337b1675-3909-4f89-a905-35432d0ff577"). InnerVolumeSpecName "kube-api-access-9jdgz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 05 17:47:05.054112 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:47:05.054069    2571 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/337b1675-3909-4f89-a905-35432d0ff577-kube-rbac-proxy-sar-config\") on node \"ip-10-0-135-2.ec2.internal\" DevicePath \"\""
Aug 05 17:47:05.054112 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:47:05.054105    2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9jdgz\" (UniqueName: \"kubernetes.io/projected/337b1675-3909-4f89-a905-35432d0ff577-kube-api-access-9jdgz\") on node \"ip-10-0-135-2.ec2.internal\" DevicePath \"\""
Aug 05 17:47:05.054112 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:47:05.054121    2571 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/337b1675-3909-4f89-a905-35432d0ff577-proxy-tls\") on node \"ip-10-0-135-2.ec2.internal\" DevicePath \"\""
Aug 05 17:47:05.141062 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:47:05.141034    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-92074-predictor-589dc7f869-8mmn2"
Aug 05 17:47:05.255486 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:47:05.255456    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-42gj7\" (UniqueName: \"kubernetes.io/projected/45df6046-112b-493c-b9bf-b746d05c3a21-kube-api-access-42gj7\") pod \"45df6046-112b-493c-b9bf-b746d05c3a21\" (UID: \"45df6046-112b-493c-b9bf-b746d05c3a21\") "
Aug 05 17:47:05.255636 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:47:05.255511    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/45df6046-112b-493c-b9bf-b746d05c3a21-kube-rbac-proxy-sar-config\") pod \"45df6046-112b-493c-b9bf-b746d05c3a21\" (UID: \"45df6046-112b-493c-b9bf-b746d05c3a21\") "
Aug 05 17:47:05.255636 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:47:05.255579    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/45df6046-112b-493c-b9bf-b746d05c3a21-proxy-tls\") pod \"45df6046-112b-493c-b9bf-b746d05c3a21\" (UID: \"45df6046-112b-493c-b9bf-b746d05c3a21\") "
Aug 05 17:47:05.255913 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:47:05.255881    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/45df6046-112b-493c-b9bf-b746d05c3a21-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "45df6046-112b-493c-b9bf-b746d05c3a21" (UID: "45df6046-112b-493c-b9bf-b746d05c3a21"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 05 17:47:05.257630 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:47:05.257608    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/45df6046-112b-493c-b9bf-b746d05c3a21-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "45df6046-112b-493c-b9bf-b746d05c3a21" (UID: "45df6046-112b-493c-b9bf-b746d05c3a21"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 05 17:47:05.257706 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:47:05.257687    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/45df6046-112b-493c-b9bf-b746d05c3a21-kube-api-access-42gj7" (OuterVolumeSpecName: "kube-api-access-42gj7") pod "45df6046-112b-493c-b9bf-b746d05c3a21" (UID: "45df6046-112b-493c-b9bf-b746d05c3a21"). InnerVolumeSpecName "kube-api-access-42gj7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 05 17:47:05.356779 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:47:05.356742    2571 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/45df6046-112b-493c-b9bf-b746d05c3a21-proxy-tls\") on node \"ip-10-0-135-2.ec2.internal\" DevicePath \"\""
Aug 05 17:47:05.356779 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:47:05.356770    2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-42gj7\" (UniqueName: \"kubernetes.io/projected/45df6046-112b-493c-b9bf-b746d05c3a21-kube-api-access-42gj7\") on node \"ip-10-0-135-2.ec2.internal\" DevicePath \"\""
Aug 05 17:47:05.356779 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:47:05.356784    2571 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/45df6046-112b-493c-b9bf-b746d05c3a21-kube-rbac-proxy-sar-config\") on node \"ip-10-0-135-2.ec2.internal\" DevicePath \"\""
Aug 05 17:47:05.691205 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:47:05.691176    2571 generic.go:356] "Generic (PLEG): container finished" podID="45df6046-112b-493c-b9bf-b746d05c3a21" containerID="fd4a5054b179d69f81577871b6c0d6fb709a755368fd464952d0de9fbd7412ed" exitCode=0
Aug 05 17:47:05.691643 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:47:05.691224    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-92074-predictor-589dc7f869-8mmn2"
Aug 05 17:47:05.691643 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:47:05.691254    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-92074-predictor-589dc7f869-8mmn2" event={"ID":"45df6046-112b-493c-b9bf-b746d05c3a21","Type":"ContainerDied","Data":"fd4a5054b179d69f81577871b6c0d6fb709a755368fd464952d0de9fbd7412ed"}
Aug 05 17:47:05.691643 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:47:05.691308    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-92074-predictor-589dc7f869-8mmn2" event={"ID":"45df6046-112b-493c-b9bf-b746d05c3a21","Type":"ContainerDied","Data":"4a7e8877fca8479efe7eaf62ba6050ed28f5fabccf461739ea104b0c568b9cd3"}
Aug 05 17:47:05.691643 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:47:05.691338    2571 scope.go:117] "RemoveContainer" containerID="d8aba550390f736c0547b246d73c65c9492807538fbf2aec54ecaa0781d46b6f"
Aug 05 17:47:05.692903 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:47:05.692880    2571 generic.go:356] "Generic (PLEG): container finished" podID="337b1675-3909-4f89-a905-35432d0ff577" containerID="7a8b5c3ce5bb204b97598752cca1de3327c9c9f9cc49b6a14468cd8bce114211" exitCode=0
Aug 05 17:47:05.693031 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:47:05.692924    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-92074-predictor-79d689c86d-w5pjl"
Aug 05 17:47:05.693031 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:47:05.692925    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-92074-predictor-79d689c86d-w5pjl" event={"ID":"337b1675-3909-4f89-a905-35432d0ff577","Type":"ContainerDied","Data":"7a8b5c3ce5bb204b97598752cca1de3327c9c9f9cc49b6a14468cd8bce114211"}
Aug 05 17:47:05.693153 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:47:05.693055    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-92074-predictor-79d689c86d-w5pjl" event={"ID":"337b1675-3909-4f89-a905-35432d0ff577","Type":"ContainerDied","Data":"95280739e6f2b2afdc05003d8de9614f9c3d78d0b7edce5cf1ca9f32a4bfc416"}
Aug 05 17:47:05.700417 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:47:05.700159    2571 scope.go:117] "RemoveContainer" containerID="fd4a5054b179d69f81577871b6c0d6fb709a755368fd464952d0de9fbd7412ed"
Aug 05 17:47:05.707338 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:47:05.707320    2571 scope.go:117] "RemoveContainer" containerID="d8aba550390f736c0547b246d73c65c9492807538fbf2aec54ecaa0781d46b6f"
Aug 05 17:47:05.707669 ip-10-0-135-2 kubenswrapper[2571]: E0805 17:47:05.707652    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d8aba550390f736c0547b246d73c65c9492807538fbf2aec54ecaa0781d46b6f\": container with ID starting with d8aba550390f736c0547b246d73c65c9492807538fbf2aec54ecaa0781d46b6f not found: ID does not exist" containerID="d8aba550390f736c0547b246d73c65c9492807538fbf2aec54ecaa0781d46b6f"
Aug 05 17:47:05.707740 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:47:05.707676    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d8aba550390f736c0547b246d73c65c9492807538fbf2aec54ecaa0781d46b6f"} err="failed to get container status \"d8aba550390f736c0547b246d73c65c9492807538fbf2aec54ecaa0781d46b6f\": rpc error: code = NotFound desc = could not find container \"d8aba550390f736c0547b246d73c65c9492807538fbf2aec54ecaa0781d46b6f\": container with ID starting with d8aba550390f736c0547b246d73c65c9492807538fbf2aec54ecaa0781d46b6f not found: ID does not exist"
Aug 05 17:47:05.707740 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:47:05.707693    2571 scope.go:117] "RemoveContainer" containerID="fd4a5054b179d69f81577871b6c0d6fb709a755368fd464952d0de9fbd7412ed"
Aug 05 17:47:05.707930 ip-10-0-135-2 kubenswrapper[2571]: E0805 17:47:05.707913    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fd4a5054b179d69f81577871b6c0d6fb709a755368fd464952d0de9fbd7412ed\": container with ID starting with fd4a5054b179d69f81577871b6c0d6fb709a755368fd464952d0de9fbd7412ed not found: ID does not exist" containerID="fd4a5054b179d69f81577871b6c0d6fb709a755368fd464952d0de9fbd7412ed"
Aug 05 17:47:05.707971 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:47:05.707936    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fd4a5054b179d69f81577871b6c0d6fb709a755368fd464952d0de9fbd7412ed"} err="failed to get container status \"fd4a5054b179d69f81577871b6c0d6fb709a755368fd464952d0de9fbd7412ed\": rpc error: code = NotFound desc = could not find container \"fd4a5054b179d69f81577871b6c0d6fb709a755368fd464952d0de9fbd7412ed\": container with ID starting with fd4a5054b179d69f81577871b6c0d6fb709a755368fd464952d0de9fbd7412ed not found: ID does not exist"
Aug 05 17:47:05.707971 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:47:05.707953    2571 scope.go:117] "RemoveContainer" containerID="258ceef332b42dcbf23de9c286b2734731bd77a950a560fbd71a5a65bf135e8f"
Aug 05 17:47:05.710523 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:47:05.710504    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-92074-predictor-79d689c86d-w5pjl"]
Aug 05 17:47:05.714778 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:47:05.714758    2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-92074-predictor-79d689c86d-w5pjl"]
Aug 05 17:47:05.715494 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:47:05.715480    2571 scope.go:117] "RemoveContainer" containerID="7a8b5c3ce5bb204b97598752cca1de3327c9c9f9cc49b6a14468cd8bce114211"
Aug 05 17:47:05.722523 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:47:05.722509    2571 scope.go:117] "RemoveContainer" containerID="258ceef332b42dcbf23de9c286b2734731bd77a950a560fbd71a5a65bf135e8f"
Aug 05 17:47:05.722773 ip-10-0-135-2 kubenswrapper[2571]: E0805 17:47:05.722753    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"258ceef332b42dcbf23de9c286b2734731bd77a950a560fbd71a5a65bf135e8f\": container with ID starting with 258ceef332b42dcbf23de9c286b2734731bd77a950a560fbd71a5a65bf135e8f not found: ID does not exist" containerID="258ceef332b42dcbf23de9c286b2734731bd77a950a560fbd71a5a65bf135e8f"
Aug 05 17:47:05.722826 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:47:05.722784    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"258ceef332b42dcbf23de9c286b2734731bd77a950a560fbd71a5a65bf135e8f"} err="failed to get container status \"258ceef332b42dcbf23de9c286b2734731bd77a950a560fbd71a5a65bf135e8f\": rpc error: code = NotFound desc = could not find container \"258ceef332b42dcbf23de9c286b2734731bd77a950a560fbd71a5a65bf135e8f\": container with ID starting with 258ceef332b42dcbf23de9c286b2734731bd77a950a560fbd71a5a65bf135e8f not found: ID does not exist"
Aug 05 17:47:05.722826 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:47:05.722799    2571 scope.go:117] "RemoveContainer" containerID="7a8b5c3ce5bb204b97598752cca1de3327c9c9f9cc49b6a14468cd8bce114211"
Aug 05 17:47:05.723035 ip-10-0-135-2 kubenswrapper[2571]: E0805 17:47:05.723018    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7a8b5c3ce5bb204b97598752cca1de3327c9c9f9cc49b6a14468cd8bce114211\": container with ID starting with 7a8b5c3ce5bb204b97598752cca1de3327c9c9f9cc49b6a14468cd8bce114211 not found: ID does not exist" containerID="7a8b5c3ce5bb204b97598752cca1de3327c9c9f9cc49b6a14468cd8bce114211"
Aug 05 17:47:05.723092 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:47:05.723044    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7a8b5c3ce5bb204b97598752cca1de3327c9c9f9cc49b6a14468cd8bce114211"} err="failed to get container status \"7a8b5c3ce5bb204b97598752cca1de3327c9c9f9cc49b6a14468cd8bce114211\": rpc error: code = NotFound desc = could not find container \"7a8b5c3ce5bb204b97598752cca1de3327c9c9f9cc49b6a14468cd8bce114211\": container with ID starting with 7a8b5c3ce5bb204b97598752cca1de3327c9c9f9cc49b6a14468cd8bce114211 not found: ID does not exist"
Aug 05 17:47:05.726995 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:47:05.726975    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-92074-predictor-589dc7f869-8mmn2"]
Aug 05 17:47:05.733933 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:47:05.733912    2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-92074-predictor-589dc7f869-8mmn2"]
Aug 05 17:47:07.136695 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:47:07.136656    2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="337b1675-3909-4f89-a905-35432d0ff577" path="/var/lib/kubelet/pods/337b1675-3909-4f89-a905-35432d0ff577/volumes"
Aug 05 17:47:07.137271 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:47:07.137247    2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="45df6046-112b-493c-b9bf-b746d05c3a21" path="/var/lib/kubelet/pods/45df6046-112b-493c-b9bf-b746d05c3a21/volumes"
Aug 05 17:47:08.624067 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:47:08.624030    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-741a0-predictor-77484d674-xt592" podUID="30ca866f-36b3-4229-9126-d138dd30da9f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.44:8080: connect: connection refused"
Aug 05 17:47:09.693815 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:47:09.693780    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-db0ba-predictor-5cd4b48554-7xcb6"
Aug 05 17:47:09.694930 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:47:09.694895    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-db0ba-predictor-5cd4b48554-7xcb6" podUID="a5dc3454-c0b1-4150-aa89-3e4034c4e681" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.46:8080: connect: connection refused"
Aug 05 17:47:09.697322 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:47:09.697292    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-db0ba-predictor-68c79c6fb7-mpttp"
Aug 05 17:47:09.697789 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:47:09.697758    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-db0ba-predictor-68c79c6fb7-mpttp" podUID="8b670052-3f8a-4f70-82a8-d772ab208ea0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.47:8080: connect: connection refused"
Aug 05 17:47:10.638158 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:47:10.638116    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-741a0-predictor-6c788cfb87-2t5vl" podUID="04ea2238-8ad7-4bc1-9021-e17922673b58" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.45:8080: connect: connection refused"
Aug 05 17:47:18.623658 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:47:18.623577    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-741a0-predictor-77484d674-xt592" podUID="30ca866f-36b3-4229-9126-d138dd30da9f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.44:8080: connect: connection refused"
Aug 05 17:47:19.695108 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:47:19.695069    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-db0ba-predictor-5cd4b48554-7xcb6" podUID="a5dc3454-c0b1-4150-aa89-3e4034c4e681" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.46:8080: connect: connection refused"
Aug 05 17:47:19.698358 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:47:19.698328    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-db0ba-predictor-68c79c6fb7-mpttp" podUID="8b670052-3f8a-4f70-82a8-d772ab208ea0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.47:8080: connect: connection refused"
Aug 05 17:47:20.638035 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:47:20.637999    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-741a0-predictor-6c788cfb87-2t5vl" podUID="04ea2238-8ad7-4bc1-9021-e17922673b58" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.45:8080: connect: connection refused"
Aug 05 17:47:28.623807 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:47:28.623768    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-741a0-predictor-77484d674-xt592" podUID="30ca866f-36b3-4229-9126-d138dd30da9f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.44:8080: connect: connection refused"
Aug 05 17:47:29.695649 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:47:29.695608    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-db0ba-predictor-5cd4b48554-7xcb6" podUID="a5dc3454-c0b1-4150-aa89-3e4034c4e681" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.46:8080: connect: connection refused"
Aug 05 17:47:29.697880 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:47:29.697853    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-db0ba-predictor-68c79c6fb7-mpttp" podUID="8b670052-3f8a-4f70-82a8-d772ab208ea0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.47:8080: connect: connection refused"
Aug 05 17:47:30.638499 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:47:30.638459    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-741a0-predictor-6c788cfb87-2t5vl" podUID="04ea2238-8ad7-4bc1-9021-e17922673b58" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.45:8080: connect: connection refused"
Aug 05 17:47:38.623742 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:47:38.623696    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-741a0-predictor-77484d674-xt592" podUID="30ca866f-36b3-4229-9126-d138dd30da9f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.44:8080: connect: connection refused"
Aug 05 17:47:39.695184 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:47:39.695143    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-db0ba-predictor-5cd4b48554-7xcb6" podUID="a5dc3454-c0b1-4150-aa89-3e4034c4e681" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.46:8080: connect: connection refused"
Aug 05 17:47:39.698461 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:47:39.698435    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-db0ba-predictor-68c79c6fb7-mpttp" podUID="8b670052-3f8a-4f70-82a8-d772ab208ea0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.47:8080: connect: connection refused"
Aug 05 17:47:40.638862 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:47:40.638827    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-741a0-predictor-6c788cfb87-2t5vl"
Aug 05 17:47:48.624988 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:47:48.624957    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-741a0-predictor-77484d674-xt592"
Aug 05 17:47:49.696554 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:47:49.696525    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-db0ba-predictor-5cd4b48554-7xcb6"
Aug 05 17:47:49.698537 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:47:49.698520    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-db0ba-predictor-68c79c6fb7-mpttp"
Aug 05 17:56:16.459585 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:56:16.459508    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-db0ba-predictor-5cd4b48554-7xcb6"]
Aug 05 17:56:16.461940 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:56:16.459723    2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-db0ba-predictor-5cd4b48554-7xcb6" podUID="a5dc3454-c0b1-4150-aa89-3e4034c4e681" containerName="kserve-container" containerID="cri-o://ad3df2a5fb51663f6d5375395e47f0ac451078a6aa07e842a8a72567ff631165" gracePeriod=30
Aug 05 17:56:16.461940 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:56:16.459767    2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-db0ba-predictor-5cd4b48554-7xcb6" podUID="a5dc3454-c0b1-4150-aa89-3e4034c4e681" containerName="kube-rbac-proxy" containerID="cri-o://2be79787d96ef2ea66149008ed22b7e16612f947ff14d9579f4df3d4c840e616" gracePeriod=30
Aug 05 17:56:16.494365 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:56:16.494338    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-db0ba-predictor-68c79c6fb7-mpttp"]
Aug 05 17:56:16.494583 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:56:16.494563    2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-db0ba-predictor-68c79c6fb7-mpttp" podUID="8b670052-3f8a-4f70-82a8-d772ab208ea0" containerName="kserve-container" containerID="cri-o://1e8c419575f0dfaeeeccaf96ae220bca2a763864c06544bffc3ff3e09b25fdb7" gracePeriod=30
Aug 05 17:56:16.494673 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:56:16.494641    2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-db0ba-predictor-68c79c6fb7-mpttp" podUID="8b670052-3f8a-4f70-82a8-d772ab208ea0" containerName="kube-rbac-proxy" containerID="cri-o://d6a8b499dd9fd29c4e05db3675052bc899051e31fe1ea8a3821b2761d0802e0e" gracePeriod=30
Aug 05 17:56:17.045439 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:56:17.045404    2571 generic.go:356] "Generic (PLEG): container finished" podID="8b670052-3f8a-4f70-82a8-d772ab208ea0" containerID="d6a8b499dd9fd29c4e05db3675052bc899051e31fe1ea8a3821b2761d0802e0e" exitCode=2
Aug 05 17:56:17.045642 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:56:17.045485    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-db0ba-predictor-68c79c6fb7-mpttp" event={"ID":"8b670052-3f8a-4f70-82a8-d772ab208ea0","Type":"ContainerDied","Data":"d6a8b499dd9fd29c4e05db3675052bc899051e31fe1ea8a3821b2761d0802e0e"}
Aug 05 17:56:17.047448 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:56:17.047420    2571 generic.go:356] "Generic (PLEG): container finished" podID="a5dc3454-c0b1-4150-aa89-3e4034c4e681" containerID="2be79787d96ef2ea66149008ed22b7e16612f947ff14d9579f4df3d4c840e616" exitCode=2
Aug 05 17:56:17.047552 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:56:17.047495    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-db0ba-predictor-5cd4b48554-7xcb6" event={"ID":"a5dc3454-c0b1-4150-aa89-3e4034c4e681","Type":"ContainerDied","Data":"2be79787d96ef2ea66149008ed22b7e16612f947ff14d9579f4df3d4c840e616"}
Aug 05 17:56:19.362922 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:56:19.362896    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-db0ba-predictor-68c79c6fb7-mpttp"
Aug 05 17:56:19.365554 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:56:19.365536    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-db0ba-predictor-5cd4b48554-7xcb6"
Aug 05 17:56:19.411896 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:56:19.411843    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-h6fx8\" (UniqueName: \"kubernetes.io/projected/a5dc3454-c0b1-4150-aa89-3e4034c4e681-kube-api-access-h6fx8\") pod \"a5dc3454-c0b1-4150-aa89-3e4034c4e681\" (UID: \"a5dc3454-c0b1-4150-aa89-3e4034c4e681\") "
Aug 05 17:56:19.412014 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:56:19.411905    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8b670052-3f8a-4f70-82a8-d772ab208ea0-proxy-tls\") pod \"8b670052-3f8a-4f70-82a8-d772ab208ea0\" (UID: \"8b670052-3f8a-4f70-82a8-d772ab208ea0\") "
Aug 05 17:56:19.412014 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:56:19.411938    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8b670052-3f8a-4f70-82a8-d772ab208ea0-kube-rbac-proxy-sar-config\") pod \"8b670052-3f8a-4f70-82a8-d772ab208ea0\" (UID: \"8b670052-3f8a-4f70-82a8-d772ab208ea0\") "
Aug 05 17:56:19.412014 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:56:19.411960    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a5dc3454-c0b1-4150-aa89-3e4034c4e681-proxy-tls\") pod \"a5dc3454-c0b1-4150-aa89-3e4034c4e681\" (UID: \"a5dc3454-c0b1-4150-aa89-3e4034c4e681\") "
Aug 05 17:56:19.412014 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:56:19.411979    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a5dc3454-c0b1-4150-aa89-3e4034c4e681-kube-rbac-proxy-sar-config\") pod \"a5dc3454-c0b1-4150-aa89-3e4034c4e681\" (UID: \"a5dc3454-c0b1-4150-aa89-3e4034c4e681\") "
Aug 05 17:56:19.412236 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:56:19.412018    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-l7kcz\" (UniqueName: \"kubernetes.io/projected/8b670052-3f8a-4f70-82a8-d772ab208ea0-kube-api-access-l7kcz\") pod \"8b670052-3f8a-4f70-82a8-d772ab208ea0\" (UID: \"8b670052-3f8a-4f70-82a8-d772ab208ea0\") "
Aug 05 17:56:19.412335 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:56:19.412307    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a5dc3454-c0b1-4150-aa89-3e4034c4e681-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "a5dc3454-c0b1-4150-aa89-3e4034c4e681" (UID: "a5dc3454-c0b1-4150-aa89-3e4034c4e681"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 05 17:56:19.412462 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:56:19.412401    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8b670052-3f8a-4f70-82a8-d772ab208ea0-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "8b670052-3f8a-4f70-82a8-d772ab208ea0" (UID: "8b670052-3f8a-4f70-82a8-d772ab208ea0"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 05 17:56:19.413956 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:56:19.413935    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a5dc3454-c0b1-4150-aa89-3e4034c4e681-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "a5dc3454-c0b1-4150-aa89-3e4034c4e681" (UID: "a5dc3454-c0b1-4150-aa89-3e4034c4e681"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 05 17:56:19.414135 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:56:19.414117    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8b670052-3f8a-4f70-82a8-d772ab208ea0-kube-api-access-l7kcz" (OuterVolumeSpecName: "kube-api-access-l7kcz") pod "8b670052-3f8a-4f70-82a8-d772ab208ea0" (UID: "8b670052-3f8a-4f70-82a8-d772ab208ea0"). InnerVolumeSpecName "kube-api-access-l7kcz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 05 17:56:19.414368 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:56:19.414354    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a5dc3454-c0b1-4150-aa89-3e4034c4e681-kube-api-access-h6fx8" (OuterVolumeSpecName: "kube-api-access-h6fx8") pod "a5dc3454-c0b1-4150-aa89-3e4034c4e681" (UID: "a5dc3454-c0b1-4150-aa89-3e4034c4e681"). InnerVolumeSpecName "kube-api-access-h6fx8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 05 17:56:19.414431 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:56:19.414367    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8b670052-3f8a-4f70-82a8-d772ab208ea0-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "8b670052-3f8a-4f70-82a8-d772ab208ea0" (UID: "8b670052-3f8a-4f70-82a8-d772ab208ea0"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 05 17:56:19.513385 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:56:19.513346    2571 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a5dc3454-c0b1-4150-aa89-3e4034c4e681-proxy-tls\") on node \"ip-10-0-135-2.ec2.internal\" DevicePath \"\""
Aug 05 17:56:19.513513 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:56:19.513413    2571 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a5dc3454-c0b1-4150-aa89-3e4034c4e681-kube-rbac-proxy-sar-config\") on node \"ip-10-0-135-2.ec2.internal\" DevicePath \"\""
Aug 05 17:56:19.513513 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:56:19.513424    2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l7kcz\" (UniqueName: \"kubernetes.io/projected/8b670052-3f8a-4f70-82a8-d772ab208ea0-kube-api-access-l7kcz\") on node \"ip-10-0-135-2.ec2.internal\" DevicePath \"\""
Aug 05 17:56:19.513513 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:56:19.513434    2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h6fx8\" (UniqueName: \"kubernetes.io/projected/a5dc3454-c0b1-4150-aa89-3e4034c4e681-kube-api-access-h6fx8\") on node \"ip-10-0-135-2.ec2.internal\" DevicePath \"\""
Aug 05 17:56:19.513513 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:56:19.513442    2571 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8b670052-3f8a-4f70-82a8-d772ab208ea0-proxy-tls\") on node \"ip-10-0-135-2.ec2.internal\" DevicePath \"\""
Aug 05 17:56:19.513513 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:56:19.513451    2571 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8b670052-3f8a-4f70-82a8-d772ab208ea0-kube-rbac-proxy-sar-config\") on node \"ip-10-0-135-2.ec2.internal\" DevicePath \"\""
Aug 05 17:56:20.061478 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:56:20.061440    2571 generic.go:356] "Generic (PLEG): container finished" podID="8b670052-3f8a-4f70-82a8-d772ab208ea0" containerID="1e8c419575f0dfaeeeccaf96ae220bca2a763864c06544bffc3ff3e09b25fdb7" exitCode=0
Aug 05 17:56:20.061681 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:56:20.061519    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-db0ba-predictor-68c79c6fb7-mpttp"
Aug 05 17:56:20.061681 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:56:20.061522    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-db0ba-predictor-68c79c6fb7-mpttp" event={"ID":"8b670052-3f8a-4f70-82a8-d772ab208ea0","Type":"ContainerDied","Data":"1e8c419575f0dfaeeeccaf96ae220bca2a763864c06544bffc3ff3e09b25fdb7"}
Aug 05 17:56:20.061797 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:56:20.061683    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-db0ba-predictor-68c79c6fb7-mpttp" event={"ID":"8b670052-3f8a-4f70-82a8-d772ab208ea0","Type":"ContainerDied","Data":"671d1fbc8188919c77cf003778f42095c8349b6b064c7bb900c4f19b6afa38d1"}
Aug 05 17:56:20.061797 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:56:20.061717    2571 scope.go:117] "RemoveContainer" containerID="d6a8b499dd9fd29c4e05db3675052bc899051e31fe1ea8a3821b2761d0802e0e"
Aug 05 17:56:20.063076 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:56:20.063056    2571 generic.go:356] "Generic (PLEG): container finished" podID="a5dc3454-c0b1-4150-aa89-3e4034c4e681" containerID="ad3df2a5fb51663f6d5375395e47f0ac451078a6aa07e842a8a72567ff631165" exitCode=0
Aug 05 17:56:20.063148 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:56:20.063114    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-db0ba-predictor-5cd4b48554-7xcb6" event={"ID":"a5dc3454-c0b1-4150-aa89-3e4034c4e681","Type":"ContainerDied","Data":"ad3df2a5fb51663f6d5375395e47f0ac451078a6aa07e842a8a72567ff631165"}
Aug 05 17:56:20.063148 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:56:20.063129    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-db0ba-predictor-5cd4b48554-7xcb6"
Aug 05 17:56:20.063269 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:56:20.063163    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-db0ba-predictor-5cd4b48554-7xcb6" event={"ID":"a5dc3454-c0b1-4150-aa89-3e4034c4e681","Type":"ContainerDied","Data":"9c43ae0a9ff2836931f3b9dd6838b11556fcd7fda2b1e6c718b4e21d9ffae08a"}
Aug 05 17:56:20.072743 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:56:20.072724    2571 scope.go:117] "RemoveContainer" containerID="1e8c419575f0dfaeeeccaf96ae220bca2a763864c06544bffc3ff3e09b25fdb7"
Aug 05 17:56:20.080009 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:56:20.079993    2571 scope.go:117] "RemoveContainer" containerID="d6a8b499dd9fd29c4e05db3675052bc899051e31fe1ea8a3821b2761d0802e0e"
Aug 05 17:56:20.080277 ip-10-0-135-2 kubenswrapper[2571]: E0805 17:56:20.080253    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d6a8b499dd9fd29c4e05db3675052bc899051e31fe1ea8a3821b2761d0802e0e\": container with ID starting with d6a8b499dd9fd29c4e05db3675052bc899051e31fe1ea8a3821b2761d0802e0e not found: ID does not exist" containerID="d6a8b499dd9fd29c4e05db3675052bc899051e31fe1ea8a3821b2761d0802e0e"
Aug 05 17:56:20.080333 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:56:20.080291    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d6a8b499dd9fd29c4e05db3675052bc899051e31fe1ea8a3821b2761d0802e0e"} err="failed to get container status \"d6a8b499dd9fd29c4e05db3675052bc899051e31fe1ea8a3821b2761d0802e0e\": rpc error: code = NotFound desc = could not find container \"d6a8b499dd9fd29c4e05db3675052bc899051e31fe1ea8a3821b2761d0802e0e\": container with ID starting with d6a8b499dd9fd29c4e05db3675052bc899051e31fe1ea8a3821b2761d0802e0e not found: ID does not exist"
Aug 05 17:56:20.080333 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:56:20.080307    2571 scope.go:117] "RemoveContainer" containerID="1e8c419575f0dfaeeeccaf96ae220bca2a763864c06544bffc3ff3e09b25fdb7"
Aug 05 17:56:20.080638 ip-10-0-135-2 kubenswrapper[2571]: E0805 17:56:20.080611    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1e8c419575f0dfaeeeccaf96ae220bca2a763864c06544bffc3ff3e09b25fdb7\": container with ID starting with 1e8c419575f0dfaeeeccaf96ae220bca2a763864c06544bffc3ff3e09b25fdb7 not found: ID does not exist" containerID="1e8c419575f0dfaeeeccaf96ae220bca2a763864c06544bffc3ff3e09b25fdb7"
Aug 05 17:56:20.080739 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:56:20.080644    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1e8c419575f0dfaeeeccaf96ae220bca2a763864c06544bffc3ff3e09b25fdb7"} err="failed to get container status \"1e8c419575f0dfaeeeccaf96ae220bca2a763864c06544bffc3ff3e09b25fdb7\": rpc error: code = NotFound desc = could not find container \"1e8c419575f0dfaeeeccaf96ae220bca2a763864c06544bffc3ff3e09b25fdb7\": container with ID starting with 1e8c419575f0dfaeeeccaf96ae220bca2a763864c06544bffc3ff3e09b25fdb7 not found: ID does not exist"
Aug 05 17:56:20.080739 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:56:20.080667    2571 scope.go:117] "RemoveContainer" containerID="2be79787d96ef2ea66149008ed22b7e16612f947ff14d9579f4df3d4c840e616"
Aug 05 17:56:20.082786 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:56:20.082755    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-db0ba-predictor-68c79c6fb7-mpttp"]
Aug 05 17:56:20.084672 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:56:20.084650    2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-db0ba-predictor-68c79c6fb7-mpttp"]
Aug 05 17:56:20.088147 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:56:20.088130    2571 scope.go:117] "RemoveContainer" containerID="ad3df2a5fb51663f6d5375395e47f0ac451078a6aa07e842a8a72567ff631165"
Aug 05 17:56:20.093941 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:56:20.093922    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-db0ba-predictor-5cd4b48554-7xcb6"]
Aug 05 17:56:20.095021 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:56:20.095009    2571 scope.go:117] "RemoveContainer" containerID="2be79787d96ef2ea66149008ed22b7e16612f947ff14d9579f4df3d4c840e616"
Aug 05 17:56:20.095268 ip-10-0-135-2 kubenswrapper[2571]: E0805 17:56:20.095252    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2be79787d96ef2ea66149008ed22b7e16612f947ff14d9579f4df3d4c840e616\": container with ID starting with 2be79787d96ef2ea66149008ed22b7e16612f947ff14d9579f4df3d4c840e616 not found: ID does not exist" containerID="2be79787d96ef2ea66149008ed22b7e16612f947ff14d9579f4df3d4c840e616"
Aug 05 17:56:20.095316 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:56:20.095274    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2be79787d96ef2ea66149008ed22b7e16612f947ff14d9579f4df3d4c840e616"} err="failed to get container status \"2be79787d96ef2ea66149008ed22b7e16612f947ff14d9579f4df3d4c840e616\": rpc error: code = NotFound desc = could not find container \"2be79787d96ef2ea66149008ed22b7e16612f947ff14d9579f4df3d4c840e616\": container with ID starting with 2be79787d96ef2ea66149008ed22b7e16612f947ff14d9579f4df3d4c840e616 not found: ID does not exist"
Aug 05 17:56:20.095316 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:56:20.095288    2571 scope.go:117] "RemoveContainer" containerID="ad3df2a5fb51663f6d5375395e47f0ac451078a6aa07e842a8a72567ff631165"
Aug 05 17:56:20.095505 ip-10-0-135-2 kubenswrapper[2571]: E0805 17:56:20.095487    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ad3df2a5fb51663f6d5375395e47f0ac451078a6aa07e842a8a72567ff631165\": container with ID starting with ad3df2a5fb51663f6d5375395e47f0ac451078a6aa07e842a8a72567ff631165 not found: ID does not exist" containerID="ad3df2a5fb51663f6d5375395e47f0ac451078a6aa07e842a8a72567ff631165"
Aug 05 17:56:20.095557 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:56:20.095509    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ad3df2a5fb51663f6d5375395e47f0ac451078a6aa07e842a8a72567ff631165"} err="failed to get container status \"ad3df2a5fb51663f6d5375395e47f0ac451078a6aa07e842a8a72567ff631165\": rpc error: code = NotFound desc = could not find container \"ad3df2a5fb51663f6d5375395e47f0ac451078a6aa07e842a8a72567ff631165\": container with ID starting with ad3df2a5fb51663f6d5375395e47f0ac451078a6aa07e842a8a72567ff631165 not found: ID does not exist"
Aug 05 17:56:20.097840 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:56:20.097819    2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-db0ba-predictor-5cd4b48554-7xcb6"]
Aug 05 17:56:21.135801 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:56:21.135770    2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8b670052-3f8a-4f70-82a8-d772ab208ea0" path="/var/lib/kubelet/pods/8b670052-3f8a-4f70-82a8-d772ab208ea0/volumes"
Aug 05 17:56:21.136197 ip-10-0-135-2 kubenswrapper[2571]: I0805 17:56:21.136183    2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a5dc3454-c0b1-4150-aa89-3e4034c4e681" path="/var/lib/kubelet/pods/a5dc3454-c0b1-4150-aa89-3e4034c4e681/volumes"
Aug 05 18:04:20.676429 ip-10-0-135-2 kubenswrapper[2571]: I0805 18:04:20.676326    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-741a0-predictor-77484d674-xt592"]
Aug 05 18:04:20.678957 ip-10-0-135-2 kubenswrapper[2571]: I0805 18:04:20.676614    2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-741a0-predictor-77484d674-xt592" podUID="30ca866f-36b3-4229-9126-d138dd30da9f" containerName="kserve-container" containerID="cri-o://becf095cf95ec3d57d9c36da8df12a1db8e814a989c511d41a41f47e0a1630ac" gracePeriod=30
Aug 05 18:04:20.678957 ip-10-0-135-2 kubenswrapper[2571]: I0805 18:04:20.676641    2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-741a0-predictor-77484d674-xt592" podUID="30ca866f-36b3-4229-9126-d138dd30da9f" containerName="kube-rbac-proxy" containerID="cri-o://142fa6953c9d9653a5008612e1b9bdf57cc8d62d3d42fb91fa2364ecf4bdf441" gracePeriod=30
Aug 05 18:04:20.722075 ip-10-0-135-2 kubenswrapper[2571]: I0805 18:04:20.722049    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-741a0-predictor-6c788cfb87-2t5vl"]
Aug 05 18:04:20.722276 ip-10-0-135-2 kubenswrapper[2571]: I0805 18:04:20.722254    2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-741a0-predictor-6c788cfb87-2t5vl" podUID="04ea2238-8ad7-4bc1-9021-e17922673b58" containerName="kserve-container" containerID="cri-o://e4f99ff07f2672d0b6e789b7e306335817c6aa20c0d3a64f8cc6f4b31453be03" gracePeriod=30
Aug 05 18:04:20.722358 ip-10-0-135-2 kubenswrapper[2571]: I0805 18:04:20.722301    2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-741a0-predictor-6c788cfb87-2t5vl" podUID="04ea2238-8ad7-4bc1-9021-e17922673b58" containerName="kube-rbac-proxy" containerID="cri-o://cef1a82358db665d62f49227fce1b8d19be3f73f80fb8f2c0640311cbd05ff50" gracePeriod=30
Aug 05 18:04:21.021443 ip-10-0-135-2 kubenswrapper[2571]: I0805 18:04:21.021403    2571 generic.go:356] "Generic (PLEG): container finished" podID="30ca866f-36b3-4229-9126-d138dd30da9f" containerID="142fa6953c9d9653a5008612e1b9bdf57cc8d62d3d42fb91fa2364ecf4bdf441" exitCode=2
Aug 05 18:04:21.021625 ip-10-0-135-2 kubenswrapper[2571]: I0805 18:04:21.021473    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-741a0-predictor-77484d674-xt592" event={"ID":"30ca866f-36b3-4229-9126-d138dd30da9f","Type":"ContainerDied","Data":"142fa6953c9d9653a5008612e1b9bdf57cc8d62d3d42fb91fa2364ecf4bdf441"}
Aug 05 18:04:21.023114 ip-10-0-135-2 kubenswrapper[2571]: I0805 18:04:21.023094    2571 generic.go:356] "Generic (PLEG): container finished" podID="04ea2238-8ad7-4bc1-9021-e17922673b58" containerID="cef1a82358db665d62f49227fce1b8d19be3f73f80fb8f2c0640311cbd05ff50" exitCode=2
Aug 05 18:04:21.023232 ip-10-0-135-2 kubenswrapper[2571]: I0805 18:04:21.023168    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-741a0-predictor-6c788cfb87-2t5vl" event={"ID":"04ea2238-8ad7-4bc1-9021-e17922673b58","Type":"ContainerDied","Data":"cef1a82358db665d62f49227fce1b8d19be3f73f80fb8f2c0640311cbd05ff50"}
Aug 05 18:04:23.619297 ip-10-0-135-2 kubenswrapper[2571]: I0805 18:04:23.619254    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-741a0-predictor-77484d674-xt592" podUID="30ca866f-36b3-4229-9126-d138dd30da9f" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.44:8643/healthz\": dial tcp 10.134.0.44:8643: connect: connection refused"
Aug 05 18:04:23.895861 ip-10-0-135-2 kubenswrapper[2571]: I0805 18:04:23.895839    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-741a0-predictor-77484d674-xt592"
Aug 05 18:04:23.898663 ip-10-0-135-2 kubenswrapper[2571]: I0805 18:04:23.898644    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-741a0-predictor-6c788cfb87-2t5vl"
Aug 05 18:04:23.989548 ip-10-0-135-2 kubenswrapper[2571]: I0805 18:04:23.989522    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/04ea2238-8ad7-4bc1-9021-e17922673b58-kube-rbac-proxy-sar-config\") pod \"04ea2238-8ad7-4bc1-9021-e17922673b58\" (UID: \"04ea2238-8ad7-4bc1-9021-e17922673b58\") "
Aug 05 18:04:23.989676 ip-10-0-135-2 kubenswrapper[2571]: I0805 18:04:23.989571    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-d77p8\" (UniqueName: \"kubernetes.io/projected/30ca866f-36b3-4229-9126-d138dd30da9f-kube-api-access-d77p8\") pod \"30ca866f-36b3-4229-9126-d138dd30da9f\" (UID: \"30ca866f-36b3-4229-9126-d138dd30da9f\") "
Aug 05 18:04:23.989676 ip-10-0-135-2 kubenswrapper[2571]: I0805 18:04:23.989605    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/30ca866f-36b3-4229-9126-d138dd30da9f-proxy-tls\") pod \"30ca866f-36b3-4229-9126-d138dd30da9f\" (UID: \"30ca866f-36b3-4229-9126-d138dd30da9f\") "
Aug 05 18:04:23.989834 ip-10-0-135-2 kubenswrapper[2571]: I0805 18:04:23.989716    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/30ca866f-36b3-4229-9126-d138dd30da9f-kube-rbac-proxy-sar-config\") pod \"30ca866f-36b3-4229-9126-d138dd30da9f\" (UID: \"30ca866f-36b3-4229-9126-d138dd30da9f\") "
Aug 05 18:04:23.989834 ip-10-0-135-2 kubenswrapper[2571]: I0805 18:04:23.989801    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gp9l2\" (UniqueName: \"kubernetes.io/projected/04ea2238-8ad7-4bc1-9021-e17922673b58-kube-api-access-gp9l2\") pod \"04ea2238-8ad7-4bc1-9021-e17922673b58\" (UID: \"04ea2238-8ad7-4bc1-9021-e17922673b58\") "
Aug 05 18:04:23.989948 ip-10-0-135-2 kubenswrapper[2571]: I0805 18:04:23.989898    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/04ea2238-8ad7-4bc1-9021-e17922673b58-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "04ea2238-8ad7-4bc1-9021-e17922673b58" (UID: "04ea2238-8ad7-4bc1-9021-e17922673b58"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 05 18:04:23.989948 ip-10-0-135-2 kubenswrapper[2571]: I0805 18:04:23.989926    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/04ea2238-8ad7-4bc1-9021-e17922673b58-proxy-tls\") pod \"04ea2238-8ad7-4bc1-9021-e17922673b58\" (UID: \"04ea2238-8ad7-4bc1-9021-e17922673b58\") "
Aug 05 18:04:23.990123 ip-10-0-135-2 kubenswrapper[2571]: I0805 18:04:23.990103    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/30ca866f-36b3-4229-9126-d138dd30da9f-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "30ca866f-36b3-4229-9126-d138dd30da9f" (UID: "30ca866f-36b3-4229-9126-d138dd30da9f"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 05 18:04:23.990185 ip-10-0-135-2 kubenswrapper[2571]: I0805 18:04:23.990170    2571 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/04ea2238-8ad7-4bc1-9021-e17922673b58-kube-rbac-proxy-sar-config\") on node \"ip-10-0-135-2.ec2.internal\" DevicePath \"\""
Aug 05 18:04:23.990239 ip-10-0-135-2 kubenswrapper[2571]: I0805 18:04:23.990188    2571 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/30ca866f-36b3-4229-9126-d138dd30da9f-kube-rbac-proxy-sar-config\") on node \"ip-10-0-135-2.ec2.internal\" DevicePath \"\""
Aug 05 18:04:23.991862 ip-10-0-135-2 kubenswrapper[2571]: I0805 18:04:23.991836    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/04ea2238-8ad7-4bc1-9021-e17922673b58-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "04ea2238-8ad7-4bc1-9021-e17922673b58" (UID: "04ea2238-8ad7-4bc1-9021-e17922673b58"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 05 18:04:23.991862 ip-10-0-135-2 kubenswrapper[2571]: I0805 18:04:23.991847    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/04ea2238-8ad7-4bc1-9021-e17922673b58-kube-api-access-gp9l2" (OuterVolumeSpecName: "kube-api-access-gp9l2") pod "04ea2238-8ad7-4bc1-9021-e17922673b58" (UID: "04ea2238-8ad7-4bc1-9021-e17922673b58"). InnerVolumeSpecName "kube-api-access-gp9l2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 05 18:04:23.992131 ip-10-0-135-2 kubenswrapper[2571]: I0805 18:04:23.992115    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/30ca866f-36b3-4229-9126-d138dd30da9f-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "30ca866f-36b3-4229-9126-d138dd30da9f" (UID: "30ca866f-36b3-4229-9126-d138dd30da9f"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 05 18:04:23.992180 ip-10-0-135-2 kubenswrapper[2571]: I0805 18:04:23.992118    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/30ca866f-36b3-4229-9126-d138dd30da9f-kube-api-access-d77p8" (OuterVolumeSpecName: "kube-api-access-d77p8") pod "30ca866f-36b3-4229-9126-d138dd30da9f" (UID: "30ca866f-36b3-4229-9126-d138dd30da9f"). InnerVolumeSpecName "kube-api-access-d77p8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 05 18:04:24.036630 ip-10-0-135-2 kubenswrapper[2571]: I0805 18:04:24.036604    2571 generic.go:356] "Generic (PLEG): container finished" podID="30ca866f-36b3-4229-9126-d138dd30da9f" containerID="becf095cf95ec3d57d9c36da8df12a1db8e814a989c511d41a41f47e0a1630ac" exitCode=0
Aug 05 18:04:24.036761 ip-10-0-135-2 kubenswrapper[2571]: I0805 18:04:24.036648    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-741a0-predictor-77484d674-xt592"
Aug 05 18:04:24.036761 ip-10-0-135-2 kubenswrapper[2571]: I0805 18:04:24.036689    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-741a0-predictor-77484d674-xt592" event={"ID":"30ca866f-36b3-4229-9126-d138dd30da9f","Type":"ContainerDied","Data":"becf095cf95ec3d57d9c36da8df12a1db8e814a989c511d41a41f47e0a1630ac"}
Aug 05 18:04:24.036855 ip-10-0-135-2 kubenswrapper[2571]: I0805 18:04:24.036758    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-741a0-predictor-77484d674-xt592" event={"ID":"30ca866f-36b3-4229-9126-d138dd30da9f","Type":"ContainerDied","Data":"1fcf45720e03332b42399961aca9785b9d1cca0495a427a86d895400b0a9d6c3"}
Aug 05 18:04:24.036855 ip-10-0-135-2 kubenswrapper[2571]: I0805 18:04:24.036799    2571 scope.go:117] "RemoveContainer" containerID="142fa6953c9d9653a5008612e1b9bdf57cc8d62d3d42fb91fa2364ecf4bdf441"
Aug 05 18:04:24.038316 ip-10-0-135-2 kubenswrapper[2571]: I0805 18:04:24.038294    2571 generic.go:356] "Generic (PLEG): container finished" podID="04ea2238-8ad7-4bc1-9021-e17922673b58" containerID="e4f99ff07f2672d0b6e789b7e306335817c6aa20c0d3a64f8cc6f4b31453be03" exitCode=0
Aug 05 18:04:24.038415 ip-10-0-135-2 kubenswrapper[2571]: I0805 18:04:24.038341    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-741a0-predictor-6c788cfb87-2t5vl"
Aug 05 18:04:24.038415 ip-10-0-135-2 kubenswrapper[2571]: I0805 18:04:24.038355    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-741a0-predictor-6c788cfb87-2t5vl" event={"ID":"04ea2238-8ad7-4bc1-9021-e17922673b58","Type":"ContainerDied","Data":"e4f99ff07f2672d0b6e789b7e306335817c6aa20c0d3a64f8cc6f4b31453be03"}
Aug 05 18:04:24.038504 ip-10-0-135-2 kubenswrapper[2571]: I0805 18:04:24.038425    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-741a0-predictor-6c788cfb87-2t5vl" event={"ID":"04ea2238-8ad7-4bc1-9021-e17922673b58","Type":"ContainerDied","Data":"dcd8d3d9653df8db7a013d300d067e66be93f5306ae0e17d7ea23cb034089045"}
Aug 05 18:04:24.045580 ip-10-0-135-2 kubenswrapper[2571]: I0805 18:04:24.045565    2571 scope.go:117] "RemoveContainer" containerID="becf095cf95ec3d57d9c36da8df12a1db8e814a989c511d41a41f47e0a1630ac"
Aug 05 18:04:24.052988 ip-10-0-135-2 kubenswrapper[2571]: I0805 18:04:24.052968    2571 scope.go:117] "RemoveContainer" containerID="142fa6953c9d9653a5008612e1b9bdf57cc8d62d3d42fb91fa2364ecf4bdf441"
Aug 05 18:04:24.053264 ip-10-0-135-2 kubenswrapper[2571]: E0805 18:04:24.053247    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"142fa6953c9d9653a5008612e1b9bdf57cc8d62d3d42fb91fa2364ecf4bdf441\": container with ID starting with 142fa6953c9d9653a5008612e1b9bdf57cc8d62d3d42fb91fa2364ecf4bdf441 not found: ID does not exist" containerID="142fa6953c9d9653a5008612e1b9bdf57cc8d62d3d42fb91fa2364ecf4bdf441"
Aug 05 18:04:24.053316 ip-10-0-135-2 kubenswrapper[2571]: I0805 18:04:24.053272    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"142fa6953c9d9653a5008612e1b9bdf57cc8d62d3d42fb91fa2364ecf4bdf441"} err="failed to get container status \"142fa6953c9d9653a5008612e1b9bdf57cc8d62d3d42fb91fa2364ecf4bdf441\": rpc error: code = NotFound desc = could not find container \"142fa6953c9d9653a5008612e1b9bdf57cc8d62d3d42fb91fa2364ecf4bdf441\": container with ID starting with 142fa6953c9d9653a5008612e1b9bdf57cc8d62d3d42fb91fa2364ecf4bdf441 not found: ID does not exist"
Aug 05 18:04:24.053316 ip-10-0-135-2 kubenswrapper[2571]: I0805 18:04:24.053289    2571 scope.go:117] "RemoveContainer" containerID="becf095cf95ec3d57d9c36da8df12a1db8e814a989c511d41a41f47e0a1630ac"
Aug 05 18:04:24.053544 ip-10-0-135-2 kubenswrapper[2571]: E0805 18:04:24.053527    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"becf095cf95ec3d57d9c36da8df12a1db8e814a989c511d41a41f47e0a1630ac\": container with ID starting with becf095cf95ec3d57d9c36da8df12a1db8e814a989c511d41a41f47e0a1630ac not found: ID does not exist" containerID="becf095cf95ec3d57d9c36da8df12a1db8e814a989c511d41a41f47e0a1630ac"
Aug 05 18:04:24.053596 ip-10-0-135-2 kubenswrapper[2571]: I0805 18:04:24.053550    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"becf095cf95ec3d57d9c36da8df12a1db8e814a989c511d41a41f47e0a1630ac"} err="failed to get container status \"becf095cf95ec3d57d9c36da8df12a1db8e814a989c511d41a41f47e0a1630ac\": rpc error: code = NotFound desc = could not find container \"becf095cf95ec3d57d9c36da8df12a1db8e814a989c511d41a41f47e0a1630ac\": container with ID starting with becf095cf95ec3d57d9c36da8df12a1db8e814a989c511d41a41f47e0a1630ac not found: ID does not exist"
Aug 05 18:04:24.053596 ip-10-0-135-2 kubenswrapper[2571]: I0805 18:04:24.053566    2571 scope.go:117] "RemoveContainer" containerID="cef1a82358db665d62f49227fce1b8d19be3f73f80fb8f2c0640311cbd05ff50"
Aug 05 18:04:24.060747 ip-10-0-135-2 kubenswrapper[2571]: I0805 18:04:24.060731    2571 scope.go:117] "RemoveContainer" containerID="e4f99ff07f2672d0b6e789b7e306335817c6aa20c0d3a64f8cc6f4b31453be03"
Aug 05 18:04:24.061665 ip-10-0-135-2 kubenswrapper[2571]: I0805 18:04:24.061604    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-741a0-predictor-6c788cfb87-2t5vl"]
Aug 05 18:04:24.067115 ip-10-0-135-2 kubenswrapper[2571]: I0805 18:04:24.067092    2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-741a0-predictor-6c788cfb87-2t5vl"]
Aug 05 18:04:24.068141 ip-10-0-135-2 kubenswrapper[2571]: I0805 18:04:24.068123    2571 scope.go:117] "RemoveContainer" containerID="cef1a82358db665d62f49227fce1b8d19be3f73f80fb8f2c0640311cbd05ff50"
Aug 05 18:04:24.068487 ip-10-0-135-2 kubenswrapper[2571]: E0805 18:04:24.068462    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cef1a82358db665d62f49227fce1b8d19be3f73f80fb8f2c0640311cbd05ff50\": container with ID starting with cef1a82358db665d62f49227fce1b8d19be3f73f80fb8f2c0640311cbd05ff50 not found: ID does not exist" containerID="cef1a82358db665d62f49227fce1b8d19be3f73f80fb8f2c0640311cbd05ff50"
Aug 05 18:04:24.068564 ip-10-0-135-2 kubenswrapper[2571]: I0805 18:04:24.068496    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cef1a82358db665d62f49227fce1b8d19be3f73f80fb8f2c0640311cbd05ff50"} err="failed to get container status \"cef1a82358db665d62f49227fce1b8d19be3f73f80fb8f2c0640311cbd05ff50\": rpc error: code = NotFound desc = could not find container \"cef1a82358db665d62f49227fce1b8d19be3f73f80fb8f2c0640311cbd05ff50\": container with ID starting with cef1a82358db665d62f49227fce1b8d19be3f73f80fb8f2c0640311cbd05ff50 not found: ID does not exist"
Aug 05 18:04:24.068564 ip-10-0-135-2 kubenswrapper[2571]: I0805 18:04:24.068518    2571 scope.go:117] "RemoveContainer" containerID="e4f99ff07f2672d0b6e789b7e306335817c6aa20c0d3a64f8cc6f4b31453be03"
Aug 05 18:04:24.068816 ip-10-0-135-2 kubenswrapper[2571]: E0805 18:04:24.068796    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e4f99ff07f2672d0b6e789b7e306335817c6aa20c0d3a64f8cc6f4b31453be03\": container with ID starting with e4f99ff07f2672d0b6e789b7e306335817c6aa20c0d3a64f8cc6f4b31453be03 not found: ID does not exist" containerID="e4f99ff07f2672d0b6e789b7e306335817c6aa20c0d3a64f8cc6f4b31453be03"
Aug 05 18:04:24.068872 ip-10-0-135-2 kubenswrapper[2571]: I0805 18:04:24.068825    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e4f99ff07f2672d0b6e789b7e306335817c6aa20c0d3a64f8cc6f4b31453be03"} err="failed to get container status \"e4f99ff07f2672d0b6e789b7e306335817c6aa20c0d3a64f8cc6f4b31453be03\": rpc error: code = NotFound desc = could not find container \"e4f99ff07f2672d0b6e789b7e306335817c6aa20c0d3a64f8cc6f4b31453be03\": container with ID starting with e4f99ff07f2672d0b6e789b7e306335817c6aa20c0d3a64f8cc6f4b31453be03 not found: ID does not exist"
Aug 05 18:04:24.077404 ip-10-0-135-2 kubenswrapper[2571]: I0805 18:04:24.077368    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-741a0-predictor-77484d674-xt592"]
Aug 05 18:04:24.079579 ip-10-0-135-2 kubenswrapper[2571]: I0805 18:04:24.079558    2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-741a0-predictor-77484d674-xt592"]
Aug 05 18:04:24.091361 ip-10-0-135-2 kubenswrapper[2571]: I0805 18:04:24.091338    2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gp9l2\" (UniqueName: \"kubernetes.io/projected/04ea2238-8ad7-4bc1-9021-e17922673b58-kube-api-access-gp9l2\") on node \"ip-10-0-135-2.ec2.internal\" DevicePath \"\""
Aug 05 18:04:24.091361 ip-10-0-135-2 kubenswrapper[2571]: I0805 18:04:24.091359    2571 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/04ea2238-8ad7-4bc1-9021-e17922673b58-proxy-tls\") on node \"ip-10-0-135-2.ec2.internal\" DevicePath \"\""
Aug 05 18:04:24.091551 ip-10-0-135-2 kubenswrapper[2571]: I0805 18:04:24.091369    2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d77p8\" (UniqueName: \"kubernetes.io/projected/30ca866f-36b3-4229-9126-d138dd30da9f-kube-api-access-d77p8\") on node \"ip-10-0-135-2.ec2.internal\" DevicePath \"\""
Aug 05 18:04:24.091551 ip-10-0-135-2 kubenswrapper[2571]: I0805 18:04:24.091400    2571 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/30ca866f-36b3-4229-9126-d138dd30da9f-proxy-tls\") on node \"ip-10-0-135-2.ec2.internal\" DevicePath \"\""
Aug 05 18:04:25.136580 ip-10-0-135-2 kubenswrapper[2571]: I0805 18:04:25.136545    2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="04ea2238-8ad7-4bc1-9021-e17922673b58" path="/var/lib/kubelet/pods/04ea2238-8ad7-4bc1-9021-e17922673b58/volumes"
Aug 05 18:04:25.137157 ip-10-0-135-2 kubenswrapper[2571]: I0805 18:04:25.137136    2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="30ca866f-36b3-4229-9126-d138dd30da9f" path="/var/lib/kubelet/pods/30ca866f-36b3-4229-9126-d138dd30da9f/volumes"
Aug 05 18:04:48.961500 ip-10-0-135-2 kubenswrapper[2571]: I0805 18:04:48.961468    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-bm58p/must-gather-lzkc9"]
Aug 05 18:04:48.961875 ip-10-0-135-2 kubenswrapper[2571]: I0805 18:04:48.961763    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="45df6046-112b-493c-b9bf-b746d05c3a21" containerName="kserve-container"
Aug 05 18:04:48.961875 ip-10-0-135-2 kubenswrapper[2571]: I0805 18:04:48.961773    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="45df6046-112b-493c-b9bf-b746d05c3a21" containerName="kserve-container"
Aug 05 18:04:48.961875 ip-10-0-135-2 kubenswrapper[2571]: I0805 18:04:48.961780    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8b670052-3f8a-4f70-82a8-d772ab208ea0" containerName="kserve-container"
Aug 05 18:04:48.961875 ip-10-0-135-2 kubenswrapper[2571]: I0805 18:04:48.961786    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="8b670052-3f8a-4f70-82a8-d772ab208ea0" containerName="kserve-container"
Aug 05 18:04:48.961875 ip-10-0-135-2 kubenswrapper[2571]: I0805 18:04:48.961798    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a5dc3454-c0b1-4150-aa89-3e4034c4e681" containerName="kserve-container"
Aug 05 18:04:48.961875 ip-10-0-135-2 kubenswrapper[2571]: I0805 18:04:48.961803    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="a5dc3454-c0b1-4150-aa89-3e4034c4e681" containerName="kserve-container"
Aug 05 18:04:48.961875 ip-10-0-135-2 kubenswrapper[2571]: I0805 18:04:48.961811    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a5dc3454-c0b1-4150-aa89-3e4034c4e681" containerName="kube-rbac-proxy"
Aug 05 18:04:48.961875 ip-10-0-135-2 kubenswrapper[2571]: I0805 18:04:48.961816    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="a5dc3454-c0b1-4150-aa89-3e4034c4e681" containerName="kube-rbac-proxy"
Aug 05 18:04:48.961875 ip-10-0-135-2 kubenswrapper[2571]: I0805 18:04:48.961827    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8b670052-3f8a-4f70-82a8-d772ab208ea0" containerName="kube-rbac-proxy"
Aug 05 18:04:48.961875 ip-10-0-135-2 kubenswrapper[2571]: I0805 18:04:48.961832    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="8b670052-3f8a-4f70-82a8-d772ab208ea0" containerName="kube-rbac-proxy"
Aug 05 18:04:48.961875 ip-10-0-135-2 kubenswrapper[2571]: I0805 18:04:48.961839    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="04ea2238-8ad7-4bc1-9021-e17922673b58" containerName="kserve-container"
Aug 05 18:04:48.961875 ip-10-0-135-2 kubenswrapper[2571]: I0805 18:04:48.961844    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="04ea2238-8ad7-4bc1-9021-e17922673b58" containerName="kserve-container"
Aug 05 18:04:48.961875 ip-10-0-135-2 kubenswrapper[2571]: I0805 18:04:48.961851    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="45df6046-112b-493c-b9bf-b746d05c3a21" containerName="kube-rbac-proxy"
Aug 05 18:04:48.961875 ip-10-0-135-2 kubenswrapper[2571]: I0805 18:04:48.961856    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="45df6046-112b-493c-b9bf-b746d05c3a21" containerName="kube-rbac-proxy"
Aug 05 18:04:48.961875 ip-10-0-135-2 kubenswrapper[2571]: I0805 18:04:48.961862    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="04ea2238-8ad7-4bc1-9021-e17922673b58" containerName="kube-rbac-proxy"
Aug 05 18:04:48.961875 ip-10-0-135-2 kubenswrapper[2571]: I0805 18:04:48.961868    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="04ea2238-8ad7-4bc1-9021-e17922673b58" containerName="kube-rbac-proxy"
Aug 05 18:04:48.961875 ip-10-0-135-2 kubenswrapper[2571]: I0805 18:04:48.961875    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="337b1675-3909-4f89-a905-35432d0ff577" containerName="kube-rbac-proxy"
Aug 05 18:04:48.961875 ip-10-0-135-2 kubenswrapper[2571]: I0805 18:04:48.961881    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="337b1675-3909-4f89-a905-35432d0ff577" containerName="kube-rbac-proxy"
Aug 05 18:04:48.962446 ip-10-0-135-2 kubenswrapper[2571]: I0805 18:04:48.961887    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="30ca866f-36b3-4229-9126-d138dd30da9f" containerName="kube-rbac-proxy"
Aug 05 18:04:48.962446 ip-10-0-135-2 kubenswrapper[2571]: I0805 18:04:48.961892    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="30ca866f-36b3-4229-9126-d138dd30da9f" containerName="kube-rbac-proxy"
Aug 05 18:04:48.962446 ip-10-0-135-2 kubenswrapper[2571]: I0805 18:04:48.961900    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="337b1675-3909-4f89-a905-35432d0ff577" containerName="kserve-container"
Aug 05 18:04:48.962446 ip-10-0-135-2 kubenswrapper[2571]: I0805 18:04:48.961905    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="337b1675-3909-4f89-a905-35432d0ff577" containerName="kserve-container"
Aug 05 18:04:48.962446 ip-10-0-135-2 kubenswrapper[2571]: I0805 18:04:48.961913    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="30ca866f-36b3-4229-9126-d138dd30da9f" containerName="kserve-container"
Aug 05 18:04:48.962446 ip-10-0-135-2 kubenswrapper[2571]: I0805 18:04:48.961918    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="30ca866f-36b3-4229-9126-d138dd30da9f" containerName="kserve-container"
Aug 05 18:04:48.962446 ip-10-0-135-2 kubenswrapper[2571]: I0805 18:04:48.961964    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="a5dc3454-c0b1-4150-aa89-3e4034c4e681" containerName="kserve-container"
Aug 05 18:04:48.962446 ip-10-0-135-2 kubenswrapper[2571]: I0805 18:04:48.961971    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="8b670052-3f8a-4f70-82a8-d772ab208ea0" containerName="kube-rbac-proxy"
Aug 05 18:04:48.962446 ip-10-0-135-2 kubenswrapper[2571]: I0805 18:04:48.961977    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="337b1675-3909-4f89-a905-35432d0ff577" containerName="kube-rbac-proxy"
Aug 05 18:04:48.962446 ip-10-0-135-2 kubenswrapper[2571]: I0805 18:04:48.961984    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="45df6046-112b-493c-b9bf-b746d05c3a21" containerName="kube-rbac-proxy"
Aug 05 18:04:48.962446 ip-10-0-135-2 kubenswrapper[2571]: I0805 18:04:48.961989    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="a5dc3454-c0b1-4150-aa89-3e4034c4e681" containerName="kube-rbac-proxy"
Aug 05 18:04:48.962446 ip-10-0-135-2 kubenswrapper[2571]: I0805 18:04:48.961995    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="337b1675-3909-4f89-a905-35432d0ff577" containerName="kserve-container"
Aug 05 18:04:48.962446 ip-10-0-135-2 kubenswrapper[2571]: I0805 18:04:48.962001    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="30ca866f-36b3-4229-9126-d138dd30da9f" containerName="kserve-container"
Aug 05 18:04:48.962446 ip-10-0-135-2 kubenswrapper[2571]: I0805 18:04:48.962005    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="30ca866f-36b3-4229-9126-d138dd30da9f" containerName="kube-rbac-proxy"
Aug 05 18:04:48.962446 ip-10-0-135-2 kubenswrapper[2571]: I0805 18:04:48.962011    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="04ea2238-8ad7-4bc1-9021-e17922673b58" containerName="kserve-container"
Aug 05 18:04:48.962446 ip-10-0-135-2 kubenswrapper[2571]: I0805 18:04:48.962017    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="45df6046-112b-493c-b9bf-b746d05c3a21" containerName="kserve-container"
Aug 05 18:04:48.962446 ip-10-0-135-2 kubenswrapper[2571]: I0805 18:04:48.962022    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="04ea2238-8ad7-4bc1-9021-e17922673b58" containerName="kube-rbac-proxy"
Aug 05 18:04:48.962446 ip-10-0-135-2 kubenswrapper[2571]: I0805 18:04:48.962026    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="8b670052-3f8a-4f70-82a8-d772ab208ea0" containerName="kserve-container"
Aug 05 18:04:48.964921 ip-10-0-135-2 kubenswrapper[2571]: I0805 18:04:48.964902    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-bm58p/must-gather-lzkc9"
Aug 05 18:04:48.967055 ip-10-0-135-2 kubenswrapper[2571]: I0805 18:04:48.967036    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-bm58p\"/\"kube-root-ca.crt\""
Aug 05 18:04:48.967172 ip-10-0-135-2 kubenswrapper[2571]: I0805 18:04:48.967036    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-bm58p\"/\"openshift-service-ca.crt\""
Aug 05 18:04:48.967172 ip-10-0-135-2 kubenswrapper[2571]: I0805 18:04:48.967035    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-bm58p\"/\"default-dockercfg-q7pnh\""
Aug 05 18:04:48.971845 ip-10-0-135-2 kubenswrapper[2571]: I0805 18:04:48.971506    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-bm58p/must-gather-lzkc9"]
Aug 05 18:04:49.081700 ip-10-0-135-2 kubenswrapper[2571]: I0805 18:04:49.081663    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pgj64\" (UniqueName: \"kubernetes.io/projected/9ccae2b4-040a-4a69-bc10-5d0c2609973a-kube-api-access-pgj64\") pod \"must-gather-lzkc9\" (UID: \"9ccae2b4-040a-4a69-bc10-5d0c2609973a\") " pod="openshift-must-gather-bm58p/must-gather-lzkc9"
Aug 05 18:04:49.081837 ip-10-0-135-2 kubenswrapper[2571]: I0805 18:04:49.081712    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/9ccae2b4-040a-4a69-bc10-5d0c2609973a-must-gather-output\") pod \"must-gather-lzkc9\" (UID: \"9ccae2b4-040a-4a69-bc10-5d0c2609973a\") " pod="openshift-must-gather-bm58p/must-gather-lzkc9"
Aug 05 18:04:49.182785 ip-10-0-135-2 kubenswrapper[2571]: I0805 18:04:49.182756    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pgj64\" (UniqueName: \"kubernetes.io/projected/9ccae2b4-040a-4a69-bc10-5d0c2609973a-kube-api-access-pgj64\") pod \"must-gather-lzkc9\" (UID: \"9ccae2b4-040a-4a69-bc10-5d0c2609973a\") " pod="openshift-must-gather-bm58p/must-gather-lzkc9"
Aug 05 18:04:49.182916 ip-10-0-135-2 kubenswrapper[2571]: I0805 18:04:49.182795    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/9ccae2b4-040a-4a69-bc10-5d0c2609973a-must-gather-output\") pod \"must-gather-lzkc9\" (UID: \"9ccae2b4-040a-4a69-bc10-5d0c2609973a\") " pod="openshift-must-gather-bm58p/must-gather-lzkc9"
Aug 05 18:04:49.183139 ip-10-0-135-2 kubenswrapper[2571]: I0805 18:04:49.183124    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/9ccae2b4-040a-4a69-bc10-5d0c2609973a-must-gather-output\") pod \"must-gather-lzkc9\" (UID: \"9ccae2b4-040a-4a69-bc10-5d0c2609973a\") " pod="openshift-must-gather-bm58p/must-gather-lzkc9"
Aug 05 18:04:49.190635 ip-10-0-135-2 kubenswrapper[2571]: I0805 18:04:49.190611    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pgj64\" (UniqueName: \"kubernetes.io/projected/9ccae2b4-040a-4a69-bc10-5d0c2609973a-kube-api-access-pgj64\") pod \"must-gather-lzkc9\" (UID: \"9ccae2b4-040a-4a69-bc10-5d0c2609973a\") " pod="openshift-must-gather-bm58p/must-gather-lzkc9"
Aug 05 18:04:49.274678 ip-10-0-135-2 kubenswrapper[2571]: I0805 18:04:49.274656    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-bm58p/must-gather-lzkc9"
Aug 05 18:04:49.399508 ip-10-0-135-2 kubenswrapper[2571]: I0805 18:04:49.399484    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-bm58p/must-gather-lzkc9"]
Aug 05 18:04:49.401853 ip-10-0-135-2 kubenswrapper[2571]: W0805 18:04:49.401830    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9ccae2b4_040a_4a69_bc10_5d0c2609973a.slice/crio-cadcd748687a86ebb9b274603fad22ad247b684ba800ae35d61272885a6d10dd WatchSource:0}: Error finding container cadcd748687a86ebb9b274603fad22ad247b684ba800ae35d61272885a6d10dd: Status 404 returned error can't find the container with id cadcd748687a86ebb9b274603fad22ad247b684ba800ae35d61272885a6d10dd
Aug 05 18:04:49.404100 ip-10-0-135-2 kubenswrapper[2571]: I0805 18:04:49.404079    2571 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Aug 05 18:04:50.155066 ip-10-0-135-2 kubenswrapper[2571]: I0805 18:04:50.155027    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-bm58p/must-gather-lzkc9" event={"ID":"9ccae2b4-040a-4a69-bc10-5d0c2609973a","Type":"ContainerStarted","Data":"cadcd748687a86ebb9b274603fad22ad247b684ba800ae35d61272885a6d10dd"}
Aug 05 18:04:51.162315 ip-10-0-135-2 kubenswrapper[2571]: I0805 18:04:51.162277    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-bm58p/must-gather-lzkc9" event={"ID":"9ccae2b4-040a-4a69-bc10-5d0c2609973a","Type":"ContainerStarted","Data":"4698326860f44076d92c3fd68631b7b4f962e1320fa58a58cf5e473ab0a8539f"}
Aug 05 18:04:51.162825 ip-10-0-135-2 kubenswrapper[2571]: I0805 18:04:51.162331    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-bm58p/must-gather-lzkc9" event={"ID":"9ccae2b4-040a-4a69-bc10-5d0c2609973a","Type":"ContainerStarted","Data":"565a2c86020f6667893080c13598d98090563dd5d06bd14bf5223819e5918932"}
Aug 05 18:04:51.177416 ip-10-0-135-2 kubenswrapper[2571]: I0805 18:04:51.177343    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-bm58p/must-gather-lzkc9" podStartSLOduration=2.294992752 podStartE2EDuration="3.177313118s" totalImagesPullingTime="882.320366ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-05 18:04:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-05 18:04:51.175754135 +0000 UTC m=+3118.543120479" watchObservedRunningTime="2026-08-05 18:04:51.177313118 +0000 UTC m=+3118.544679464"
Aug 05 18:04:51.579330 ip-10-0-135-2 kubenswrapper[2571]: I0805 18:04:51.579297    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-9jxp4_f2ecc651-01ec-4fa1-a7c1-780b3a963996/global-pull-secret-syncer/0.log"
Aug 05 18:04:51.760179 ip-10-0-135-2 kubenswrapper[2571]: I0805 18:04:51.760144    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-zmr54_b0520f9c-efe2-43b9-8623-411b55bbc874/konnectivity-agent/0.log"
Aug 05 18:04:51.815269 ip-10-0-135-2 kubenswrapper[2571]: I0805 18:04:51.815234    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-135-2.ec2.internal_bb3fac1e9119c8f4580f4cf12eb2ffb1/haproxy/0.log"
Aug 05 18:04:55.911845 ip-10-0-135-2 kubenswrapper[2571]: I0805 18:04:55.911778    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_10e66bf1-f170-4f1b-a4ab-28ff387a260b/alertmanager/0.log"
Aug 05 18:04:55.942744 ip-10-0-135-2 kubenswrapper[2571]: I0805 18:04:55.942702    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_10e66bf1-f170-4f1b-a4ab-28ff387a260b/config-reloader/0.log"
Aug 05 18:04:55.969943 ip-10-0-135-2 kubenswrapper[2571]: I0805 18:04:55.969915    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_10e66bf1-f170-4f1b-a4ab-28ff387a260b/kube-rbac-proxy-web/0.log"
Aug 05 18:04:55.993186 ip-10-0-135-2 kubenswrapper[2571]: I0805 18:04:55.993150    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_10e66bf1-f170-4f1b-a4ab-28ff387a260b/kube-rbac-proxy/0.log"
Aug 05 18:04:56.015518 ip-10-0-135-2 kubenswrapper[2571]: I0805 18:04:56.015490    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_10e66bf1-f170-4f1b-a4ab-28ff387a260b/kube-rbac-proxy-metric/0.log"
Aug 05 18:04:56.038060 ip-10-0-135-2 kubenswrapper[2571]: I0805 18:04:56.038033    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_10e66bf1-f170-4f1b-a4ab-28ff387a260b/prom-label-proxy/0.log"
Aug 05 18:04:56.062927 ip-10-0-135-2 kubenswrapper[2571]: I0805 18:04:56.062904    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_10e66bf1-f170-4f1b-a4ab-28ff387a260b/init-config-reloader/0.log"
Aug 05 18:04:56.260123 ip-10-0-135-2 kubenswrapper[2571]: I0805 18:04:56.260091    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-6dffbbd67-7fzsj_a6075869-345c-4047-863a-ff43ba6ae645/monitoring-plugin/0.log"
Aug 05 18:04:56.376356 ip-10-0-135-2 kubenswrapper[2571]: I0805 18:04:56.376327    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-6jh6w_5de6dda3-4f32-4fbe-be23-c9fd0f0b9f8d/node-exporter/0.log"
Aug 05 18:04:56.402494 ip-10-0-135-2 kubenswrapper[2571]: I0805 18:04:56.402455    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-6jh6w_5de6dda3-4f32-4fbe-be23-c9fd0f0b9f8d/kube-rbac-proxy/0.log"
Aug 05 18:04:56.432020 ip-10-0-135-2 kubenswrapper[2571]: I0805 18:04:56.431982    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-6jh6w_5de6dda3-4f32-4fbe-be23-c9fd0f0b9f8d/init-textfile/0.log"
Aug 05 18:04:56.859937 ip-10-0-135-2 kubenswrapper[2571]: I0805 18:04:56.859824    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-9df4c78b4-8mg98_f72d410a-a6cf-4f12-8d00-917f1550c160/telemeter-client/0.log"
Aug 05 18:04:56.879583 ip-10-0-135-2 kubenswrapper[2571]: I0805 18:04:56.879533    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-9df4c78b4-8mg98_f72d410a-a6cf-4f12-8d00-917f1550c160/reload/0.log"
Aug 05 18:04:56.901655 ip-10-0-135-2 kubenswrapper[2571]: I0805 18:04:56.901621    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-9df4c78b4-8mg98_f72d410a-a6cf-4f12-8d00-917f1550c160/kube-rbac-proxy/0.log"
Aug 05 18:04:56.934092 ip-10-0-135-2 kubenswrapper[2571]: I0805 18:04:56.934066    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-756dfccb84-z56bd_6c1987e0-88ac-427b-95d5-d728bff74e89/thanos-query/0.log"
Aug 05 18:04:56.959432 ip-10-0-135-2 kubenswrapper[2571]: I0805 18:04:56.959391    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-756dfccb84-z56bd_6c1987e0-88ac-427b-95d5-d728bff74e89/kube-rbac-proxy-web/0.log"
Aug 05 18:04:56.982090 ip-10-0-135-2 kubenswrapper[2571]: I0805 18:04:56.981937    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-756dfccb84-z56bd_6c1987e0-88ac-427b-95d5-d728bff74e89/kube-rbac-proxy/0.log"
Aug 05 18:04:57.009560 ip-10-0-135-2 kubenswrapper[2571]: I0805 18:04:57.005457    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-756dfccb84-z56bd_6c1987e0-88ac-427b-95d5-d728bff74e89/prom-label-proxy/0.log"
Aug 05 18:04:57.031849 ip-10-0-135-2 kubenswrapper[2571]: I0805 18:04:57.031814    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-756dfccb84-z56bd_6c1987e0-88ac-427b-95d5-d728bff74e89/kube-rbac-proxy-rules/0.log"
Aug 05 18:04:57.059475 ip-10-0-135-2 kubenswrapper[2571]: I0805 18:04:57.059443    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-756dfccb84-z56bd_6c1987e0-88ac-427b-95d5-d728bff74e89/kube-rbac-proxy-metrics/0.log"
Aug 05 18:04:58.340066 ip-10-0-135-2 kubenswrapper[2571]: I0805 18:04:58.340005    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-console_networking-console-plugin-f77f48c6d-rd4jh_a0be5476-71e0-4242-878f-a37806d86403/networking-console-plugin/0.log"
Aug 05 18:04:59.229886 ip-10-0-135-2 kubenswrapper[2571]: I0805 18:04:59.229853    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-ffb9f44c4-xvvq9_90eef495-5e96-4ffa-ae69-d1a0a4ae6da6/console/0.log"
Aug 05 18:04:59.264619 ip-10-0-135-2 kubenswrapper[2571]: I0805 18:04:59.264574    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_downloads-77c974ffcf-6zshd_23b66e37-d633-4bed-a6dc-357d8bbb585e/download-server/0.log"
Aug 05 18:04:59.613967 ip-10-0-135-2 kubenswrapper[2571]: I0805 18:04:59.613888    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-bm58p/perf-node-gather-daemonset-5p8mt"]
Aug 05 18:04:59.620649 ip-10-0-135-2 kubenswrapper[2571]: I0805 18:04:59.620623    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-bm58p/perf-node-gather-daemonset-5p8mt"
Aug 05 18:04:59.629293 ip-10-0-135-2 kubenswrapper[2571]: I0805 18:04:59.629184    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-bm58p/perf-node-gather-daemonset-5p8mt"]
Aug 05 18:04:59.787502 ip-10-0-135-2 kubenswrapper[2571]: I0805 18:04:59.787463    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f8dc2e8e-0d1f-4b96-86b9-e70a46748503-sys\") pod \"perf-node-gather-daemonset-5p8mt\" (UID: \"f8dc2e8e-0d1f-4b96-86b9-e70a46748503\") " pod="openshift-must-gather-bm58p/perf-node-gather-daemonset-5p8mt"
Aug 05 18:04:59.787853 ip-10-0-135-2 kubenswrapper[2571]: I0805 18:04:59.787512    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/f8dc2e8e-0d1f-4b96-86b9-e70a46748503-proc\") pod \"perf-node-gather-daemonset-5p8mt\" (UID: \"f8dc2e8e-0d1f-4b96-86b9-e70a46748503\") " pod="openshift-must-gather-bm58p/perf-node-gather-daemonset-5p8mt"
Aug 05 18:04:59.787853 ip-10-0-135-2 kubenswrapper[2571]: I0805 18:04:59.787537    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/f8dc2e8e-0d1f-4b96-86b9-e70a46748503-lib-modules\") pod \"perf-node-gather-daemonset-5p8mt\" (UID: \"f8dc2e8e-0d1f-4b96-86b9-e70a46748503\") " pod="openshift-must-gather-bm58p/perf-node-gather-daemonset-5p8mt"
Aug 05 18:04:59.787853 ip-10-0-135-2 kubenswrapper[2571]: I0805 18:04:59.787579    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/f8dc2e8e-0d1f-4b96-86b9-e70a46748503-podres\") pod \"perf-node-gather-daemonset-5p8mt\" (UID: \"f8dc2e8e-0d1f-4b96-86b9-e70a46748503\") " pod="openshift-must-gather-bm58p/perf-node-gather-daemonset-5p8mt"
Aug 05 18:04:59.787853 ip-10-0-135-2 kubenswrapper[2571]: I0805 18:04:59.787696    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xb52r\" (UniqueName: \"kubernetes.io/projected/f8dc2e8e-0d1f-4b96-86b9-e70a46748503-kube-api-access-xb52r\") pod \"perf-node-gather-daemonset-5p8mt\" (UID: \"f8dc2e8e-0d1f-4b96-86b9-e70a46748503\") " pod="openshift-must-gather-bm58p/perf-node-gather-daemonset-5p8mt"
Aug 05 18:04:59.888846 ip-10-0-135-2 kubenswrapper[2571]: I0805 18:04:59.888750    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xb52r\" (UniqueName: \"kubernetes.io/projected/f8dc2e8e-0d1f-4b96-86b9-e70a46748503-kube-api-access-xb52r\") pod \"perf-node-gather-daemonset-5p8mt\" (UID: \"f8dc2e8e-0d1f-4b96-86b9-e70a46748503\") " pod="openshift-must-gather-bm58p/perf-node-gather-daemonset-5p8mt"
Aug 05 18:04:59.889017 ip-10-0-135-2 kubenswrapper[2571]: I0805 18:04:59.888874    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f8dc2e8e-0d1f-4b96-86b9-e70a46748503-sys\") pod \"perf-node-gather-daemonset-5p8mt\" (UID: \"f8dc2e8e-0d1f-4b96-86b9-e70a46748503\") " pod="openshift-must-gather-bm58p/perf-node-gather-daemonset-5p8mt"
Aug 05 18:04:59.889017 ip-10-0-135-2 kubenswrapper[2571]: I0805 18:04:59.888901    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/f8dc2e8e-0d1f-4b96-86b9-e70a46748503-proc\") pod \"perf-node-gather-daemonset-5p8mt\" (UID: \"f8dc2e8e-0d1f-4b96-86b9-e70a46748503\") " pod="openshift-must-gather-bm58p/perf-node-gather-daemonset-5p8mt"
Aug 05 18:04:59.889017 ip-10-0-135-2 kubenswrapper[2571]: I0805 18:04:59.888946    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/f8dc2e8e-0d1f-4b96-86b9-e70a46748503-lib-modules\") pod \"perf-node-gather-daemonset-5p8mt\" (UID: \"f8dc2e8e-0d1f-4b96-86b9-e70a46748503\") " pod="openshift-must-gather-bm58p/perf-node-gather-daemonset-5p8mt"
Aug 05 18:04:59.889017 ip-10-0-135-2 kubenswrapper[2571]: I0805 18:04:59.889010    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/f8dc2e8e-0d1f-4b96-86b9-e70a46748503-podres\") pod \"perf-node-gather-daemonset-5p8mt\" (UID: \"f8dc2e8e-0d1f-4b96-86b9-e70a46748503\") " pod="openshift-must-gather-bm58p/perf-node-gather-daemonset-5p8mt"
Aug 05 18:04:59.889239 ip-10-0-135-2 kubenswrapper[2571]: I0805 18:04:59.889194    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/f8dc2e8e-0d1f-4b96-86b9-e70a46748503-podres\") pod \"perf-node-gather-daemonset-5p8mt\" (UID: \"f8dc2e8e-0d1f-4b96-86b9-e70a46748503\") " pod="openshift-must-gather-bm58p/perf-node-gather-daemonset-5p8mt"
Aug 05 18:04:59.889297 ip-10-0-135-2 kubenswrapper[2571]: I0805 18:04:59.889240    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/f8dc2e8e-0d1f-4b96-86b9-e70a46748503-proc\") pod \"perf-node-gather-daemonset-5p8mt\" (UID: \"f8dc2e8e-0d1f-4b96-86b9-e70a46748503\") " pod="openshift-must-gather-bm58p/perf-node-gather-daemonset-5p8mt"
Aug 05 18:04:59.889364 ip-10-0-135-2 kubenswrapper[2571]: I0805 18:04:59.889310    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f8dc2e8e-0d1f-4b96-86b9-e70a46748503-sys\") pod \"perf-node-gather-daemonset-5p8mt\" (UID: \"f8dc2e8e-0d1f-4b96-86b9-e70a46748503\") " pod="openshift-must-gather-bm58p/perf-node-gather-daemonset-5p8mt"
Aug 05 18:04:59.889463 ip-10-0-135-2 kubenswrapper[2571]: I0805 18:04:59.889352    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/f8dc2e8e-0d1f-4b96-86b9-e70a46748503-lib-modules\") pod \"perf-node-gather-daemonset-5p8mt\" (UID: \"f8dc2e8e-0d1f-4b96-86b9-e70a46748503\") " pod="openshift-must-gather-bm58p/perf-node-gather-daemonset-5p8mt"
Aug 05 18:04:59.897206 ip-10-0-135-2 kubenswrapper[2571]: I0805 18:04:59.897179    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xb52r\" (UniqueName: \"kubernetes.io/projected/f8dc2e8e-0d1f-4b96-86b9-e70a46748503-kube-api-access-xb52r\") pod \"perf-node-gather-daemonset-5p8mt\" (UID: \"f8dc2e8e-0d1f-4b96-86b9-e70a46748503\") " pod="openshift-must-gather-bm58p/perf-node-gather-daemonset-5p8mt"
Aug 05 18:04:59.940558 ip-10-0-135-2 kubenswrapper[2571]: I0805 18:04:59.940531    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-bm58p/perf-node-gather-daemonset-5p8mt"
Aug 05 18:05:00.095566 ip-10-0-135-2 kubenswrapper[2571]: I0805 18:05:00.095514    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-bm58p/perf-node-gather-daemonset-5p8mt"]
Aug 05 18:05:00.215459 ip-10-0-135-2 kubenswrapper[2571]: I0805 18:05:00.215414    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-bm58p/perf-node-gather-daemonset-5p8mt" event={"ID":"f8dc2e8e-0d1f-4b96-86b9-e70a46748503","Type":"ContainerStarted","Data":"60b935e44e438270570ae4ff4ffb445a1763ed31b9f539abb53948517c1a4867"}
Aug 05 18:05:00.676829 ip-10-0-135-2 kubenswrapper[2571]: I0805 18:05:00.676804    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-gn486_972f76e8-cfb8-4568-81e4-a90a0e5171d8/dns/0.log"
Aug 05 18:05:00.695767 ip-10-0-135-2 kubenswrapper[2571]: I0805 18:05:00.695738    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-gn486_972f76e8-cfb8-4568-81e4-a90a0e5171d8/kube-rbac-proxy/0.log"
Aug 05 18:05:00.787362 ip-10-0-135-2 kubenswrapper[2571]: I0805 18:05:00.787331    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-qz2cd_c680deac-4ffe-40ff-b521-80beb96c5196/dns-node-resolver/0.log"
Aug 05 18:05:01.221266 ip-10-0-135-2 kubenswrapper[2571]: I0805 18:05:01.221230    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-bm58p/perf-node-gather-daemonset-5p8mt" event={"ID":"f8dc2e8e-0d1f-4b96-86b9-e70a46748503","Type":"ContainerStarted","Data":"b4bd721125eaffce3957041b46627e685f1f848d1f9269462fe65b6ba4ffad91"}
Aug 05 18:05:01.221460 ip-10-0-135-2 kubenswrapper[2571]: I0805 18:05:01.221366    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-bm58p/perf-node-gather-daemonset-5p8mt"
Aug 05 18:05:01.238532 ip-10-0-135-2 kubenswrapper[2571]: I0805 18:05:01.238480    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-bm58p/perf-node-gather-daemonset-5p8mt" podStartSLOduration=2.2384646 podStartE2EDuration="2.2384646s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-08-05 18:04:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-05 18:05:01.236676916 +0000 UTC m=+3128.604043258" watchObservedRunningTime="2026-08-05 18:05:01.2384646 +0000 UTC m=+3128.605830948"
Aug 05 18:05:01.292956 ip-10-0-135-2 kubenswrapper[2571]: I0805 18:05:01.292931    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-lrdmn_61599c74-5517-43f4-bed8-a232408b44c0/node-ca/0.log"
Aug 05 18:05:02.375640 ip-10-0-135-2 kubenswrapper[2571]: I0805 18:05:02.375613    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-phtkl_b59bced4-389d-494c-8376-d050e02710c4/serve-healthcheck-canary/0.log"
Aug 05 18:05:02.738661 ip-10-0-135-2 kubenswrapper[2571]: I0805 18:05:02.738637    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-5jjlh_fbccf6ae-01cd-4681-9f55-c702a5b85112/kube-rbac-proxy/0.log"
Aug 05 18:05:02.757052 ip-10-0-135-2 kubenswrapper[2571]: I0805 18:05:02.757029    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-5jjlh_fbccf6ae-01cd-4681-9f55-c702a5b85112/exporter/0.log"
Aug 05 18:05:02.779338 ip-10-0-135-2 kubenswrapper[2571]: I0805 18:05:02.779315    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-5jjlh_fbccf6ae-01cd-4681-9f55-c702a5b85112/extractor/0.log"
Aug 05 18:05:05.174908 ip-10-0-135-2 kubenswrapper[2571]: I0805 18:05:05.174874    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_odh-model-controller-8588677cfb-f2rz7_47655fb1-9898-4205-9753-a80dbc39a6ce/manager/0.log"
Aug 05 18:05:05.253726 ip-10-0-135-2 kubenswrapper[2571]: I0805 18:05:05.253702    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_seaweedfs-64568bcd49-7qwp7_851488c2-41ee-4d47-95da-5710eca8d9e7/seaweedfs/0.log"
Aug 05 18:05:07.238073 ip-10-0-135-2 kubenswrapper[2571]: I0805 18:05:07.238048    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-bm58p/perf-node-gather-daemonset-5p8mt"
Aug 05 18:05:10.295891 ip-10-0-135-2 kubenswrapper[2571]: I0805 18:05:10.295861    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-mgfr4_100f6b72-e895-453f-8647-1f615c2fb780/kube-multus-additional-cni-plugins/0.log"
Aug 05 18:05:10.320062 ip-10-0-135-2 kubenswrapper[2571]: I0805 18:05:10.320037    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-mgfr4_100f6b72-e895-453f-8647-1f615c2fb780/egress-router-binary-copy/0.log"
Aug 05 18:05:10.339196 ip-10-0-135-2 kubenswrapper[2571]: I0805 18:05:10.339173    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-mgfr4_100f6b72-e895-453f-8647-1f615c2fb780/cni-plugins/0.log"
Aug 05 18:05:10.358487 ip-10-0-135-2 kubenswrapper[2571]: I0805 18:05:10.358463    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-mgfr4_100f6b72-e895-453f-8647-1f615c2fb780/bond-cni-plugin/0.log"
Aug 05 18:05:10.377040 ip-10-0-135-2 kubenswrapper[2571]: I0805 18:05:10.377017    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-mgfr4_100f6b72-e895-453f-8647-1f615c2fb780/routeoverride-cni/0.log"
Aug 05 18:05:10.396526 ip-10-0-135-2 kubenswrapper[2571]: I0805 18:05:10.396505    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-mgfr4_100f6b72-e895-453f-8647-1f615c2fb780/whereabouts-cni-bincopy/0.log"
Aug 05 18:05:10.416608 ip-10-0-135-2 kubenswrapper[2571]: I0805 18:05:10.416589    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-mgfr4_100f6b72-e895-453f-8647-1f615c2fb780/whereabouts-cni/0.log"
Aug 05 18:05:10.795709 ip-10-0-135-2 kubenswrapper[2571]: I0805 18:05:10.795680    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-vzp52_99b228eb-7a25-47a1-a90d-06b62e8f17e0/kube-multus/0.log"
Aug 05 18:05:10.814944 ip-10-0-135-2 kubenswrapper[2571]: I0805 18:05:10.814916    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-677x7_da6f032b-35a1-4c51-b115-defaf9b2a9a5/network-metrics-daemon/0.log"
Aug 05 18:05:10.832775 ip-10-0-135-2 kubenswrapper[2571]: I0805 18:05:10.832749    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-677x7_da6f032b-35a1-4c51-b115-defaf9b2a9a5/kube-rbac-proxy/0.log"
Aug 05 18:05:11.983749 ip-10-0-135-2 kubenswrapper[2571]: I0805 18:05:11.983724    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-tsjgl_d662bf46-8fcd-40b5-94fa-eded9b062699/ovn-controller/0.log"
Aug 05 18:05:11.998878 ip-10-0-135-2 kubenswrapper[2571]: I0805 18:05:11.998850    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-tsjgl_d662bf46-8fcd-40b5-94fa-eded9b062699/ovn-acl-logging/0.log"
Aug 05 18:05:12.019697 ip-10-0-135-2 kubenswrapper[2571]: I0805 18:05:12.019667    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-tsjgl_d662bf46-8fcd-40b5-94fa-eded9b062699/kube-rbac-proxy-node/0.log"
Aug 05 18:05:12.041899 ip-10-0-135-2 kubenswrapper[2571]: I0805 18:05:12.041874    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-tsjgl_d662bf46-8fcd-40b5-94fa-eded9b062699/kube-rbac-proxy-ovn-metrics/0.log"
Aug 05 18:05:12.059162 ip-10-0-135-2 kubenswrapper[2571]: I0805 18:05:12.059141    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-tsjgl_d662bf46-8fcd-40b5-94fa-eded9b062699/northd/0.log"
Aug 05 18:05:12.078522 ip-10-0-135-2 kubenswrapper[2571]: I0805 18:05:12.078496    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-tsjgl_d662bf46-8fcd-40b5-94fa-eded9b062699/nbdb/0.log"
Aug 05 18:05:12.098304 ip-10-0-135-2 kubenswrapper[2571]: I0805 18:05:12.098284    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-tsjgl_d662bf46-8fcd-40b5-94fa-eded9b062699/sbdb/0.log"
Aug 05 18:05:12.140264 ip-10-0-135-2 kubenswrapper[2571]: I0805 18:05:12.140219    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-tsjgl_d662bf46-8fcd-40b5-94fa-eded9b062699/ovnkube-controller/0.log"
Aug 05 18:05:13.075061 ip-10-0-135-2 kubenswrapper[2571]: I0805 18:05:13.075027    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-diagnostics_network-check-target-whps7_a78df3c5-f92c-4dc6-a56c-a794929aa911/network-check-target-container/0.log"
Aug 05 18:05:13.981984 ip-10-0-135-2 kubenswrapper[2571]: I0805 18:05:13.981957    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-operator_iptables-alerter-txqn4_b934c28e-a416-4ddf-8b90-0c9acdd8ce4c/iptables-alerter/0.log"
Aug 05 18:05:14.611807 ip-10-0-135-2 kubenswrapper[2571]: I0805 18:05:14.611778    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-node-tuning-operator_tuned-nb82s_ac2f6cf0-11ca-47f3-8597-5cc1db578e78/tuned/0.log"