Jul 31 23:46:15.933537 ip-10-0-136-108 systemd[1]: Starting Kubernetes Kubelet...
Jul 31 23:46:16.250939 ip-10-0-136-108 kubenswrapper[2578]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 31 23:46:16.250939 ip-10-0-136-108 kubenswrapper[2578]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Jul 31 23:46:16.250939 ip-10-0-136-108 kubenswrapper[2578]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 31 23:46:16.250939 ip-10-0-136-108 kubenswrapper[2578]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Jul 31 23:46:16.250939 ip-10-0-136-108 kubenswrapper[2578]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 31 23:46:16.252659 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:16.252474    2578 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Jul 31 23:46:16.258984 ip-10-0-136-108 kubenswrapper[2578]: W0731 23:46:16.258954    2578 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 31 23:46:16.258984 ip-10-0-136-108 kubenswrapper[2578]: W0731 23:46:16.258980    2578 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 31 23:46:16.258984 ip-10-0-136-108 kubenswrapper[2578]: W0731 23:46:16.258986    2578 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 31 23:46:16.258984 ip-10-0-136-108 kubenswrapper[2578]: W0731 23:46:16.258990    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 31 23:46:16.259162 ip-10-0-136-108 kubenswrapper[2578]: W0731 23:46:16.258995    2578 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 31 23:46:16.259162 ip-10-0-136-108 kubenswrapper[2578]: W0731 23:46:16.259000    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 31 23:46:16.259162 ip-10-0-136-108 kubenswrapper[2578]: W0731 23:46:16.259005    2578 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 31 23:46:16.259162 ip-10-0-136-108 kubenswrapper[2578]: W0731 23:46:16.259010    2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 31 23:46:16.259162 ip-10-0-136-108 kubenswrapper[2578]: W0731 23:46:16.259015    2578 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 31 23:46:16.259162 ip-10-0-136-108 kubenswrapper[2578]: W0731 23:46:16.259023    2578 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 31 23:46:16.259162 ip-10-0-136-108 kubenswrapper[2578]: W0731 23:46:16.259030    2578 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 31 23:46:16.259162 ip-10-0-136-108 kubenswrapper[2578]: W0731 23:46:16.259034    2578 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 31 23:46:16.259162 ip-10-0-136-108 kubenswrapper[2578]: W0731 23:46:16.259038    2578 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 31 23:46:16.259162 ip-10-0-136-108 kubenswrapper[2578]: W0731 23:46:16.259042    2578 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 31 23:46:16.259162 ip-10-0-136-108 kubenswrapper[2578]: W0731 23:46:16.259047    2578 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 31 23:46:16.259162 ip-10-0-136-108 kubenswrapper[2578]: W0731 23:46:16.259051    2578 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 31 23:46:16.259162 ip-10-0-136-108 kubenswrapper[2578]: W0731 23:46:16.259055    2578 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 31 23:46:16.259162 ip-10-0-136-108 kubenswrapper[2578]: W0731 23:46:16.259060    2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 31 23:46:16.259162 ip-10-0-136-108 kubenswrapper[2578]: W0731 23:46:16.259065    2578 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 31 23:46:16.259162 ip-10-0-136-108 kubenswrapper[2578]: W0731 23:46:16.259069    2578 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 31 23:46:16.259162 ip-10-0-136-108 kubenswrapper[2578]: W0731 23:46:16.259074    2578 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 31 23:46:16.259162 ip-10-0-136-108 kubenswrapper[2578]: W0731 23:46:16.259078    2578 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 31 23:46:16.259162 ip-10-0-136-108 kubenswrapper[2578]: W0731 23:46:16.259083    2578 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 31 23:46:16.259162 ip-10-0-136-108 kubenswrapper[2578]: W0731 23:46:16.259087    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 31 23:46:16.259659 ip-10-0-136-108 kubenswrapper[2578]: W0731 23:46:16.259092    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 31 23:46:16.259659 ip-10-0-136-108 kubenswrapper[2578]: W0731 23:46:16.259096    2578 feature_gate.go:328] unrecognized feature gate: Example2
Jul 31 23:46:16.259659 ip-10-0-136-108 kubenswrapper[2578]: W0731 23:46:16.259101    2578 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 31 23:46:16.259659 ip-10-0-136-108 kubenswrapper[2578]: W0731 23:46:16.259105    2578 feature_gate.go:328] unrecognized feature gate: Example
Jul 31 23:46:16.259659 ip-10-0-136-108 kubenswrapper[2578]: W0731 23:46:16.259111    2578 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 31 23:46:16.259659 ip-10-0-136-108 kubenswrapper[2578]: W0731 23:46:16.259115    2578 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 31 23:46:16.259659 ip-10-0-136-108 kubenswrapper[2578]: W0731 23:46:16.259121    2578 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 31 23:46:16.259659 ip-10-0-136-108 kubenswrapper[2578]: W0731 23:46:16.259124    2578 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 31 23:46:16.259659 ip-10-0-136-108 kubenswrapper[2578]: W0731 23:46:16.259127    2578 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 31 23:46:16.259659 ip-10-0-136-108 kubenswrapper[2578]: W0731 23:46:16.259131    2578 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 31 23:46:16.259659 ip-10-0-136-108 kubenswrapper[2578]: W0731 23:46:16.259133    2578 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 31 23:46:16.259659 ip-10-0-136-108 kubenswrapper[2578]: W0731 23:46:16.259137    2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 31 23:46:16.259659 ip-10-0-136-108 kubenswrapper[2578]: W0731 23:46:16.259140    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 31 23:46:16.259659 ip-10-0-136-108 kubenswrapper[2578]: W0731 23:46:16.259143    2578 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 31 23:46:16.259659 ip-10-0-136-108 kubenswrapper[2578]: W0731 23:46:16.259146    2578 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 31 23:46:16.259659 ip-10-0-136-108 kubenswrapper[2578]: W0731 23:46:16.259149    2578 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 31 23:46:16.259659 ip-10-0-136-108 kubenswrapper[2578]: W0731 23:46:16.259151    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 31 23:46:16.259659 ip-10-0-136-108 kubenswrapper[2578]: W0731 23:46:16.259154    2578 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 31 23:46:16.259659 ip-10-0-136-108 kubenswrapper[2578]: W0731 23:46:16.259157    2578 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 31 23:46:16.259659 ip-10-0-136-108 kubenswrapper[2578]: W0731 23:46:16.259159    2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 31 23:46:16.260198 ip-10-0-136-108 kubenswrapper[2578]: W0731 23:46:16.259162    2578 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 31 23:46:16.260198 ip-10-0-136-108 kubenswrapper[2578]: W0731 23:46:16.259165    2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 31 23:46:16.260198 ip-10-0-136-108 kubenswrapper[2578]: W0731 23:46:16.259168    2578 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 31 23:46:16.260198 ip-10-0-136-108 kubenswrapper[2578]: W0731 23:46:16.259187    2578 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 31 23:46:16.260198 ip-10-0-136-108 kubenswrapper[2578]: W0731 23:46:16.259190    2578 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 31 23:46:16.260198 ip-10-0-136-108 kubenswrapper[2578]: W0731 23:46:16.259195    2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 31 23:46:16.260198 ip-10-0-136-108 kubenswrapper[2578]: W0731 23:46:16.259200    2578 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 31 23:46:16.260198 ip-10-0-136-108 kubenswrapper[2578]: W0731 23:46:16.259203    2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 31 23:46:16.260198 ip-10-0-136-108 kubenswrapper[2578]: W0731 23:46:16.259206    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 31 23:46:16.260198 ip-10-0-136-108 kubenswrapper[2578]: W0731 23:46:16.259210    2578 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 31 23:46:16.260198 ip-10-0-136-108 kubenswrapper[2578]: W0731 23:46:16.259214    2578 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 31 23:46:16.260198 ip-10-0-136-108 kubenswrapper[2578]: W0731 23:46:16.259217    2578 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 31 23:46:16.260198 ip-10-0-136-108 kubenswrapper[2578]: W0731 23:46:16.259220    2578 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 31 23:46:16.260198 ip-10-0-136-108 kubenswrapper[2578]: W0731 23:46:16.259223    2578 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 31 23:46:16.260198 ip-10-0-136-108 kubenswrapper[2578]: W0731 23:46:16.259225    2578 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 31 23:46:16.260198 ip-10-0-136-108 kubenswrapper[2578]: W0731 23:46:16.259228    2578 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 31 23:46:16.260198 ip-10-0-136-108 kubenswrapper[2578]: W0731 23:46:16.259231    2578 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 31 23:46:16.260198 ip-10-0-136-108 kubenswrapper[2578]: W0731 23:46:16.259233    2578 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 31 23:46:16.260198 ip-10-0-136-108 kubenswrapper[2578]: W0731 23:46:16.259236    2578 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 31 23:46:16.260677 ip-10-0-136-108 kubenswrapper[2578]: W0731 23:46:16.259238    2578 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 31 23:46:16.260677 ip-10-0-136-108 kubenswrapper[2578]: W0731 23:46:16.259241    2578 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 31 23:46:16.260677 ip-10-0-136-108 kubenswrapper[2578]: W0731 23:46:16.259246    2578 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 31 23:46:16.260677 ip-10-0-136-108 kubenswrapper[2578]: W0731 23:46:16.259250    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 31 23:46:16.260677 ip-10-0-136-108 kubenswrapper[2578]: W0731 23:46:16.259255    2578 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 31 23:46:16.260677 ip-10-0-136-108 kubenswrapper[2578]: W0731 23:46:16.259258    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 31 23:46:16.260677 ip-10-0-136-108 kubenswrapper[2578]: W0731 23:46:16.259261    2578 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 31 23:46:16.260677 ip-10-0-136-108 kubenswrapper[2578]: W0731 23:46:16.259264    2578 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 31 23:46:16.260677 ip-10-0-136-108 kubenswrapper[2578]: W0731 23:46:16.259267    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 31 23:46:16.260677 ip-10-0-136-108 kubenswrapper[2578]: W0731 23:46:16.259269    2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 31 23:46:16.260677 ip-10-0-136-108 kubenswrapper[2578]: W0731 23:46:16.259272    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 31 23:46:16.260677 ip-10-0-136-108 kubenswrapper[2578]: W0731 23:46:16.259275    2578 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 31 23:46:16.260677 ip-10-0-136-108 kubenswrapper[2578]: W0731 23:46:16.259278    2578 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 31 23:46:16.260677 ip-10-0-136-108 kubenswrapper[2578]: W0731 23:46:16.259281    2578 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 31 23:46:16.260677 ip-10-0-136-108 kubenswrapper[2578]: W0731 23:46:16.259284    2578 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 31 23:46:16.260677 ip-10-0-136-108 kubenswrapper[2578]: W0731 23:46:16.259287    2578 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 31 23:46:16.260677 ip-10-0-136-108 kubenswrapper[2578]: W0731 23:46:16.259291    2578 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 31 23:46:16.260677 ip-10-0-136-108 kubenswrapper[2578]: W0731 23:46:16.259294    2578 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 31 23:46:16.260677 ip-10-0-136-108 kubenswrapper[2578]: W0731 23:46:16.259296    2578 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 31 23:46:16.261140 ip-10-0-136-108 kubenswrapper[2578]: W0731 23:46:16.259299    2578 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 31 23:46:16.261140 ip-10-0-136-108 kubenswrapper[2578]: W0731 23:46:16.259303    2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 31 23:46:16.261140 ip-10-0-136-108 kubenswrapper[2578]: W0731 23:46:16.259305    2578 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 31 23:46:16.261140 ip-10-0-136-108 kubenswrapper[2578]: W0731 23:46:16.259308    2578 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 31 23:46:16.261140 ip-10-0-136-108 kubenswrapper[2578]: W0731 23:46:16.259310    2578 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 31 23:46:16.261140 ip-10-0-136-108 kubenswrapper[2578]: W0731 23:46:16.259313    2578 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 31 23:46:16.261140 ip-10-0-136-108 kubenswrapper[2578]: W0731 23:46:16.259315    2578 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 31 23:46:16.261140 ip-10-0-136-108 kubenswrapper[2578]: W0731 23:46:16.259318    2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 31 23:46:16.261140 ip-10-0-136-108 kubenswrapper[2578]: W0731 23:46:16.259320    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 31 23:46:16.261140 ip-10-0-136-108 kubenswrapper[2578]: W0731 23:46:16.259323    2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 31 23:46:16.261140 ip-10-0-136-108 kubenswrapper[2578]: W0731 23:46:16.259326    2578 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 31 23:46:16.261140 ip-10-0-136-108 kubenswrapper[2578]: W0731 23:46:16.259328    2578 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 31 23:46:16.261140 ip-10-0-136-108 kubenswrapper[2578]: W0731 23:46:16.259331    2578 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 31 23:46:16.261140 ip-10-0-136-108 kubenswrapper[2578]: W0731 23:46:16.259333    2578 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 31 23:46:16.261140 ip-10-0-136-108 kubenswrapper[2578]: W0731 23:46:16.259336    2578 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 31 23:46:16.261140 ip-10-0-136-108 kubenswrapper[2578]: W0731 23:46:16.259338    2578 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 31 23:46:16.261140 ip-10-0-136-108 kubenswrapper[2578]: W0731 23:46:16.259341    2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 31 23:46:16.261682 ip-10-0-136-108 kubenswrapper[2578]: W0731 23:46:16.261331    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 31 23:46:16.261682 ip-10-0-136-108 kubenswrapper[2578]: W0731 23:46:16.261341    2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 31 23:46:16.261682 ip-10-0-136-108 kubenswrapper[2578]: W0731 23:46:16.261345    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 31 23:46:16.261682 ip-10-0-136-108 kubenswrapper[2578]: W0731 23:46:16.261348    2578 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 31 23:46:16.261682 ip-10-0-136-108 kubenswrapper[2578]: W0731 23:46:16.261351    2578 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 31 23:46:16.261682 ip-10-0-136-108 kubenswrapper[2578]: W0731 23:46:16.261355    2578 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 31 23:46:16.261682 ip-10-0-136-108 kubenswrapper[2578]: W0731 23:46:16.261358    2578 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 31 23:46:16.261682 ip-10-0-136-108 kubenswrapper[2578]: W0731 23:46:16.261361    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 31 23:46:16.261682 ip-10-0-136-108 kubenswrapper[2578]: W0731 23:46:16.261363    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 31 23:46:16.261682 ip-10-0-136-108 kubenswrapper[2578]: W0731 23:46:16.261366    2578 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 31 23:46:16.261682 ip-10-0-136-108 kubenswrapper[2578]: W0731 23:46:16.261368    2578 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 31 23:46:16.261682 ip-10-0-136-108 kubenswrapper[2578]: W0731 23:46:16.261371    2578 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 31 23:46:16.261682 ip-10-0-136-108 kubenswrapper[2578]: W0731 23:46:16.261374    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 31 23:46:16.261682 ip-10-0-136-108 kubenswrapper[2578]: W0731 23:46:16.261377    2578 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 31 23:46:16.261682 ip-10-0-136-108 kubenswrapper[2578]: W0731 23:46:16.261379    2578 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 31 23:46:16.261682 ip-10-0-136-108 kubenswrapper[2578]: W0731 23:46:16.261382    2578 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 31 23:46:16.261682 ip-10-0-136-108 kubenswrapper[2578]: W0731 23:46:16.261385    2578 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 31 23:46:16.261682 ip-10-0-136-108 kubenswrapper[2578]: W0731 23:46:16.261389    2578 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 31 23:46:16.261682 ip-10-0-136-108 kubenswrapper[2578]: W0731 23:46:16.261391    2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 31 23:46:16.262147 ip-10-0-136-108 kubenswrapper[2578]: W0731 23:46:16.261396    2578 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 31 23:46:16.262147 ip-10-0-136-108 kubenswrapper[2578]: W0731 23:46:16.261399    2578 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 31 23:46:16.262147 ip-10-0-136-108 kubenswrapper[2578]: W0731 23:46:16.261402    2578 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 31 23:46:16.262147 ip-10-0-136-108 kubenswrapper[2578]: W0731 23:46:16.261404    2578 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 31 23:46:16.262147 ip-10-0-136-108 kubenswrapper[2578]: W0731 23:46:16.261407    2578 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 31 23:46:16.262147 ip-10-0-136-108 kubenswrapper[2578]: W0731 23:46:16.261409    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 31 23:46:16.262147 ip-10-0-136-108 kubenswrapper[2578]: W0731 23:46:16.261412    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 31 23:46:16.262147 ip-10-0-136-108 kubenswrapper[2578]: W0731 23:46:16.261415    2578 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 31 23:46:16.262147 ip-10-0-136-108 kubenswrapper[2578]: W0731 23:46:16.261417    2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 31 23:46:16.262147 ip-10-0-136-108 kubenswrapper[2578]: W0731 23:46:16.261420    2578 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 31 23:46:16.262147 ip-10-0-136-108 kubenswrapper[2578]: W0731 23:46:16.261423    2578 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 31 23:46:16.262147 ip-10-0-136-108 kubenswrapper[2578]: W0731 23:46:16.261425    2578 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 31 23:46:16.262147 ip-10-0-136-108 kubenswrapper[2578]: W0731 23:46:16.261428    2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 31 23:46:16.262147 ip-10-0-136-108 kubenswrapper[2578]: W0731 23:46:16.261431    2578 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 31 23:46:16.262147 ip-10-0-136-108 kubenswrapper[2578]: W0731 23:46:16.261434    2578 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 31 23:46:16.262147 ip-10-0-136-108 kubenswrapper[2578]: W0731 23:46:16.261437    2578 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 31 23:46:16.262147 ip-10-0-136-108 kubenswrapper[2578]: W0731 23:46:16.261441    2578 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 31 23:46:16.262147 ip-10-0-136-108 kubenswrapper[2578]: W0731 23:46:16.261443    2578 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 31 23:46:16.262147 ip-10-0-136-108 kubenswrapper[2578]: W0731 23:46:16.261446    2578 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 31 23:46:16.262634 ip-10-0-136-108 kubenswrapper[2578]: W0731 23:46:16.261449    2578 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 31 23:46:16.262634 ip-10-0-136-108 kubenswrapper[2578]: W0731 23:46:16.261451    2578 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 31 23:46:16.262634 ip-10-0-136-108 kubenswrapper[2578]: W0731 23:46:16.261454    2578 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 31 23:46:16.262634 ip-10-0-136-108 kubenswrapper[2578]: W0731 23:46:16.261457    2578 feature_gate.go:328] unrecognized feature gate: Example2
Jul 31 23:46:16.262634 ip-10-0-136-108 kubenswrapper[2578]: W0731 23:46:16.261460    2578 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 31 23:46:16.262634 ip-10-0-136-108 kubenswrapper[2578]: W0731 23:46:16.261462    2578 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 31 23:46:16.262634 ip-10-0-136-108 kubenswrapper[2578]: W0731 23:46:16.261465    2578 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 31 23:46:16.262634 ip-10-0-136-108 kubenswrapper[2578]: W0731 23:46:16.261468    2578 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 31 23:46:16.262634 ip-10-0-136-108 kubenswrapper[2578]: W0731 23:46:16.261470    2578 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 31 23:46:16.262634 ip-10-0-136-108 kubenswrapper[2578]: W0731 23:46:16.261473    2578 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 31 23:46:16.262634 ip-10-0-136-108 kubenswrapper[2578]: W0731 23:46:16.261475    2578 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 31 23:46:16.262634 ip-10-0-136-108 kubenswrapper[2578]: W0731 23:46:16.261478    2578 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 31 23:46:16.262634 ip-10-0-136-108 kubenswrapper[2578]: W0731 23:46:16.261481    2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 31 23:46:16.262634 ip-10-0-136-108 kubenswrapper[2578]: W0731 23:46:16.261484    2578 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 31 23:46:16.262634 ip-10-0-136-108 kubenswrapper[2578]: W0731 23:46:16.261487    2578 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 31 23:46:16.262634 ip-10-0-136-108 kubenswrapper[2578]: W0731 23:46:16.261490    2578 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 31 23:46:16.262634 ip-10-0-136-108 kubenswrapper[2578]: W0731 23:46:16.261494    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 31 23:46:16.262634 ip-10-0-136-108 kubenswrapper[2578]: W0731 23:46:16.261497    2578 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 31 23:46:16.262634 ip-10-0-136-108 kubenswrapper[2578]: W0731 23:46:16.261499    2578 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 31 23:46:16.263102 ip-10-0-136-108 kubenswrapper[2578]: W0731 23:46:16.261502    2578 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 31 23:46:16.263102 ip-10-0-136-108 kubenswrapper[2578]: W0731 23:46:16.261505    2578 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 31 23:46:16.263102 ip-10-0-136-108 kubenswrapper[2578]: W0731 23:46:16.261508    2578 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 31 23:46:16.263102 ip-10-0-136-108 kubenswrapper[2578]: W0731 23:46:16.261510    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 31 23:46:16.263102 ip-10-0-136-108 kubenswrapper[2578]: W0731 23:46:16.261513    2578 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 31 23:46:16.263102 ip-10-0-136-108 kubenswrapper[2578]: W0731 23:46:16.261515    2578 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 31 23:46:16.263102 ip-10-0-136-108 kubenswrapper[2578]: W0731 23:46:16.261519    2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 31 23:46:16.263102 ip-10-0-136-108 kubenswrapper[2578]: W0731 23:46:16.261522    2578 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 31 23:46:16.263102 ip-10-0-136-108 kubenswrapper[2578]: W0731 23:46:16.261526    2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 31 23:46:16.263102 ip-10-0-136-108 kubenswrapper[2578]: W0731 23:46:16.261529    2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 31 23:46:16.263102 ip-10-0-136-108 kubenswrapper[2578]: W0731 23:46:16.261531    2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 31 23:46:16.263102 ip-10-0-136-108 kubenswrapper[2578]: W0731 23:46:16.261535    2578 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 31 23:46:16.263102 ip-10-0-136-108 kubenswrapper[2578]: W0731 23:46:16.261540    2578 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 31 23:46:16.263102 ip-10-0-136-108 kubenswrapper[2578]: W0731 23:46:16.261544    2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 31 23:46:16.263102 ip-10-0-136-108 kubenswrapper[2578]: W0731 23:46:16.261548    2578 feature_gate.go:328] unrecognized feature gate: Example
Jul 31 23:46:16.263102 ip-10-0-136-108 kubenswrapper[2578]: W0731 23:46:16.261551    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 31 23:46:16.263102 ip-10-0-136-108 kubenswrapper[2578]: W0731 23:46:16.261554    2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 31 23:46:16.263102 ip-10-0-136-108 kubenswrapper[2578]: W0731 23:46:16.261556    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 31 23:46:16.263102 ip-10-0-136-108 kubenswrapper[2578]: W0731 23:46:16.261559    2578 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 31 23:46:16.263579 ip-10-0-136-108 kubenswrapper[2578]: W0731 23:46:16.261562    2578 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 31 23:46:16.263579 ip-10-0-136-108 kubenswrapper[2578]: W0731 23:46:16.261565    2578 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 31 23:46:16.263579 ip-10-0-136-108 kubenswrapper[2578]: W0731 23:46:16.261567    2578 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 31 23:46:16.263579 ip-10-0-136-108 kubenswrapper[2578]: W0731 23:46:16.261570    2578 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 31 23:46:16.263579 ip-10-0-136-108 kubenswrapper[2578]: W0731 23:46:16.261572    2578 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 31 23:46:16.263579 ip-10-0-136-108 kubenswrapper[2578]: W0731 23:46:16.261575    2578 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 31 23:46:16.263579 ip-10-0-136-108 kubenswrapper[2578]: W0731 23:46:16.261577    2578 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 31 23:46:16.263579 ip-10-0-136-108 kubenswrapper[2578]: W0731 23:46:16.261585    2578 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 31 23:46:16.263579 ip-10-0-136-108 kubenswrapper[2578]: W0731 23:46:16.261588    2578 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 31 23:46:16.263579 ip-10-0-136-108 kubenswrapper[2578]: W0731 23:46:16.261590    2578 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 31 23:46:16.263579 ip-10-0-136-108 kubenswrapper[2578]: W0731 23:46:16.261593    2578 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 31 23:46:16.263579 ip-10-0-136-108 kubenswrapper[2578]: W0731 23:46:16.261595    2578 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 31 23:46:16.263579 ip-10-0-136-108 kubenswrapper[2578]: W0731 23:46:16.261598    2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 31 23:46:16.263579 ip-10-0-136-108 kubenswrapper[2578]: W0731 23:46:16.261601    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 31 23:46:16.263579 ip-10-0-136-108 kubenswrapper[2578]: W0731 23:46:16.261603    2578 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 31 23:46:16.263579 ip-10-0-136-108 kubenswrapper[2578]: W0731 23:46:16.261606    2578 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 31 23:46:16.263579 ip-10-0-136-108 kubenswrapper[2578]: W0731 23:46:16.261608    2578 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 31 23:46:16.263579 ip-10-0-136-108 kubenswrapper[2578]: W0731 23:46:16.261611    2578 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 31 23:46:16.263579 ip-10-0-136-108 kubenswrapper[2578]: W0731 23:46:16.261613    2578 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 31 23:46:16.263579 ip-10-0-136-108 kubenswrapper[2578]: W0731 23:46:16.261616    2578 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 31 23:46:16.264100 ip-10-0-136-108 kubenswrapper[2578]: W0731 23:46:16.261619    2578 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 31 23:46:16.264100 ip-10-0-136-108 kubenswrapper[2578]: W0731 23:46:16.261622    2578 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 31 23:46:16.264100 ip-10-0-136-108 kubenswrapper[2578]: W0731 23:46:16.261625    2578 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 31 23:46:16.264100 ip-10-0-136-108 kubenswrapper[2578]: W0731 23:46:16.262038    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 31 23:46:16.264100 ip-10-0-136-108 kubenswrapper[2578]: W0731 23:46:16.262044    2578 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 31 23:46:16.264100 ip-10-0-136-108 kubenswrapper[2578]: W0731 23:46:16.262048    2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 31 23:46:16.264100 ip-10-0-136-108 kubenswrapper[2578]: W0731 23:46:16.262051    2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 31 23:46:16.264100 ip-10-0-136-108 kubenswrapper[2578]: W0731 23:46:16.262055    2578 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 31 23:46:16.264100 ip-10-0-136-108 kubenswrapper[2578]: W0731 23:46:16.262058    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 31 23:46:16.264100 ip-10-0-136-108 kubenswrapper[2578]: W0731 23:46:16.262061    2578 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 31 23:46:16.264100 ip-10-0-136-108 kubenswrapper[2578]: W0731 23:46:16.262064    2578 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 31 23:46:16.264100 ip-10-0-136-108 kubenswrapper[2578]: W0731 23:46:16.262066    2578 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 31 23:46:16.264100 ip-10-0-136-108 kubenswrapper[2578]: W0731 23:46:16.262069    2578 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 31 23:46:16.264100 ip-10-0-136-108 kubenswrapper[2578]: W0731 23:46:16.262072    2578 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 31 23:46:16.264100 ip-10-0-136-108 kubenswrapper[2578]: W0731 23:46:16.262074    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 31 23:46:16.264100 ip-10-0-136-108 kubenswrapper[2578]: W0731 23:46:16.262077    2578 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 31 23:46:16.264100 ip-10-0-136-108 kubenswrapper[2578]: W0731 23:46:16.262080    2578 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 31 23:46:16.264100 ip-10-0-136-108 kubenswrapper[2578]: W0731 23:46:16.262084    2578 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 31 23:46:16.264100 ip-10-0-136-108 kubenswrapper[2578]: W0731 23:46:16.262088    2578 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 31 23:46:16.264609 ip-10-0-136-108 kubenswrapper[2578]: W0731 23:46:16.262092    2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 31 23:46:16.264609 ip-10-0-136-108 kubenswrapper[2578]: W0731 23:46:16.262095    2578 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 31 23:46:16.264609 ip-10-0-136-108 kubenswrapper[2578]: W0731 23:46:16.262097    2578 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 31 23:46:16.264609 ip-10-0-136-108 kubenswrapper[2578]: W0731 23:46:16.262100    2578 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 31 23:46:16.264609 ip-10-0-136-108 kubenswrapper[2578]: W0731 23:46:16.262103    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 31 23:46:16.264609 ip-10-0-136-108 kubenswrapper[2578]: W0731 23:46:16.262105    2578 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 31 23:46:16.264609 ip-10-0-136-108 kubenswrapper[2578]: W0731 23:46:16.262108    2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 31 23:46:16.264609 ip-10-0-136-108 kubenswrapper[2578]: W0731 23:46:16.262111    2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 31 23:46:16.264609 ip-10-0-136-108 kubenswrapper[2578]: W0731 23:46:16.262113    2578 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 31 23:46:16.264609 ip-10-0-136-108 kubenswrapper[2578]: W0731 23:46:16.262116    2578 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 31 23:46:16.264609 ip-10-0-136-108 kubenswrapper[2578]: W0731 23:46:16.262118    2578 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 31 23:46:16.264609 ip-10-0-136-108 kubenswrapper[2578]: W0731 23:46:16.262121    2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 31 23:46:16.264609 ip-10-0-136-108 kubenswrapper[2578]: W0731 23:46:16.262124    2578 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 31 23:46:16.264609 ip-10-0-136-108 kubenswrapper[2578]: W0731 23:46:16.262127    2578 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 31 23:46:16.264609 ip-10-0-136-108 kubenswrapper[2578]: W0731 23:46:16.262130    2578 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 31 23:46:16.264609 ip-10-0-136-108 kubenswrapper[2578]: W0731 23:46:16.262132    2578 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 31 23:46:16.264609 ip-10-0-136-108 kubenswrapper[2578]: W0731 23:46:16.262135    2578 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 31 23:46:16.264609 ip-10-0-136-108 kubenswrapper[2578]: W0731 23:46:16.262137    2578 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 31 23:46:16.264609 ip-10-0-136-108 kubenswrapper[2578]: W0731 23:46:16.262140    2578 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 31 23:46:16.264609 ip-10-0-136-108 kubenswrapper[2578]: W0731 23:46:16.262142    2578 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 31 23:46:16.265094 ip-10-0-136-108 kubenswrapper[2578]: W0731 23:46:16.262145    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 31 23:46:16.265094 ip-10-0-136-108 kubenswrapper[2578]: W0731 23:46:16.262148    2578 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 31 23:46:16.265094 ip-10-0-136-108 kubenswrapper[2578]: W0731 23:46:16.262151    2578 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 31 23:46:16.265094 ip-10-0-136-108 kubenswrapper[2578]: W0731 23:46:16.262154    2578 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 31 23:46:16.265094 ip-10-0-136-108 kubenswrapper[2578]: W0731 23:46:16.262157    2578 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 31 23:46:16.265094 ip-10-0-136-108 kubenswrapper[2578]: W0731 23:46:16.262160    2578 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 31 23:46:16.265094 ip-10-0-136-108 kubenswrapper[2578]: W0731 23:46:16.262163    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 31 23:46:16.265094 ip-10-0-136-108 kubenswrapper[2578]: W0731 23:46:16.262167    2578 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 31 23:46:16.265094 ip-10-0-136-108 kubenswrapper[2578]: W0731 23:46:16.262183    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 31 23:46:16.265094 ip-10-0-136-108 kubenswrapper[2578]: W0731 23:46:16.262187    2578 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 31 23:46:16.265094 ip-10-0-136-108 kubenswrapper[2578]: W0731 23:46:16.262190    2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 31 23:46:16.265094 ip-10-0-136-108 kubenswrapper[2578]: W0731 23:46:16.262193    2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 31 23:46:16.265094 ip-10-0-136-108 kubenswrapper[2578]: W0731 23:46:16.262198    2578 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 31 23:46:16.265094 ip-10-0-136-108 kubenswrapper[2578]: W0731 23:46:16.262201    2578 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 31 23:46:16.265094 ip-10-0-136-108 kubenswrapper[2578]: W0731 23:46:16.262204    2578 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 31 23:46:16.265094 ip-10-0-136-108 kubenswrapper[2578]: W0731 23:46:16.262207    2578 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 31 23:46:16.265094 ip-10-0-136-108 kubenswrapper[2578]: W0731 23:46:16.262210    2578 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 31 23:46:16.265094 ip-10-0-136-108 kubenswrapper[2578]: W0731 23:46:16.262212    2578 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 31 23:46:16.265094 ip-10-0-136-108 kubenswrapper[2578]: W0731 23:46:16.262215    2578 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 31 23:46:16.265620 ip-10-0-136-108 kubenswrapper[2578]: W0731 23:46:16.262217    2578 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 31 23:46:16.265620 ip-10-0-136-108 kubenswrapper[2578]: W0731 23:46:16.262220    2578 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 31 23:46:16.265620 ip-10-0-136-108 kubenswrapper[2578]: W0731 23:46:16.262223    2578 feature_gate.go:328] unrecognized feature gate: Example
Jul 31 23:46:16.265620 ip-10-0-136-108 kubenswrapper[2578]: W0731 23:46:16.262226    2578 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 31 23:46:16.265620 ip-10-0-136-108 kubenswrapper[2578]: W0731 23:46:16.262228    2578 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 31 23:46:16.265620 ip-10-0-136-108 kubenswrapper[2578]: W0731 23:46:16.262231    2578 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 31 23:46:16.265620 ip-10-0-136-108 kubenswrapper[2578]: W0731 23:46:16.262234    2578 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 31 23:46:16.265620 ip-10-0-136-108 kubenswrapper[2578]: W0731 23:46:16.262237    2578 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 31 23:46:16.265620 ip-10-0-136-108 kubenswrapper[2578]: W0731 23:46:16.262240    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 31 23:46:16.265620 ip-10-0-136-108 kubenswrapper[2578]: W0731 23:46:16.262242    2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 31 23:46:16.265620 ip-10-0-136-108 kubenswrapper[2578]: W0731 23:46:16.262245    2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 31 23:46:16.265620 ip-10-0-136-108 kubenswrapper[2578]: W0731 23:46:16.262248    2578 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 31 23:46:16.265620 ip-10-0-136-108 kubenswrapper[2578]: W0731 23:46:16.262251    2578 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 31 23:46:16.265620 ip-10-0-136-108 kubenswrapper[2578]: W0731 23:46:16.262253    2578 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 31 23:46:16.265620 ip-10-0-136-108 kubenswrapper[2578]: W0731 23:46:16.262256    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 31 23:46:16.265620 ip-10-0-136-108 kubenswrapper[2578]: W0731 23:46:16.262259    2578 feature_gate.go:328] unrecognized feature gate: Example2
Jul 31 23:46:16.265620 ip-10-0-136-108 kubenswrapper[2578]: W0731 23:46:16.262261    2578 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 31 23:46:16.265620 ip-10-0-136-108 kubenswrapper[2578]: W0731 23:46:16.262264    2578 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 31 23:46:16.265620 ip-10-0-136-108 kubenswrapper[2578]: W0731 23:46:16.262267    2578 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 31 23:46:16.265620 ip-10-0-136-108 kubenswrapper[2578]: W0731 23:46:16.262269    2578 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 31 23:46:16.265620 ip-10-0-136-108 kubenswrapper[2578]: W0731 23:46:16.262272    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 31 23:46:16.266137 ip-10-0-136-108 kubenswrapper[2578]: W0731 23:46:16.262274    2578 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 31 23:46:16.266137 ip-10-0-136-108 kubenswrapper[2578]: W0731 23:46:16.262277    2578 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 31 23:46:16.266137 ip-10-0-136-108 kubenswrapper[2578]: W0731 23:46:16.262279    2578 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 31 23:46:16.266137 ip-10-0-136-108 kubenswrapper[2578]: W0731 23:46:16.262282    2578 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 31 23:46:16.266137 ip-10-0-136-108 kubenswrapper[2578]: W0731 23:46:16.262284    2578 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 31 23:46:16.266137 ip-10-0-136-108 kubenswrapper[2578]: W0731 23:46:16.262287    2578 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 31 23:46:16.266137 ip-10-0-136-108 kubenswrapper[2578]: W0731 23:46:16.262290    2578 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 31 23:46:16.266137 ip-10-0-136-108 kubenswrapper[2578]: W0731 23:46:16.262293    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 31 23:46:16.266137 ip-10-0-136-108 kubenswrapper[2578]: W0731 23:46:16.262295    2578 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 31 23:46:16.266137 ip-10-0-136-108 kubenswrapper[2578]: W0731 23:46:16.262298    2578 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 31 23:46:16.266137 ip-10-0-136-108 kubenswrapper[2578]: W0731 23:46:16.262300    2578 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 31 23:46:16.266137 ip-10-0-136-108 kubenswrapper[2578]: W0731 23:46:16.262304    2578 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 31 23:46:16.266137 ip-10-0-136-108 kubenswrapper[2578]: W0731 23:46:16.262307    2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 31 23:46:16.266137 ip-10-0-136-108 kubenswrapper[2578]: W0731 23:46:16.262310    2578 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 31 23:46:16.266137 ip-10-0-136-108 kubenswrapper[2578]: W0731 23:46:16.262313    2578 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 31 23:46:16.266137 ip-10-0-136-108 kubenswrapper[2578]: W0731 23:46:16.262315    2578 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 31 23:46:16.266137 ip-10-0-136-108 kubenswrapper[2578]: W0731 23:46:16.262318    2578 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 31 23:46:16.266137 ip-10-0-136-108 kubenswrapper[2578]: W0731 23:46:16.262321    2578 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 31 23:46:16.266137 ip-10-0-136-108 kubenswrapper[2578]: W0731 23:46:16.262324    2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 31 23:46:16.266616 ip-10-0-136-108 kubenswrapper[2578]: W0731 23:46:16.262327    2578 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 31 23:46:16.266616 ip-10-0-136-108 kubenswrapper[2578]: W0731 23:46:16.262329    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 31 23:46:16.266616 ip-10-0-136-108 kubenswrapper[2578]: W0731 23:46:16.262332    2578 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 31 23:46:16.266616 ip-10-0-136-108 kubenswrapper[2578]: W0731 23:46:16.262334    2578 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 31 23:46:16.266616 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:16.262409    2578 flags.go:64] FLAG: --address="0.0.0.0"
Jul 31 23:46:16.266616 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:16.262422    2578 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Jul 31 23:46:16.266616 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:16.262433    2578 flags.go:64] FLAG: --anonymous-auth="true"
Jul 31 23:46:16.266616 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:16.262439    2578 flags.go:64] FLAG: --application-metrics-count-limit="100"
Jul 31 23:46:16.266616 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:16.262446    2578 flags.go:64] FLAG: --authentication-token-webhook="false"
Jul 31 23:46:16.266616 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:16.262451    2578 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Jul 31 23:46:16.266616 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:16.262458    2578 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Jul 31 23:46:16.266616 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:16.262462    2578 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Jul 31 23:46:16.266616 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:16.262466    2578 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Jul 31 23:46:16.266616 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:16.262469    2578 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Jul 31 23:46:16.266616 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:16.262472    2578 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Jul 31 23:46:16.266616 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:16.262475    2578 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Jul 31 23:46:16.266616 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:16.262478    2578 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Jul 31 23:46:16.266616 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:16.262481    2578 flags.go:64] FLAG: --cgroup-root=""
Jul 31 23:46:16.266616 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:16.262484    2578 flags.go:64] FLAG: --cgroups-per-qos="true"
Jul 31 23:46:16.266616 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:16.262487    2578 flags.go:64] FLAG: --client-ca-file=""
Jul 31 23:46:16.266616 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:16.262491    2578 flags.go:64] FLAG: --cloud-provider="external"
Jul 31 23:46:16.266616 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:16.262494    2578 flags.go:64] FLAG: --cluster-dns="[]"
Jul 31 23:46:16.266616 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:16.262498    2578 flags.go:64] FLAG: --cluster-domain=""
Jul 31 23:46:16.266616 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:16.262501    2578 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Jul 31 23:46:16.267280 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:16.262504    2578 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Jul 31 23:46:16.267280 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:16.262507    2578 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Jul 31 23:46:16.267280 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:16.262510    2578 flags.go:64] FLAG: --container-log-max-files="5"
Jul 31 23:46:16.267280 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:16.262521    2578 flags.go:64] FLAG: --container-log-max-size="10Mi"
Jul 31 23:46:16.267280 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:16.262524    2578 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Jul 31 23:46:16.267280 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:16.262527    2578 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Jul 31 23:46:16.267280 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:16.262530    2578 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Jul 31 23:46:16.267280 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:16.262534    2578 flags.go:64] FLAG: --contention-profiling="false"
Jul 31 23:46:16.267280 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:16.262536    2578 flags.go:64] FLAG: --cpu-cfs-quota="true"
Jul 31 23:46:16.267280 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:16.262539    2578 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Jul 31 23:46:16.267280 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:16.262543    2578 flags.go:64] FLAG: --cpu-manager-policy="none"
Jul 31 23:46:16.267280 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:16.262545    2578 flags.go:64] FLAG: --cpu-manager-policy-options=""
Jul 31 23:46:16.267280 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:16.262550    2578 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Jul 31 23:46:16.267280 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:16.262553    2578 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Jul 31 23:46:16.267280 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:16.262556    2578 flags.go:64] FLAG: --enable-debugging-handlers="true"
Jul 31 23:46:16.267280 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:16.262559    2578 flags.go:64] FLAG: --enable-load-reader="false"
Jul 31 23:46:16.267280 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:16.262562    2578 flags.go:64] FLAG: --enable-server="true"
Jul 31 23:46:16.267280 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:16.262565    2578 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Jul 31 23:46:16.267280 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:16.262570    2578 flags.go:64] FLAG: --event-burst="100"
Jul 31 23:46:16.267280 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:16.262573    2578 flags.go:64] FLAG: --event-qps="50"
Jul 31 23:46:16.267280 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:16.262577    2578 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Jul 31 23:46:16.267280 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:16.262580    2578 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Jul 31 23:46:16.267280 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:16.262583    2578 flags.go:64] FLAG: --eviction-hard=""
Jul 31 23:46:16.267280 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:16.262588    2578 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Jul 31 23:46:16.267874 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:16.262590    2578 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Jul 31 23:46:16.267874 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:16.262593    2578 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Jul 31 23:46:16.267874 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:16.262596    2578 flags.go:64] FLAG: --eviction-soft=""
Jul 31 23:46:16.267874 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:16.262599    2578 flags.go:64] FLAG: --eviction-soft-grace-period=""
Jul 31 23:46:16.267874 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:16.262602    2578 flags.go:64] FLAG: --exit-on-lock-contention="false"
Jul 31 23:46:16.267874 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:16.262606    2578 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Jul 31 23:46:16.267874 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:16.262609    2578 flags.go:64] FLAG: --experimental-mounter-path=""
Jul 31 23:46:16.267874 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:16.262612    2578 flags.go:64] FLAG: --fail-cgroupv1="false"
Jul 31 23:46:16.267874 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:16.262615    2578 flags.go:64] FLAG: --fail-swap-on="true"
Jul 31 23:46:16.267874 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:16.262618    2578 flags.go:64] FLAG: --feature-gates=""
Jul 31 23:46:16.267874 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:16.262622    2578 flags.go:64] FLAG: --file-check-frequency="20s"
Jul 31 23:46:16.267874 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:16.262627    2578 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Jul 31 23:46:16.267874 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:16.262630    2578 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Jul 31 23:46:16.267874 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:16.262634    2578 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Jul 31 23:46:16.267874 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:16.262637    2578 flags.go:64] FLAG: --healthz-port="10248"
Jul 31 23:46:16.267874 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:16.262641    2578 flags.go:64] FLAG: --help="false"
Jul 31 23:46:16.267874 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:16.262644    2578 flags.go:64] FLAG: --hostname-override="ip-10-0-136-108.ec2.internal"
Jul 31 23:46:16.267874 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:16.262647    2578 flags.go:64] FLAG: --housekeeping-interval="10s"
Jul 31 23:46:16.267874 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:16.262650    2578 flags.go:64] FLAG: --http-check-frequency="20s"
Jul 31 23:46:16.267874 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:16.262653    2578 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Jul 31 23:46:16.267874 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:16.262656    2578 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Jul 31 23:46:16.267874 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:16.262660    2578 flags.go:64] FLAG: --image-gc-high-threshold="85"
Jul 31 23:46:16.267874 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:16.262663    2578 flags.go:64] FLAG: --image-gc-low-threshold="80"
Jul 31 23:46:16.267874 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:16.262666    2578 flags.go:64] FLAG: --image-service-endpoint=""
Jul 31 23:46:16.268529 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:16.262669    2578 flags.go:64] FLAG: --kernel-memcg-notification="false"
Jul 31 23:46:16.268529 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:16.262672    2578 flags.go:64] FLAG: --kube-api-burst="100"
Jul 31 23:46:16.268529 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:16.262675    2578 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Jul 31 23:46:16.268529 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:16.262678    2578 flags.go:64] FLAG: --kube-api-qps="50"
Jul 31 23:46:16.268529 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:16.262681    2578 flags.go:64] FLAG: --kube-reserved=""
Jul 31 23:46:16.268529 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:16.262684    2578 flags.go:64] FLAG: --kube-reserved-cgroup=""
Jul 31 23:46:16.268529 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:16.262687    2578 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Jul 31 23:46:16.268529 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:16.262690    2578 flags.go:64] FLAG: --kubelet-cgroups=""
Jul 31 23:46:16.268529 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:16.262692    2578 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Jul 31 23:46:16.268529 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:16.262695    2578 flags.go:64] FLAG: --lock-file=""
Jul 31 23:46:16.268529 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:16.262698    2578 flags.go:64] FLAG: --log-cadvisor-usage="false"
Jul 31 23:46:16.268529 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:16.262701    2578 flags.go:64] FLAG: --log-flush-frequency="5s"
Jul 31 23:46:16.268529 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:16.262704    2578 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Jul 31 23:46:16.268529 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:16.262709    2578 flags.go:64] FLAG: --log-json-split-stream="false"
Jul 31 23:46:16.268529 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:16.262713    2578 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Jul 31 23:46:16.268529 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:16.262716    2578 flags.go:64] FLAG: --log-text-split-stream="false"
Jul 31 23:46:16.268529 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:16.262719    2578 flags.go:64] FLAG: --logging-format="text"
Jul 31 23:46:16.268529 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:16.262721    2578 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Jul 31 23:46:16.268529 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:16.262725    2578 flags.go:64] FLAG: --make-iptables-util-chains="true"
Jul 31 23:46:16.268529 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:16.262729    2578 flags.go:64] FLAG: --manifest-url=""
Jul 31 23:46:16.268529 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:16.262732    2578 flags.go:64] FLAG: --manifest-url-header=""
Jul 31 23:46:16.268529 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:16.262737    2578 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Jul 31 23:46:16.268529 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:16.262740    2578 flags.go:64] FLAG: --max-open-files="1000000"
Jul 31 23:46:16.268529 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:16.262744    2578 flags.go:64] FLAG: --max-pods="110"
Jul 31 23:46:16.268529 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:16.262747    2578 flags.go:64] FLAG: --maximum-dead-containers="-1"
Jul 31 23:46:16.269165 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:16.262750    2578 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Jul 31 23:46:16.269165 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:16.262753    2578 flags.go:64] FLAG: --memory-manager-policy="None"
Jul 31 23:46:16.269165 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:16.262756    2578 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Jul 31 23:46:16.269165 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:16.262759    2578 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Jul 31 23:46:16.269165 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:16.262762    2578 flags.go:64] FLAG: --node-ip="0.0.0.0"
Jul 31 23:46:16.269165 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:16.262766    2578 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Jul 31 23:46:16.269165 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:16.262774    2578 flags.go:64] FLAG: --node-status-max-images="50"
Jul 31 23:46:16.269165 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:16.262777    2578 flags.go:64] FLAG: --node-status-update-frequency="10s"
Jul 31 23:46:16.269165 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:16.262780    2578 flags.go:64] FLAG: --oom-score-adj="-999"
Jul 31 23:46:16.269165 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:16.262783    2578 flags.go:64] FLAG: --pod-cidr=""
Jul 31 23:46:16.269165 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:16.262786    2578 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:267c8c33005f5dc3d4121af550b1c396e30e2928a13624d7362adfbfb7f440f8"
Jul 31 23:46:16.269165 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:16.262791    2578 flags.go:64] FLAG: --pod-manifest-path=""
Jul 31 23:46:16.269165 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:16.262794    2578 flags.go:64] FLAG: --pod-max-pids="-1"
Jul 31 23:46:16.269165 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:16.262797    2578 flags.go:64] FLAG: --pods-per-core="0"
Jul 31 23:46:16.269165 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:16.262800    2578 flags.go:64] FLAG: --port="10250"
Jul 31 23:46:16.269165 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:16.262804    2578 flags.go:64] FLAG: --protect-kernel-defaults="false"
Jul 31 23:46:16.269165 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:16.262807    2578 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0f19492d0232515f3"
Jul 31 23:46:16.269165 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:16.262810    2578 flags.go:64] FLAG: --qos-reserved=""
Jul 31 23:46:16.269165 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:16.262813    2578 flags.go:64] FLAG: --read-only-port="10255"
Jul 31 23:46:16.269165 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:16.262816    2578 flags.go:64] FLAG: --register-node="true"
Jul 31 23:46:16.269165 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:16.262819    2578 flags.go:64] FLAG: --register-with-taints=""
Jul 31 23:46:16.269165 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:16.262827    2578 flags.go:64] FLAG: --registry-burst="10"
Jul 31 23:46:16.269165 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:16.262830    2578 flags.go:64] FLAG: --registry-qps="5"
Jul 31 23:46:16.269165 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:16.262833    2578 flags.go:64] FLAG: --reserved-cpus=""
Jul 31 23:46:16.269843 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:16.262836    2578 flags.go:64] FLAG: --reserved-memory=""
Jul 31 23:46:16.269843 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:16.262840    2578 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Jul 31 23:46:16.269843 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:16.262843    2578 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Jul 31 23:46:16.269843 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:16.262848    2578 flags.go:64] FLAG: --rotate-certificates="false"
Jul 31 23:46:16.269843 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:16.262853    2578 flags.go:64] FLAG: --rotate-server-certificates="false"
Jul 31 23:46:16.269843 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:16.262856    2578 flags.go:64] FLAG: --runonce="false"
Jul 31 23:46:16.269843 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:16.262859    2578 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Jul 31 23:46:16.269843 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:16.262862    2578 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Jul 31 23:46:16.269843 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:16.262865    2578 flags.go:64] FLAG: --seccomp-default="false"
Jul 31 23:46:16.269843 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:16.262868    2578 flags.go:64] FLAG: --serialize-image-pulls="true"
Jul 31 23:46:16.269843 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:16.262871    2578 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Jul 31 23:46:16.269843 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:16.262874    2578 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Jul 31 23:46:16.269843 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:16.262877    2578 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Jul 31 23:46:16.269843 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:16.262880    2578 flags.go:64] FLAG: --storage-driver-password="root"
Jul 31 23:46:16.269843 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:16.262884    2578 flags.go:64] FLAG: --storage-driver-secure="false"
Jul 31 23:46:16.269843 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:16.262887    2578 flags.go:64] FLAG: --storage-driver-table="stats"
Jul 31 23:46:16.269843 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:16.262889    2578 flags.go:64] FLAG: --storage-driver-user="root"
Jul 31 23:46:16.269843 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:16.262892    2578 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Jul 31 23:46:16.269843 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:16.262895    2578 flags.go:64] FLAG: --sync-frequency="1m0s"
Jul 31 23:46:16.269843 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:16.262898    2578 flags.go:64] FLAG: --system-cgroups=""
Jul 31 23:46:16.269843 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:16.262901    2578 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Jul 31 23:46:16.269843 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:16.262906    2578 flags.go:64] FLAG: --system-reserved-cgroup=""
Jul 31 23:46:16.269843 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:16.262909    2578 flags.go:64] FLAG: --tls-cert-file=""
Jul 31 23:46:16.269843 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:16.262912    2578 flags.go:64] FLAG: --tls-cipher-suites="[]"
Jul 31 23:46:16.269843 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:16.262916    2578 flags.go:64] FLAG: --tls-min-version=""
Jul 31 23:46:16.270518 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:16.262919    2578 flags.go:64] FLAG: --tls-private-key-file=""
Jul 31 23:46:16.270518 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:16.262922    2578 flags.go:64] FLAG: --topology-manager-policy="none"
Jul 31 23:46:16.270518 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:16.262925    2578 flags.go:64] FLAG: --topology-manager-policy-options=""
Jul 31 23:46:16.270518 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:16.262928    2578 flags.go:64] FLAG: --topology-manager-scope="container"
Jul 31 23:46:16.270518 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:16.262931    2578 flags.go:64] FLAG: --v="2"
Jul 31 23:46:16.270518 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:16.262935    2578 flags.go:64] FLAG: --version="false"
Jul 31 23:46:16.270518 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:16.262940    2578 flags.go:64] FLAG: --vmodule=""
Jul 31 23:46:16.270518 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:16.262944    2578 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Jul 31 23:46:16.270518 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:16.262947    2578 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Jul 31 23:46:16.270518 ip-10-0-136-108 kubenswrapper[2578]: W0731 23:46:16.263065    2578 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 31 23:46:16.270518 ip-10-0-136-108 kubenswrapper[2578]: W0731 23:46:16.263070    2578 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 31 23:46:16.270518 ip-10-0-136-108 kubenswrapper[2578]: W0731 23:46:16.263075    2578 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 31 23:46:16.270518 ip-10-0-136-108 kubenswrapper[2578]: W0731 23:46:16.263078    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 31 23:46:16.270518 ip-10-0-136-108 kubenswrapper[2578]: W0731 23:46:16.263080    2578 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 31 23:46:16.270518 ip-10-0-136-108 kubenswrapper[2578]: W0731 23:46:16.263083    2578 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 31 23:46:16.270518 ip-10-0-136-108 kubenswrapper[2578]: W0731 23:46:16.263087    2578 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 31 23:46:16.270518 ip-10-0-136-108 kubenswrapper[2578]: W0731 23:46:16.263090    2578 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 31 23:46:16.270518 ip-10-0-136-108 kubenswrapper[2578]: W0731 23:46:16.263093    2578 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 31 23:46:16.270518 ip-10-0-136-108 kubenswrapper[2578]: W0731 23:46:16.263096    2578 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 31 23:46:16.270518 ip-10-0-136-108 kubenswrapper[2578]: W0731 23:46:16.263099    2578 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 31 23:46:16.270518 ip-10-0-136-108 kubenswrapper[2578]: W0731 23:46:16.263102    2578 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 31 23:46:16.270518 ip-10-0-136-108 kubenswrapper[2578]: W0731 23:46:16.263105    2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 31 23:46:16.271110 ip-10-0-136-108 kubenswrapper[2578]: W0731 23:46:16.263108    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 31 23:46:16.271110 ip-10-0-136-108 kubenswrapper[2578]: W0731 23:46:16.263110    2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 31 23:46:16.271110 ip-10-0-136-108 kubenswrapper[2578]: W0731 23:46:16.263113    2578 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 31 23:46:16.271110 ip-10-0-136-108 kubenswrapper[2578]: W0731 23:46:16.263116    2578 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 31 23:46:16.271110 ip-10-0-136-108 kubenswrapper[2578]: W0731 23:46:16.263119    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 31 23:46:16.271110 ip-10-0-136-108 kubenswrapper[2578]: W0731 23:46:16.263123    2578 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 31 23:46:16.271110 ip-10-0-136-108 kubenswrapper[2578]: W0731 23:46:16.263127    2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 31 23:46:16.271110 ip-10-0-136-108 kubenswrapper[2578]: W0731 23:46:16.263130    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 31 23:46:16.271110 ip-10-0-136-108 kubenswrapper[2578]: W0731 23:46:16.263133    2578 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 31 23:46:16.271110 ip-10-0-136-108 kubenswrapper[2578]: W0731 23:46:16.263135    2578 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 31 23:46:16.271110 ip-10-0-136-108 kubenswrapper[2578]: W0731 23:46:16.263138    2578 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 31 23:46:16.271110 ip-10-0-136-108 kubenswrapper[2578]: W0731 23:46:16.263140    2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 31 23:46:16.271110 ip-10-0-136-108 kubenswrapper[2578]: W0731 23:46:16.263143    2578 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 31 23:46:16.271110 ip-10-0-136-108 kubenswrapper[2578]: W0731 23:46:16.263146    2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 31 23:46:16.271110 ip-10-0-136-108 kubenswrapper[2578]: W0731 23:46:16.263149    2578 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 31 23:46:16.271110 ip-10-0-136-108 kubenswrapper[2578]: W0731 23:46:16.263152    2578 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 31 23:46:16.271110 ip-10-0-136-108 kubenswrapper[2578]: W0731 23:46:16.263155    2578 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 31 23:46:16.271110 ip-10-0-136-108 kubenswrapper[2578]: W0731 23:46:16.263158    2578 feature_gate.go:328] unrecognized feature gate: Example
Jul 31 23:46:16.271110 ip-10-0-136-108 kubenswrapper[2578]: W0731 23:46:16.263160    2578 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 31 23:46:16.271602 ip-10-0-136-108 kubenswrapper[2578]: W0731 23:46:16.263163    2578 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 31 23:46:16.271602 ip-10-0-136-108 kubenswrapper[2578]: W0731 23:46:16.263166    2578 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 31 23:46:16.271602 ip-10-0-136-108 kubenswrapper[2578]: W0731 23:46:16.263182    2578 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 31 23:46:16.271602 ip-10-0-136-108 kubenswrapper[2578]: W0731 23:46:16.263185    2578 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 31 23:46:16.271602 ip-10-0-136-108 kubenswrapper[2578]: W0731 23:46:16.263188    2578 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 31 23:46:16.271602 ip-10-0-136-108 kubenswrapper[2578]: W0731 23:46:16.263190    2578 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 31 23:46:16.271602 ip-10-0-136-108 kubenswrapper[2578]: W0731 23:46:16.263193    2578 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 31 23:46:16.271602 ip-10-0-136-108 kubenswrapper[2578]: W0731 23:46:16.263195    2578 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 31 23:46:16.271602 ip-10-0-136-108 kubenswrapper[2578]: W0731 23:46:16.263198    2578 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 31 23:46:16.271602 ip-10-0-136-108 kubenswrapper[2578]: W0731 23:46:16.263201    2578 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 31 23:46:16.271602 ip-10-0-136-108 kubenswrapper[2578]: W0731 23:46:16.263203    2578 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 31 23:46:16.271602 ip-10-0-136-108 kubenswrapper[2578]: W0731 23:46:16.263206    2578 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 31 23:46:16.271602 ip-10-0-136-108 kubenswrapper[2578]: W0731 23:46:16.263208    2578 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 31 23:46:16.271602 ip-10-0-136-108 kubenswrapper[2578]: W0731 23:46:16.263211    2578 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 31 23:46:16.271602 ip-10-0-136-108 kubenswrapper[2578]: W0731 23:46:16.263213    2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 31 23:46:16.271602 ip-10-0-136-108 kubenswrapper[2578]: W0731 23:46:16.263216    2578 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 31 23:46:16.271602 ip-10-0-136-108 kubenswrapper[2578]: W0731 23:46:16.263219    2578 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 31 23:46:16.271602 ip-10-0-136-108 kubenswrapper[2578]: W0731 23:46:16.263221    2578 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 31 23:46:16.271602 ip-10-0-136-108 kubenswrapper[2578]: W0731 23:46:16.263224    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 31 23:46:16.272083 ip-10-0-136-108 kubenswrapper[2578]: W0731 23:46:16.263226    2578 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 31 23:46:16.272083 ip-10-0-136-108 kubenswrapper[2578]: W0731 23:46:16.263229    2578 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 31 23:46:16.272083 ip-10-0-136-108 kubenswrapper[2578]: W0731 23:46:16.263231    2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 31 23:46:16.272083 ip-10-0-136-108 kubenswrapper[2578]: W0731 23:46:16.263235    2578 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 31 23:46:16.272083 ip-10-0-136-108 kubenswrapper[2578]: W0731 23:46:16.263239    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 31 23:46:16.272083 ip-10-0-136-108 kubenswrapper[2578]: W0731 23:46:16.263241    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 31 23:46:16.272083 ip-10-0-136-108 kubenswrapper[2578]: W0731 23:46:16.263244    2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 31 23:46:16.272083 ip-10-0-136-108 kubenswrapper[2578]: W0731 23:46:16.263247    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 31 23:46:16.272083 ip-10-0-136-108 kubenswrapper[2578]: W0731 23:46:16.263251    2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 31 23:46:16.272083 ip-10-0-136-108 kubenswrapper[2578]: W0731 23:46:16.263253    2578 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 31 23:46:16.272083 ip-10-0-136-108 kubenswrapper[2578]: W0731 23:46:16.263256    2578 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 31 23:46:16.272083 ip-10-0-136-108 kubenswrapper[2578]: W0731 23:46:16.263259    2578 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 31 23:46:16.272083 ip-10-0-136-108 kubenswrapper[2578]: W0731 23:46:16.263263    2578 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 31 23:46:16.272083 ip-10-0-136-108 kubenswrapper[2578]: W0731 23:46:16.263266    2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 31 23:46:16.272083 ip-10-0-136-108 kubenswrapper[2578]: W0731 23:46:16.263269    2578 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 31 23:46:16.272083 ip-10-0-136-108 kubenswrapper[2578]: W0731 23:46:16.263273    2578 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 31 23:46:16.272083 ip-10-0-136-108 kubenswrapper[2578]: W0731 23:46:16.263275    2578 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 31 23:46:16.272083 ip-10-0-136-108 kubenswrapper[2578]: W0731 23:46:16.263278    2578 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 31 23:46:16.272083 ip-10-0-136-108 kubenswrapper[2578]: W0731 23:46:16.263281    2578 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 31 23:46:16.272083 ip-10-0-136-108 kubenswrapper[2578]: W0731 23:46:16.263283    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 31 23:46:16.272600 ip-10-0-136-108 kubenswrapper[2578]: W0731 23:46:16.263286    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 31 23:46:16.272600 ip-10-0-136-108 kubenswrapper[2578]: W0731 23:46:16.263289    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 31 23:46:16.272600 ip-10-0-136-108 kubenswrapper[2578]: W0731 23:46:16.263291    2578 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 31 23:46:16.272600 ip-10-0-136-108 kubenswrapper[2578]: W0731 23:46:16.263294    2578 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 31 23:46:16.272600 ip-10-0-136-108 kubenswrapper[2578]: W0731 23:46:16.263298    2578 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 31 23:46:16.272600 ip-10-0-136-108 kubenswrapper[2578]: W0731 23:46:16.263300    2578 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 31 23:46:16.272600 ip-10-0-136-108 kubenswrapper[2578]: W0731 23:46:16.263303    2578 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 31 23:46:16.272600 ip-10-0-136-108 kubenswrapper[2578]: W0731 23:46:16.263306    2578 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 31 23:46:16.272600 ip-10-0-136-108 kubenswrapper[2578]: W0731 23:46:16.263308    2578 feature_gate.go:328] unrecognized feature gate: Example2
Jul 31 23:46:16.272600 ip-10-0-136-108 kubenswrapper[2578]: W0731 23:46:16.263311    2578 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 31 23:46:16.272600 ip-10-0-136-108 kubenswrapper[2578]: W0731 23:46:16.263314    2578 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 31 23:46:16.272600 ip-10-0-136-108 kubenswrapper[2578]: W0731 23:46:16.263317    2578 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 31 23:46:16.272600 ip-10-0-136-108 kubenswrapper[2578]: W0731 23:46:16.263319    2578 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 31 23:46:16.272600 ip-10-0-136-108 kubenswrapper[2578]: W0731 23:46:16.263323    2578 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 31 23:46:16.272600 ip-10-0-136-108 kubenswrapper[2578]: W0731 23:46:16.263327    2578 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 31 23:46:16.272600 ip-10-0-136-108 kubenswrapper[2578]: W0731 23:46:16.263330    2578 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 31 23:46:16.272600 ip-10-0-136-108 kubenswrapper[2578]: W0731 23:46:16.263332    2578 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 31 23:46:16.272600 ip-10-0-136-108 kubenswrapper[2578]: W0731 23:46:16.263335    2578 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 31 23:46:16.272600 ip-10-0-136-108 kubenswrapper[2578]: W0731 23:46:16.263338    2578 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 31 23:46:16.272600 ip-10-0-136-108 kubenswrapper[2578]: W0731 23:46:16.263341    2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 31 23:46:16.273097 ip-10-0-136-108 kubenswrapper[2578]: W0731 23:46:16.263343    2578 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 31 23:46:16.273097 ip-10-0-136-108 kubenswrapper[2578]: W0731 23:46:16.263346    2578 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 31 23:46:16.273097 ip-10-0-136-108 kubenswrapper[2578]: W0731 23:46:16.263348    2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 31 23:46:16.273097 ip-10-0-136-108 kubenswrapper[2578]: W0731 23:46:16.263351    2578 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 31 23:46:16.273097 ip-10-0-136-108 kubenswrapper[2578]: W0731 23:46:16.263353    2578 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 31 23:46:16.273097 ip-10-0-136-108 kubenswrapper[2578]: W0731 23:46:16.263356    2578 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 31 23:46:16.273097 ip-10-0-136-108 kubenswrapper[2578]: W0731 23:46:16.263359    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 31 23:46:16.273097 ip-10-0-136-108 kubenswrapper[2578]: W0731 23:46:16.263363    2578 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 31 23:46:16.273097 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:16.263868    2578 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 31 23:46:16.273097 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:16.271377    2578 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Jul 31 23:46:16.273097 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:16.271410    2578 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Jul 31 23:46:16.273097 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:16.272047    2578 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Jul 31 23:46:16.273097 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:16.272058    2578 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Jul 31 23:46:16.273097 ip-10-0-136-108 kubenswrapper[2578]: W0731 23:46:16.272130    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 31 23:46:16.273489 ip-10-0-136-108 kubenswrapper[2578]: W0731 23:46:16.272137    2578 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 31 23:46:16.273489 ip-10-0-136-108 kubenswrapper[2578]: W0731 23:46:16.272141    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 31 23:46:16.273489 ip-10-0-136-108 kubenswrapper[2578]: W0731 23:46:16.272144    2578 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 31 23:46:16.273489 ip-10-0-136-108 kubenswrapper[2578]: W0731 23:46:16.272148    2578 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 31 23:46:16.273489 ip-10-0-136-108 kubenswrapper[2578]: W0731 23:46:16.272151    2578 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 31 23:46:16.273489 ip-10-0-136-108 kubenswrapper[2578]: W0731 23:46:16.272154    2578 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 31 23:46:16.273489 ip-10-0-136-108 kubenswrapper[2578]: W0731 23:46:16.272157    2578 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 31 23:46:16.273489 ip-10-0-136-108 kubenswrapper[2578]: W0731 23:46:16.272160    2578 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 31 23:46:16.273489 ip-10-0-136-108 kubenswrapper[2578]: W0731 23:46:16.272163    2578 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 31 23:46:16.273489 ip-10-0-136-108 kubenswrapper[2578]: W0731 23:46:16.272166    2578 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 31 23:46:16.273489 ip-10-0-136-108 kubenswrapper[2578]: W0731 23:46:16.272182    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 31 23:46:16.273489 ip-10-0-136-108 kubenswrapper[2578]: W0731 23:46:16.272186    2578 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 31 23:46:16.273489 ip-10-0-136-108 kubenswrapper[2578]: W0731 23:46:16.272189    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 31 23:46:16.273489 ip-10-0-136-108 kubenswrapper[2578]: W0731 23:46:16.272192    2578 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 31 23:46:16.273489 ip-10-0-136-108 kubenswrapper[2578]: W0731 23:46:16.272195    2578 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 31 23:46:16.273489 ip-10-0-136-108 kubenswrapper[2578]: W0731 23:46:16.272198    2578 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 31 23:46:16.273489 ip-10-0-136-108 kubenswrapper[2578]: W0731 23:46:16.272200    2578 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 31 23:46:16.273489 ip-10-0-136-108 kubenswrapper[2578]: W0731 23:46:16.272203    2578 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 31 23:46:16.273489 ip-10-0-136-108 kubenswrapper[2578]: W0731 23:46:16.272205    2578 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 31 23:46:16.273489 ip-10-0-136-108 kubenswrapper[2578]: W0731 23:46:16.272208    2578 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 31 23:46:16.274000 ip-10-0-136-108 kubenswrapper[2578]: W0731 23:46:16.272211    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 31 23:46:16.274000 ip-10-0-136-108 kubenswrapper[2578]: W0731 23:46:16.272214    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 31 23:46:16.274000 ip-10-0-136-108 kubenswrapper[2578]: W0731 23:46:16.272217    2578 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 31 23:46:16.274000 ip-10-0-136-108 kubenswrapper[2578]: W0731 23:46:16.272219    2578 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 31 23:46:16.274000 ip-10-0-136-108 kubenswrapper[2578]: W0731 23:46:16.272222    2578 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 31 23:46:16.274000 ip-10-0-136-108 kubenswrapper[2578]: W0731 23:46:16.272224    2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 31 23:46:16.274000 ip-10-0-136-108 kubenswrapper[2578]: W0731 23:46:16.272227    2578 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 31 23:46:16.274000 ip-10-0-136-108 kubenswrapper[2578]: W0731 23:46:16.272230    2578 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 31 23:46:16.274000 ip-10-0-136-108 kubenswrapper[2578]: W0731 23:46:16.272238    2578 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 31 23:46:16.274000 ip-10-0-136-108 kubenswrapper[2578]: W0731 23:46:16.272242    2578 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 31 23:46:16.274000 ip-10-0-136-108 kubenswrapper[2578]: W0731 23:46:16.272246    2578 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 31 23:46:16.274000 ip-10-0-136-108 kubenswrapper[2578]: W0731 23:46:16.272250    2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 31 23:46:16.274000 ip-10-0-136-108 kubenswrapper[2578]: W0731 23:46:16.272253    2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 31 23:46:16.274000 ip-10-0-136-108 kubenswrapper[2578]: W0731 23:46:16.272256    2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 31 23:46:16.274000 ip-10-0-136-108 kubenswrapper[2578]: W0731 23:46:16.272259    2578 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 31 23:46:16.274000 ip-10-0-136-108 kubenswrapper[2578]: W0731 23:46:16.272262    2578 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 31 23:46:16.274000 ip-10-0-136-108 kubenswrapper[2578]: W0731 23:46:16.272265    2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 31 23:46:16.274000 ip-10-0-136-108 kubenswrapper[2578]: W0731 23:46:16.272268    2578 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 31 23:46:16.274000 ip-10-0-136-108 kubenswrapper[2578]: W0731 23:46:16.272271    2578 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 31 23:46:16.274637 ip-10-0-136-108 kubenswrapper[2578]: W0731 23:46:16.272274    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 31 23:46:16.274637 ip-10-0-136-108 kubenswrapper[2578]: W0731 23:46:16.272277    2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 31 23:46:16.274637 ip-10-0-136-108 kubenswrapper[2578]: W0731 23:46:16.272280    2578 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 31 23:46:16.274637 ip-10-0-136-108 kubenswrapper[2578]: W0731 23:46:16.272283    2578 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 31 23:46:16.274637 ip-10-0-136-108 kubenswrapper[2578]: W0731 23:46:16.272285    2578 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 31 23:46:16.274637 ip-10-0-136-108 kubenswrapper[2578]: W0731 23:46:16.272288    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 31 23:46:16.274637 ip-10-0-136-108 kubenswrapper[2578]: W0731 23:46:16.272291    2578 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 31 23:46:16.274637 ip-10-0-136-108 kubenswrapper[2578]: W0731 23:46:16.272294    2578 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 31 23:46:16.274637 ip-10-0-136-108 kubenswrapper[2578]: W0731 23:46:16.272298    2578 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 31 23:46:16.274637 ip-10-0-136-108 kubenswrapper[2578]: W0731 23:46:16.272301    2578 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 31 23:46:16.274637 ip-10-0-136-108 kubenswrapper[2578]: W0731 23:46:16.272305    2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 31 23:46:16.274637 ip-10-0-136-108 kubenswrapper[2578]: W0731 23:46:16.272308    2578 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 31 23:46:16.274637 ip-10-0-136-108 kubenswrapper[2578]: W0731 23:46:16.272311    2578 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 31 23:46:16.274637 ip-10-0-136-108 kubenswrapper[2578]: W0731 23:46:16.272314    2578 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 31 23:46:16.274637 ip-10-0-136-108 kubenswrapper[2578]: W0731 23:46:16.272317    2578 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 31 23:46:16.274637 ip-10-0-136-108 kubenswrapper[2578]: W0731 23:46:16.272320    2578 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 31 23:46:16.274637 ip-10-0-136-108 kubenswrapper[2578]: W0731 23:46:16.272322    2578 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 31 23:46:16.274637 ip-10-0-136-108 kubenswrapper[2578]: W0731 23:46:16.272325    2578 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 31 23:46:16.274637 ip-10-0-136-108 kubenswrapper[2578]: W0731 23:46:16.272328    2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 31 23:46:16.275109 ip-10-0-136-108 kubenswrapper[2578]: W0731 23:46:16.272332    2578 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 31 23:46:16.275109 ip-10-0-136-108 kubenswrapper[2578]: W0731 23:46:16.272335    2578 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 31 23:46:16.275109 ip-10-0-136-108 kubenswrapper[2578]: W0731 23:46:16.272338    2578 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 31 23:46:16.275109 ip-10-0-136-108 kubenswrapper[2578]: W0731 23:46:16.272342    2578 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 31 23:46:16.275109 ip-10-0-136-108 kubenswrapper[2578]: W0731 23:46:16.272346    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 31 23:46:16.275109 ip-10-0-136-108 kubenswrapper[2578]: W0731 23:46:16.272348    2578 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 31 23:46:16.275109 ip-10-0-136-108 kubenswrapper[2578]: W0731 23:46:16.272351    2578 feature_gate.go:328] unrecognized feature gate: Example
Jul 31 23:46:16.275109 ip-10-0-136-108 kubenswrapper[2578]: W0731 23:46:16.272354    2578 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 31 23:46:16.275109 ip-10-0-136-108 kubenswrapper[2578]: W0731 23:46:16.272356    2578 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 31 23:46:16.275109 ip-10-0-136-108 kubenswrapper[2578]: W0731 23:46:16.272359    2578 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 31 23:46:16.275109 ip-10-0-136-108 kubenswrapper[2578]: W0731 23:46:16.272363    2578 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 31 23:46:16.275109 ip-10-0-136-108 kubenswrapper[2578]: W0731 23:46:16.272367    2578 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 31 23:46:16.275109 ip-10-0-136-108 kubenswrapper[2578]: W0731 23:46:16.272370    2578 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 31 23:46:16.275109 ip-10-0-136-108 kubenswrapper[2578]: W0731 23:46:16.272373    2578 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 31 23:46:16.275109 ip-10-0-136-108 kubenswrapper[2578]: W0731 23:46:16.272375    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 31 23:46:16.275109 ip-10-0-136-108 kubenswrapper[2578]: W0731 23:46:16.272378    2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 31 23:46:16.275109 ip-10-0-136-108 kubenswrapper[2578]: W0731 23:46:16.272381    2578 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 31 23:46:16.275109 ip-10-0-136-108 kubenswrapper[2578]: W0731 23:46:16.272385    2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 31 23:46:16.275109 ip-10-0-136-108 kubenswrapper[2578]: W0731 23:46:16.272387    2578 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 31 23:46:16.275617 ip-10-0-136-108 kubenswrapper[2578]: W0731 23:46:16.272390    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 31 23:46:16.275617 ip-10-0-136-108 kubenswrapper[2578]: W0731 23:46:16.272393    2578 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 31 23:46:16.275617 ip-10-0-136-108 kubenswrapper[2578]: W0731 23:46:16.272396    2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 31 23:46:16.275617 ip-10-0-136-108 kubenswrapper[2578]: W0731 23:46:16.272399    2578 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 31 23:46:16.275617 ip-10-0-136-108 kubenswrapper[2578]: W0731 23:46:16.272402    2578 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 31 23:46:16.275617 ip-10-0-136-108 kubenswrapper[2578]: W0731 23:46:16.272404    2578 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 31 23:46:16.275617 ip-10-0-136-108 kubenswrapper[2578]: W0731 23:46:16.272407    2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 31 23:46:16.275617 ip-10-0-136-108 kubenswrapper[2578]: W0731 23:46:16.272410    2578 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 31 23:46:16.275617 ip-10-0-136-108 kubenswrapper[2578]: W0731 23:46:16.272413    2578 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 31 23:46:16.275617 ip-10-0-136-108 kubenswrapper[2578]: W0731 23:46:16.272415    2578 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 31 23:46:16.275617 ip-10-0-136-108 kubenswrapper[2578]: W0731 23:46:16.272418    2578 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 31 23:46:16.275617 ip-10-0-136-108 kubenswrapper[2578]: W0731 23:46:16.272422    2578 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 31 23:46:16.275617 ip-10-0-136-108 kubenswrapper[2578]: W0731 23:46:16.272425    2578 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 31 23:46:16.275617 ip-10-0-136-108 kubenswrapper[2578]: W0731 23:46:16.272427    2578 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 31 23:46:16.275617 ip-10-0-136-108 kubenswrapper[2578]: W0731 23:46:16.272431    2578 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 31 23:46:16.275617 ip-10-0-136-108 kubenswrapper[2578]: W0731 23:46:16.272434    2578 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 31 23:46:16.275617 ip-10-0-136-108 kubenswrapper[2578]: W0731 23:46:16.272437    2578 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 31 23:46:16.275617 ip-10-0-136-108 kubenswrapper[2578]: W0731 23:46:16.272441    2578 feature_gate.go:328] unrecognized feature gate: Example2
Jul 31 23:46:16.275617 ip-10-0-136-108 kubenswrapper[2578]: W0731 23:46:16.272443    2578 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 31 23:46:16.275617 ip-10-0-136-108 kubenswrapper[2578]: W0731 23:46:16.272446    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 31 23:46:16.276112 ip-10-0-136-108 kubenswrapper[2578]: W0731 23:46:16.272448    2578 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 31 23:46:16.276112 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:16.272454    2578 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 31 23:46:16.276112 ip-10-0-136-108 kubenswrapper[2578]: W0731 23:46:16.272595    2578 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 31 23:46:16.276112 ip-10-0-136-108 kubenswrapper[2578]: W0731 23:46:16.272602    2578 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 31 23:46:16.276112 ip-10-0-136-108 kubenswrapper[2578]: W0731 23:46:16.272606    2578 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 31 23:46:16.276112 ip-10-0-136-108 kubenswrapper[2578]: W0731 23:46:16.272609    2578 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 31 23:46:16.276112 ip-10-0-136-108 kubenswrapper[2578]: W0731 23:46:16.272613    2578 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 31 23:46:16.276112 ip-10-0-136-108 kubenswrapper[2578]: W0731 23:46:16.272616    2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 31 23:46:16.276112 ip-10-0-136-108 kubenswrapper[2578]: W0731 23:46:16.272619    2578 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 31 23:46:16.276112 ip-10-0-136-108 kubenswrapper[2578]: W0731 23:46:16.272622    2578 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 31 23:46:16.276112 ip-10-0-136-108 kubenswrapper[2578]: W0731 23:46:16.272625    2578 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 31 23:46:16.276112 ip-10-0-136-108 kubenswrapper[2578]: W0731 23:46:16.272627    2578 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 31 23:46:16.276112 ip-10-0-136-108 kubenswrapper[2578]: W0731 23:46:16.272631    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 31 23:46:16.276112 ip-10-0-136-108 kubenswrapper[2578]: W0731 23:46:16.272633    2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 31 23:46:16.276546 ip-10-0-136-108 kubenswrapper[2578]: W0731 23:46:16.272636    2578 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 31 23:46:16.276546 ip-10-0-136-108 kubenswrapper[2578]: W0731 23:46:16.272639    2578 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 31 23:46:16.276546 ip-10-0-136-108 kubenswrapper[2578]: W0731 23:46:16.272642    2578 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 31 23:46:16.276546 ip-10-0-136-108 kubenswrapper[2578]: W0731 23:46:16.272646    2578 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 31 23:46:16.276546 ip-10-0-136-108 kubenswrapper[2578]: W0731 23:46:16.272651    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 31 23:46:16.276546 ip-10-0-136-108 kubenswrapper[2578]: W0731 23:46:16.272654    2578 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 31 23:46:16.276546 ip-10-0-136-108 kubenswrapper[2578]: W0731 23:46:16.272657    2578 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 31 23:46:16.276546 ip-10-0-136-108 kubenswrapper[2578]: W0731 23:46:16.272659    2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 31 23:46:16.276546 ip-10-0-136-108 kubenswrapper[2578]: W0731 23:46:16.272662    2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 31 23:46:16.276546 ip-10-0-136-108 kubenswrapper[2578]: W0731 23:46:16.272665    2578 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 31 23:46:16.276546 ip-10-0-136-108 kubenswrapper[2578]: W0731 23:46:16.272667    2578 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 31 23:46:16.276546 ip-10-0-136-108 kubenswrapper[2578]: W0731 23:46:16.272670    2578 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 31 23:46:16.276546 ip-10-0-136-108 kubenswrapper[2578]: W0731 23:46:16.272674    2578 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 31 23:46:16.276546 ip-10-0-136-108 kubenswrapper[2578]: W0731 23:46:16.272677    2578 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 31 23:46:16.276546 ip-10-0-136-108 kubenswrapper[2578]: W0731 23:46:16.272681    2578 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 31 23:46:16.276546 ip-10-0-136-108 kubenswrapper[2578]: W0731 23:46:16.272684    2578 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 31 23:46:16.276546 ip-10-0-136-108 kubenswrapper[2578]: W0731 23:46:16.272688    2578 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 31 23:46:16.276546 ip-10-0-136-108 kubenswrapper[2578]: W0731 23:46:16.272691    2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 31 23:46:16.276546 ip-10-0-136-108 kubenswrapper[2578]: W0731 23:46:16.272694    2578 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 31 23:46:16.276546 ip-10-0-136-108 kubenswrapper[2578]: W0731 23:46:16.272697    2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 31 23:46:16.277061 ip-10-0-136-108 kubenswrapper[2578]: W0731 23:46:16.272699    2578 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 31 23:46:16.277061 ip-10-0-136-108 kubenswrapper[2578]: W0731 23:46:16.272702    2578 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 31 23:46:16.277061 ip-10-0-136-108 kubenswrapper[2578]: W0731 23:46:16.272704    2578 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 31 23:46:16.277061 ip-10-0-136-108 kubenswrapper[2578]: W0731 23:46:16.272707    2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 31 23:46:16.277061 ip-10-0-136-108 kubenswrapper[2578]: W0731 23:46:16.272710    2578 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 31 23:46:16.277061 ip-10-0-136-108 kubenswrapper[2578]: W0731 23:46:16.272712    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 31 23:46:16.277061 ip-10-0-136-108 kubenswrapper[2578]: W0731 23:46:16.272715    2578 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 31 23:46:16.277061 ip-10-0-136-108 kubenswrapper[2578]: W0731 23:46:16.272717    2578 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 31 23:46:16.277061 ip-10-0-136-108 kubenswrapper[2578]: W0731 23:46:16.272720    2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 31 23:46:16.277061 ip-10-0-136-108 kubenswrapper[2578]: W0731 23:46:16.272723    2578 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 31 23:46:16.277061 ip-10-0-136-108 kubenswrapper[2578]: W0731 23:46:16.272727    2578 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 31 23:46:16.277061 ip-10-0-136-108 kubenswrapper[2578]: W0731 23:46:16.272729    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 31 23:46:16.277061 ip-10-0-136-108 kubenswrapper[2578]: W0731 23:46:16.272732    2578 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 31 23:46:16.277061 ip-10-0-136-108 kubenswrapper[2578]: W0731 23:46:16.272734    2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 31 23:46:16.277061 ip-10-0-136-108 kubenswrapper[2578]: W0731 23:46:16.272737    2578 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 31 23:46:16.277061 ip-10-0-136-108 kubenswrapper[2578]: W0731 23:46:16.272739    2578 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 31 23:46:16.277061 ip-10-0-136-108 kubenswrapper[2578]: W0731 23:46:16.272742    2578 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 31 23:46:16.277061 ip-10-0-136-108 kubenswrapper[2578]: W0731 23:46:16.272744    2578 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 31 23:46:16.277061 ip-10-0-136-108 kubenswrapper[2578]: W0731 23:46:16.272747    2578 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 31 23:46:16.277061 ip-10-0-136-108 kubenswrapper[2578]: W0731 23:46:16.272749    2578 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 31 23:46:16.277603 ip-10-0-136-108 kubenswrapper[2578]: W0731 23:46:16.272752    2578 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 31 23:46:16.277603 ip-10-0-136-108 kubenswrapper[2578]: W0731 23:46:16.272755    2578 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 31 23:46:16.277603 ip-10-0-136-108 kubenswrapper[2578]: W0731 23:46:16.272757    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 31 23:46:16.277603 ip-10-0-136-108 kubenswrapper[2578]: W0731 23:46:16.272760    2578 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 31 23:46:16.277603 ip-10-0-136-108 kubenswrapper[2578]: W0731 23:46:16.272764    2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 31 23:46:16.277603 ip-10-0-136-108 kubenswrapper[2578]: W0731 23:46:16.272767    2578 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 31 23:46:16.277603 ip-10-0-136-108 kubenswrapper[2578]: W0731 23:46:16.272769    2578 feature_gate.go:328] unrecognized feature gate: Example2
Jul 31 23:46:16.277603 ip-10-0-136-108 kubenswrapper[2578]: W0731 23:46:16.272772    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 31 23:46:16.277603 ip-10-0-136-108 kubenswrapper[2578]: W0731 23:46:16.272775    2578 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 31 23:46:16.277603 ip-10-0-136-108 kubenswrapper[2578]: W0731 23:46:16.272778    2578 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 31 23:46:16.277603 ip-10-0-136-108 kubenswrapper[2578]: W0731 23:46:16.272781    2578 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 31 23:46:16.277603 ip-10-0-136-108 kubenswrapper[2578]: W0731 23:46:16.272784    2578 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 31 23:46:16.277603 ip-10-0-136-108 kubenswrapper[2578]: W0731 23:46:16.272786    2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 31 23:46:16.277603 ip-10-0-136-108 kubenswrapper[2578]: W0731 23:46:16.272788    2578 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 31 23:46:16.277603 ip-10-0-136-108 kubenswrapper[2578]: W0731 23:46:16.272791    2578 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 31 23:46:16.277603 ip-10-0-136-108 kubenswrapper[2578]: W0731 23:46:16.272793    2578 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 31 23:46:16.277603 ip-10-0-136-108 kubenswrapper[2578]: W0731 23:46:16.272796    2578 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 31 23:46:16.277603 ip-10-0-136-108 kubenswrapper[2578]: W0731 23:46:16.272798    2578 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 31 23:46:16.277603 ip-10-0-136-108 kubenswrapper[2578]: W0731 23:46:16.272802    2578 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 31 23:46:16.277603 ip-10-0-136-108 kubenswrapper[2578]: W0731 23:46:16.272804    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 31 23:46:16.278137 ip-10-0-136-108 kubenswrapper[2578]: W0731 23:46:16.272807    2578 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 31 23:46:16.278137 ip-10-0-136-108 kubenswrapper[2578]: W0731 23:46:16.272809    2578 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 31 23:46:16.278137 ip-10-0-136-108 kubenswrapper[2578]: W0731 23:46:16.272812    2578 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 31 23:46:16.278137 ip-10-0-136-108 kubenswrapper[2578]: W0731 23:46:16.272814    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 31 23:46:16.278137 ip-10-0-136-108 kubenswrapper[2578]: W0731 23:46:16.272817    2578 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 31 23:46:16.278137 ip-10-0-136-108 kubenswrapper[2578]: W0731 23:46:16.272819    2578 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 31 23:46:16.278137 ip-10-0-136-108 kubenswrapper[2578]: W0731 23:46:16.272824    2578 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 31 23:46:16.278137 ip-10-0-136-108 kubenswrapper[2578]: W0731 23:46:16.272828    2578 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 31 23:46:16.278137 ip-10-0-136-108 kubenswrapper[2578]: W0731 23:46:16.272831    2578 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 31 23:46:16.278137 ip-10-0-136-108 kubenswrapper[2578]: W0731 23:46:16.272834    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 31 23:46:16.278137 ip-10-0-136-108 kubenswrapper[2578]: W0731 23:46:16.272837    2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 31 23:46:16.278137 ip-10-0-136-108 kubenswrapper[2578]: W0731 23:46:16.272840    2578 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 31 23:46:16.278137 ip-10-0-136-108 kubenswrapper[2578]: W0731 23:46:16.272842    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 31 23:46:16.278137 ip-10-0-136-108 kubenswrapper[2578]: W0731 23:46:16.272845    2578 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 31 23:46:16.278137 ip-10-0-136-108 kubenswrapper[2578]: W0731 23:46:16.272848    2578 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 31 23:46:16.278137 ip-10-0-136-108 kubenswrapper[2578]: W0731 23:46:16.272851    2578 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 31 23:46:16.278137 ip-10-0-136-108 kubenswrapper[2578]: W0731 23:46:16.272858    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 31 23:46:16.278137 ip-10-0-136-108 kubenswrapper[2578]: W0731 23:46:16.272861    2578 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 31 23:46:16.278137 ip-10-0-136-108 kubenswrapper[2578]: W0731 23:46:16.272864    2578 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 31 23:46:16.278632 ip-10-0-136-108 kubenswrapper[2578]: W0731 23:46:16.272866    2578 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 31 23:46:16.278632 ip-10-0-136-108 kubenswrapper[2578]: W0731 23:46:16.272869    2578 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 31 23:46:16.278632 ip-10-0-136-108 kubenswrapper[2578]: W0731 23:46:16.272871    2578 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 31 23:46:16.278632 ip-10-0-136-108 kubenswrapper[2578]: W0731 23:46:16.272875    2578 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 31 23:46:16.278632 ip-10-0-136-108 kubenswrapper[2578]: W0731 23:46:16.272877    2578 feature_gate.go:328] unrecognized feature gate: Example
Jul 31 23:46:16.278632 ip-10-0-136-108 kubenswrapper[2578]: W0731 23:46:16.272881    2578 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 31 23:46:16.278632 ip-10-0-136-108 kubenswrapper[2578]: W0731 23:46:16.272883    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 31 23:46:16.278632 ip-10-0-136-108 kubenswrapper[2578]: W0731 23:46:16.272886    2578 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 31 23:46:16.278632 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:16.272892    2578 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 31 23:46:16.278632 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:16.273056    2578 server.go:962] "Client rotation is on, will bootstrap in background"
Jul 31 23:46:16.278632 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:16.276157    2578 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Jul 31 23:46:16.278632 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:16.277468    2578 server.go:1019] "Starting client certificate rotation"
Jul 31 23:46:16.278632 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:16.277596    2578 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 31 23:46:16.278632 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:16.277654    2578 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 31 23:46:16.279976 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:16.279956    2578 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 31 23:46:16.281631 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:16.281604    2578 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 31 23:46:16.290364 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:16.290337    2578 log.go:25] "Validated CRI v1 runtime API"
Jul 31 23:46:16.292766 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:16.292742    2578 log.go:25] "Validated CRI v1 image API"
Jul 31 23:46:16.296120 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:16.296093    2578 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Jul 31 23:46:16.300507 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:16.300479    2578 fs.go:135] Filesystem UUIDs: map[35e073df-5238-40db-92b3-2f203d5ca2fb:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 d3ad1162-1637-4ec2-8ea8-99f4f8235834:/dev/nvme0n1p3]
Jul 31 23:46:16.300577 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:16.300506    2578 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Jul 31 23:46:16.305776 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:16.305646    2578 manager.go:216] Machine: {Timestamp:2026-07-31 23:46:16.304469877 +0000 UTC m=+0.272220626 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3103243 MemoryCapacity:33164496896 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec297924749512961f4db85f1b529c6a SystemUUID:ec297924-7495-1296-1f4d-b85f1b529c6a BootID:eb0c34c0-8d16-4844-b798-bf7f078fa839 Filesystems:[{Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582250496 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582246400 Type:vfs Inodes:4048400 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632902656 Type:vfs Inodes:819200 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:0f:f1:2d:05:11 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:0f:f1:2d:05:11 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:3a:33:ea:dc:6b:37 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164496896 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Jul 31 23:46:16.305776 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:16.305759    2578 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Jul 31 23:46:16.305932 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:16.305845    2578 manager.go:232] Version: {KernelVersion:5.14.0-570.131.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260727-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Jul 31 23:46:16.306538 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:16.306522    2578 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 31 23:46:16.308331 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:16.308301    2578 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Jul 31 23:46:16.308481 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:16.308334    2578 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-136-108.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Jul 31 23:46:16.309004 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:16.308994    2578 topology_manager.go:138] "Creating topology manager with none policy"
Jul 31 23:46:16.309040 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:16.309006    2578 container_manager_linux.go:309] "Creating device plugin manager"
Jul 31 23:46:16.309040 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:16.309014    2578 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 31 23:46:16.309598 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:16.309587    2578 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 31 23:46:16.310481 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:16.310471    2578 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Jul 31 23:46:16.314824 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:16.314811    2578 state_mem.go:36] "Initialized new in-memory state store"
Jul 31 23:46:16.314954 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:16.314943    2578 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Jul 31 23:46:16.314987 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:16.314977    2578 kubelet.go:486] "Attempting to sync node with API server"
Jul 31 23:46:16.315020 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:16.314988    2578 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Jul 31 23:46:16.315020 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:16.314999    2578 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Jul 31 23:46:16.315020 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:16.315008    2578 kubelet.go:398] "Adding apiserver pod source"
Jul 31 23:46:16.315020 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:16.315018    2578 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Jul 31 23:46:16.315443 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:16.315428    2578 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 31 23:46:16.315521 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:16.315449    2578 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 31 23:46:16.317343 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:16.317325    2578 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.10-2.rhaos4.21.git83ee25b.el9" apiVersion="v1"
Jul 31 23:46:16.319349 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:16.319334    2578 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Jul 31 23:46:16.319393 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:16.319369    2578 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Jul 31 23:46:16.321053 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:16.321039    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Jul 31 23:46:16.321117 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:16.321058    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Jul 31 23:46:16.321117 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:16.321065    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Jul 31 23:46:16.321117 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:16.321072    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Jul 31 23:46:16.321117 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:16.321086    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Jul 31 23:46:16.321117 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:16.321095    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Jul 31 23:46:16.321117 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:16.321106    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Jul 31 23:46:16.321117 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:16.321111    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Jul 31 23:46:16.321117 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:16.321117    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Jul 31 23:46:16.321367 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:16.321123    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Jul 31 23:46:16.321367 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:16.321132    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Jul 31 23:46:16.321367 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:16.321137    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Jul 31 23:46:16.321794 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:16.321784    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Jul 31 23:46:16.321794 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:16.321793    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Jul 31 23:46:16.325129 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:16.325116    2578 server.go:1268] "Started kubelet"
Jul 31 23:46:16.325234 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:16.325206    2578 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Jul 31 23:46:16.325234 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:16.325215    2578 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Jul 31 23:46:16.325340 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:16.325269    2578 server_v1.go:49] "podresources" method="list" useActivePods=true
Jul 31 23:46:16.328809 ip-10-0-136-108 systemd[1]: Started Kubernetes Kubelet.
Jul 31 23:46:16.330710 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:16.330091    2578 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Jul 31 23:46:16.330710 ip-10-0-136-108 kubenswrapper[2578]: E0731 23:46:16.330361    2578 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-136-108.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Jul 31 23:46:16.330710 ip-10-0-136-108 kubenswrapper[2578]: E0731 23:46:16.330465    2578 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Jul 31 23:46:16.330710 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:16.330484    2578 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-136-108.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Jul 31 23:46:16.331689 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:16.331655    2578 server.go:310] "Adding debug handlers to kubelet server"
Jul 31 23:46:16.333243 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:16.333220    2578 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-78j8g"
Jul 31 23:46:16.333411 ip-10-0-136-108 kubenswrapper[2578]: E0731 23:46:16.333386    2578 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Jul 31 23:46:16.336307 ip-10-0-136-108 kubenswrapper[2578]: E0731 23:46:16.335541    2578 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-136-108.ec2.internal.18c7849a7c8c36a7  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-136-108.ec2.internal,UID:ip-10-0-136-108.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-136-108.ec2.internal,},FirstTimestamp:2026-07-31 23:46:16.325084839 +0000 UTC m=+0.292835589,LastTimestamp:2026-07-31 23:46:16.325084839 +0000 UTC m=+0.292835589,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-136-108.ec2.internal,}"
Jul 31 23:46:16.336461 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:16.336441    2578 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Jul 31 23:46:16.336461 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:16.336458    2578 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Jul 31 23:46:16.336613 ip-10-0-136-108 kubenswrapper[2578]: E0731 23:46:16.336534    2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-108.ec2.internal\" not found"
Jul 31 23:46:16.337086 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:16.337022    2578 volume_manager.go:311] "The desired_state_of_world populator starts"
Jul 31 23:46:16.337086 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:16.337022    2578 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Jul 31 23:46:16.337086 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:16.337041    2578 volume_manager.go:313] "Starting Kubelet Volume Manager"
Jul 31 23:46:16.337279 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:16.337150    2578 reconstruct.go:97] "Volume reconstruction finished"
Jul 31 23:46:16.337279 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:16.337157    2578 reconciler.go:29] "Reconciler: start to sync state"
Jul 31 23:46:16.337373 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:16.337302    2578 factory.go:55] Registering systemd factory
Jul 31 23:46:16.337373 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:16.337319    2578 factory.go:223] Registration of the systemd container factory successfully
Jul 31 23:46:16.337579 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:16.337563    2578 factory.go:153] Registering CRI-O factory
Jul 31 23:46:16.337633 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:16.337582    2578 factory.go:223] Registration of the crio container factory successfully
Jul 31 23:46:16.337674 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:16.337634    2578 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Jul 31 23:46:16.337674 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:16.337665    2578 factory.go:103] Registering Raw factory
Jul 31 23:46:16.337733 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:16.337678    2578 manager.go:1196] Started watching for new ooms in manager
Jul 31 23:46:16.338651 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:16.338628    2578 manager.go:349] Starting recovery of all containers
Jul 31 23:46:16.341252 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:16.341229    2578 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-78j8g"
Jul 31 23:46:16.344678 ip-10-0-136-108 kubenswrapper[2578]: E0731 23:46:16.344654    2578 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Jul 31 23:46:16.344784 ip-10-0-136-108 kubenswrapper[2578]: E0731 23:46:16.344766    2578 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-136-108.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Jul 31 23:46:16.347783 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:16.347761    2578 manager.go:354] Recovery completed
Jul 31 23:46:16.353836 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:16.353823    2578 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 31 23:46:16.356679 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:16.356661    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-108.ec2.internal" event="NodeHasSufficientMemory"
Jul 31 23:46:16.356760 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:16.356697    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-108.ec2.internal" event="NodeHasNoDiskPressure"
Jul 31 23:46:16.356760 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:16.356712    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-108.ec2.internal" event="NodeHasSufficientPID"
Jul 31 23:46:16.357239 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:16.357221    2578 cpu_manager.go:222] "Starting CPU manager" policy="none"
Jul 31 23:46:16.357239 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:16.357234    2578 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Jul 31 23:46:16.357341 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:16.357249    2578 state_mem.go:36] "Initialized new in-memory state store"
Jul 31 23:46:16.357648 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:16.357240    2578 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 31 23:46:16.358305 ip-10-0-136-108 kubenswrapper[2578]: E0731 23:46:16.358224    2578 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-136-108.ec2.internal.18c7849a7e6e4fab  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-136-108.ec2.internal,UID:ip-10-0-136-108.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-136-108.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-136-108.ec2.internal,},FirstTimestamp:2026-07-31 23:46:16.356679595 +0000 UTC m=+0.324430350,LastTimestamp:2026-07-31 23:46:16.356679595 +0000 UTC m=+0.324430350,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-136-108.ec2.internal,}"
Jul 31 23:46:16.359231 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:16.359215    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-108.ec2.internal" event="NodeHasSufficientMemory"
Jul 31 23:46:16.359307 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:16.359243    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-108.ec2.internal" event="NodeHasNoDiskPressure"
Jul 31 23:46:16.359307 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:16.359266    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-108.ec2.internal" event="NodeHasSufficientPID"
Jul 31 23:46:16.359382 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:16.359370    2578 policy_none.go:49] "None policy: Start"
Jul 31 23:46:16.359412 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:16.359388    2578 memory_manager.go:187] "Starting memorymanager" policy="None"
Jul 31 23:46:16.359412 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:16.359399    2578 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Jul 31 23:46:16.361444 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:16.361432    2578 policy_none.go:47] "Start"
Jul 31 23:46:16.395001 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:16.394769    2578 manager.go:337] "Starting Device Plugin manager"
Jul 31 23:46:16.395121 ip-10-0-136-108 kubenswrapper[2578]: E0731 23:46:16.395027    2578 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Jul 31 23:46:16.395121 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:16.395040    2578 server.go:87] "Starting device plugin registration server"
Jul 31 23:46:16.395324 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:16.395311    2578 eviction_manager.go:189] "Eviction manager: starting control loop"
Jul 31 23:46:16.395377 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:16.395327    2578 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Jul 31 23:46:16.395480 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:16.395458    2578 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Jul 31 23:46:16.395592 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:16.395560    2578 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Jul 31 23:46:16.395592 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:16.395574    2578 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Jul 31 23:46:16.396316 ip-10-0-136-108 kubenswrapper[2578]: E0731 23:46:16.396294    2578 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Jul 31 23:46:16.396391 ip-10-0-136-108 kubenswrapper[2578]: E0731 23:46:16.396345    2578 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-136-108.ec2.internal\" not found"
Jul 31 23:46:16.481495 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:16.481451    2578 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Jul 31 23:46:16.483843 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:16.482843    2578 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Jul 31 23:46:16.483843 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:16.482880    2578 status_manager.go:244] "Starting to sync pod status with apiserver"
Jul 31 23:46:16.483843 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:16.482903    2578 kubelet.go:2448] "Starting kubelet main sync loop"
Jul 31 23:46:16.483843 ip-10-0-136-108 kubenswrapper[2578]: E0731 23:46:16.482943    2578 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Jul 31 23:46:16.485586 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:16.485567    2578 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 31 23:46:16.495464 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:16.495446    2578 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 31 23:46:16.496349 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:16.496328    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-108.ec2.internal" event="NodeHasSufficientMemory"
Jul 31 23:46:16.496442 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:16.496366    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-108.ec2.internal" event="NodeHasNoDiskPressure"
Jul 31 23:46:16.496442 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:16.496380    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-108.ec2.internal" event="NodeHasSufficientPID"
Jul 31 23:46:16.496442 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:16.496406    2578 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-136-108.ec2.internal"
Jul 31 23:46:16.503231 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:16.503189    2578 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-136-108.ec2.internal"
Jul 31 23:46:16.503231 ip-10-0-136-108 kubenswrapper[2578]: E0731 23:46:16.503211    2578 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-136-108.ec2.internal\": node \"ip-10-0-136-108.ec2.internal\" not found"
Jul 31 23:46:16.522295 ip-10-0-136-108 kubenswrapper[2578]: E0731 23:46:16.522278    2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-108.ec2.internal\" not found"
Jul 31 23:46:16.584955 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:16.584906    2578 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-108.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-136-108.ec2.internal"]
Jul 31 23:46:16.585068 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:16.585000    2578 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 31 23:46:16.586653 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:16.586636    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-108.ec2.internal" event="NodeHasSufficientMemory"
Jul 31 23:46:16.586745 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:16.586669    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-108.ec2.internal" event="NodeHasNoDiskPressure"
Jul 31 23:46:16.586745 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:16.586687    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-108.ec2.internal" event="NodeHasSufficientPID"
Jul 31 23:46:16.589184 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:16.589168    2578 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 31 23:46:16.589322 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:16.589306    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-108.ec2.internal"
Jul 31 23:46:16.589369 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:16.589359    2578 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 31 23:46:16.590031 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:16.590015    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-108.ec2.internal" event="NodeHasSufficientMemory"
Jul 31 23:46:16.590107 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:16.590048    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-108.ec2.internal" event="NodeHasNoDiskPressure"
Jul 31 23:46:16.590107 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:16.590065    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-108.ec2.internal" event="NodeHasSufficientPID"
Jul 31 23:46:16.590107 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:16.590070    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-108.ec2.internal" event="NodeHasSufficientMemory"
Jul 31 23:46:16.590107 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:16.590092    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-108.ec2.internal" event="NodeHasNoDiskPressure"
Jul 31 23:46:16.590107 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:16.590101    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-108.ec2.internal" event="NodeHasSufficientPID"
Jul 31 23:46:16.592452 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:16.592438    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-108.ec2.internal"
Jul 31 23:46:16.592515 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:16.592467    2578 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 31 23:46:16.593267 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:16.593253    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-108.ec2.internal" event="NodeHasSufficientMemory"
Jul 31 23:46:16.593337 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:16.593279    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-108.ec2.internal" event="NodeHasNoDiskPressure"
Jul 31 23:46:16.593337 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:16.593289    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-108.ec2.internal" event="NodeHasSufficientPID"
Jul 31 23:46:16.607793 ip-10-0-136-108 kubenswrapper[2578]: E0731 23:46:16.607769    2578 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-136-108.ec2.internal\" not found" node="ip-10-0-136-108.ec2.internal"
Jul 31 23:46:16.611977 ip-10-0-136-108 kubenswrapper[2578]: E0731 23:46:16.611960    2578 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-136-108.ec2.internal\" not found" node="ip-10-0-136-108.ec2.internal"
Jul 31 23:46:16.622330 ip-10-0-136-108 kubenswrapper[2578]: E0731 23:46:16.622314    2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-108.ec2.internal\" not found"
Jul 31 23:46:16.722567 ip-10-0-136-108 kubenswrapper[2578]: E0731 23:46:16.722532    2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-108.ec2.internal\" not found"
Jul 31 23:46:16.738898 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:16.738866    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/cc577a92ce0dd873599a996e54a72791-config\") pod \"kube-apiserver-proxy-ip-10-0-136-108.ec2.internal\" (UID: \"cc577a92ce0dd873599a996e54a72791\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-136-108.ec2.internal"
Jul 31 23:46:16.739049 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:16.738904    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/739886fda1f37e5f1a9572bed84e6cb4-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-136-108.ec2.internal\" (UID: \"739886fda1f37e5f1a9572bed84e6cb4\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-108.ec2.internal"
Jul 31 23:46:16.739049 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:16.738935    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/739886fda1f37e5f1a9572bed84e6cb4-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-136-108.ec2.internal\" (UID: \"739886fda1f37e5f1a9572bed84e6cb4\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-108.ec2.internal"
Jul 31 23:46:16.822890 ip-10-0-136-108 kubenswrapper[2578]: E0731 23:46:16.822821    2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-108.ec2.internal\" not found"
Jul 31 23:46:16.839403 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:16.839359    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/cc577a92ce0dd873599a996e54a72791-config\") pod \"kube-apiserver-proxy-ip-10-0-136-108.ec2.internal\" (UID: \"cc577a92ce0dd873599a996e54a72791\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-136-108.ec2.internal"
Jul 31 23:46:16.839526 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:16.839411    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/739886fda1f37e5f1a9572bed84e6cb4-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-136-108.ec2.internal\" (UID: \"739886fda1f37e5f1a9572bed84e6cb4\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-108.ec2.internal"
Jul 31 23:46:16.839526 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:16.839442    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/739886fda1f37e5f1a9572bed84e6cb4-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-136-108.ec2.internal\" (UID: \"739886fda1f37e5f1a9572bed84e6cb4\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-108.ec2.internal"
Jul 31 23:46:16.839526 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:16.839494    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/739886fda1f37e5f1a9572bed84e6cb4-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-136-108.ec2.internal\" (UID: \"739886fda1f37e5f1a9572bed84e6cb4\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-108.ec2.internal"
Jul 31 23:46:16.839646 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:16.839496    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/739886fda1f37e5f1a9572bed84e6cb4-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-136-108.ec2.internal\" (UID: \"739886fda1f37e5f1a9572bed84e6cb4\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-108.ec2.internal"
Jul 31 23:46:16.839646 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:16.839496    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/cc577a92ce0dd873599a996e54a72791-config\") pod \"kube-apiserver-proxy-ip-10-0-136-108.ec2.internal\" (UID: \"cc577a92ce0dd873599a996e54a72791\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-136-108.ec2.internal"
Jul 31 23:46:16.915356 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:16.915321    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-108.ec2.internal"
Jul 31 23:46:16.915493 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:16.915321    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-108.ec2.internal"
Jul 31 23:46:16.923131 ip-10-0-136-108 kubenswrapper[2578]: E0731 23:46:16.923110    2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-108.ec2.internal\" not found"
Jul 31 23:46:17.026313 ip-10-0-136-108 kubenswrapper[2578]: E0731 23:46:17.026264    2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-108.ec2.internal\" not found"
Jul 31 23:46:17.126938 ip-10-0-136-108 kubenswrapper[2578]: E0731 23:46:17.126844    2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-108.ec2.internal\" not found"
Jul 31 23:46:17.234140 ip-10-0-136-108 kubenswrapper[2578]: E0731 23:46:17.234102    2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-108.ec2.internal\" not found"
Jul 31 23:46:17.277798 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:17.277760    2578 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Jul 31 23:46:17.278336 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:17.277948    2578 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Jul 31 23:46:17.334848 ip-10-0-136-108 kubenswrapper[2578]: E0731 23:46:17.334812    2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-108.ec2.internal\" not found"
Jul 31 23:46:17.346933 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:17.346881    2578 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-30 23:41:17 +0000 UTC" deadline="2028-04-26 06:10:08.451078974 +0000 UTC"
Jul 31 23:46:17.346933 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:17.346928    2578 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="15222h23m51.104154971s"
Jul 31 23:46:17.381657 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:17.381590    2578 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 31 23:46:17.435552 ip-10-0-136-108 kubenswrapper[2578]: E0731 23:46:17.435498    2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-108.ec2.internal\" not found"
Jul 31 23:46:17.495964 ip-10-0-136-108 kubenswrapper[2578]: W0731 23:46:17.495933    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcc577a92ce0dd873599a996e54a72791.slice/crio-642ce8073770e2e16ebc1ce259523e225d2bbd63dece78f533c7d452a5dca311 WatchSource:0}: Error finding container 642ce8073770e2e16ebc1ce259523e225d2bbd63dece78f533c7d452a5dca311: Status 404 returned error can't find the container with id 642ce8073770e2e16ebc1ce259523e225d2bbd63dece78f533c7d452a5dca311
Jul 31 23:46:17.500046 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:17.500030    2578 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 31 23:46:17.536307 ip-10-0-136-108 kubenswrapper[2578]: E0731 23:46:17.536273    2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-108.ec2.internal\" not found"
Jul 31 23:46:17.640967 ip-10-0-136-108 kubenswrapper[2578]: E0731 23:46:17.640872    2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-108.ec2.internal\" not found"
Jul 31 23:46:17.715917 ip-10-0-136-108 kubenswrapper[2578]: W0731 23:46:17.715889    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod739886fda1f37e5f1a9572bed84e6cb4.slice/crio-3d5536fad15f0660c7b4cb51f341a50aeea3a7870f5d7cdfdb16691cd42c231a WatchSource:0}: Error finding container 3d5536fad15f0660c7b4cb51f341a50aeea3a7870f5d7cdfdb16691cd42c231a: Status 404 returned error can't find the container with id 3d5536fad15f0660c7b4cb51f341a50aeea3a7870f5d7cdfdb16691cd42c231a
Jul 31 23:46:17.741538 ip-10-0-136-108 kubenswrapper[2578]: E0731 23:46:17.741505    2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-108.ec2.internal\" not found"
Jul 31 23:46:17.839525 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:17.839492    2578 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 31 23:46:17.913424 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:17.913336    2578 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 31 23:46:17.936824 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:17.936772    2578 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-108.ec2.internal"
Jul 31 23:46:17.947909 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:17.947862    2578 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 31 23:46:17.948840 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:17.948821    2578 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-108.ec2.internal"
Jul 31 23:46:17.957589 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:17.957562    2578 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 31 23:46:18.318570 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:18.318480    2578 apiserver.go:52] "Watching apiserver"
Jul 31 23:46:18.326608 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:18.326582    2578 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Jul 31 23:46:18.327185 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:18.327145    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-108.ec2.internal","openshift-multus/multus-additional-cni-plugins-h8nq8","openshift-ovn-kubernetes/ovnkube-node-m9px5","kube-system/konnectivity-agent-gttdm","openshift-cluster-node-tuning-operator/tuned-p7bwg","openshift-dns/node-resolver-wr2rk","openshift-image-registry/node-ca-9hxzf","openshift-multus/multus-hlpzq","openshift-multus/network-metrics-daemon-qs7p7","openshift-network-diagnostics/network-check-target-8d9rr","openshift-network-operator/iptables-alerter-xssj7","kube-system/global-pull-secret-syncer-xlv68","kube-system/kube-apiserver-proxy-ip-10-0-136-108.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z94nv"]
Jul 31 23:46:18.330225 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:18.329919    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-9hxzf"
Jul 31 23:46:18.332395 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:18.332364    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-xssj7"
Jul 31 23:46:18.332395 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:18.332387    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Jul 31 23:46:18.332834 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:18.332748    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Jul 31 23:46:18.332834 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:18.332766    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Jul 31 23:46:18.332834 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:18.332773    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-cj8hb\""
Jul 31 23:46:18.334902 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:18.334213    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Jul 31 23:46:18.334902 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:18.334645    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Jul 31 23:46:18.334902 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:18.334836    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Jul 31 23:46:18.335180 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:18.335061    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-bvnmt\""
Jul 31 23:46:18.335523 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:18.335500    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-m9px5"
Jul 31 23:46:18.337332 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:18.337231    2578 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Jul 31 23:46:18.338018 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:18.337474    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Jul 31 23:46:18.338018 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:18.337870    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Jul 31 23:46:18.338018 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:18.337890    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Jul 31 23:46:18.338018 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:18.337936    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-9jqzw\""
Jul 31 23:46:18.338290 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:18.338158    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Jul 31 23:46:18.338290 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:18.338170    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Jul 31 23:46:18.338617 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:18.338459    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Jul 31 23:46:18.338968 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:18.338868    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-gttdm"
Jul 31 23:46:18.340503 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:18.340477    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-clznw\""
Jul 31 23:46:18.340857 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:18.340842    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Jul 31 23:46:18.341171 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:18.341151    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Jul 31 23:46:18.342090 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:18.342072    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-p7bwg"
Jul 31 23:46:18.344375 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:18.344108    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Jul 31 23:46:18.344375 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:18.344111    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-mklcr\""
Jul 31 23:46:18.344375 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:18.344218    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Jul 31 23:46:18.344586 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:18.344524    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-wr2rk"
Jul 31 23:46:18.347076 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:18.347047    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/fe1c93d5-e51c-4bea-b395-b782a588e7e0-iptables-alerter-script\") pod \"iptables-alerter-xssj7\" (UID: \"fe1c93d5-e51c-4bea-b395-b782a588e7e0\") " pod="openshift-network-operator/iptables-alerter-xssj7"
Jul 31 23:46:18.347172 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:18.347084    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/8d650e41-5af5-435a-81ab-aaf3898a490c-host-kubelet\") pod \"ovnkube-node-m9px5\" (UID: \"8d650e41-5af5-435a-81ab-aaf3898a490c\") " pod="openshift-ovn-kubernetes/ovnkube-node-m9px5"
Jul 31 23:46:18.347172 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:18.347113    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/8d650e41-5af5-435a-81ab-aaf3898a490c-node-log\") pod \"ovnkube-node-m9px5\" (UID: \"8d650e41-5af5-435a-81ab-aaf3898a490c\") " pod="openshift-ovn-kubernetes/ovnkube-node-m9px5"
Jul 31 23:46:18.347172 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:18.347139    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8d650e41-5af5-435a-81ab-aaf3898a490c-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-m9px5\" (UID: \"8d650e41-5af5-435a-81ab-aaf3898a490c\") " pod="openshift-ovn-kubernetes/ovnkube-node-m9px5"
Jul 31 23:46:18.347172 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:18.347167    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/8d650e41-5af5-435a-81ab-aaf3898a490c-ovnkube-config\") pod \"ovnkube-node-m9px5\" (UID: \"8d650e41-5af5-435a-81ab-aaf3898a490c\") " pod="openshift-ovn-kubernetes/ovnkube-node-m9px5"
Jul 31 23:46:18.347408 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:18.347218    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/8d650e41-5af5-435a-81ab-aaf3898a490c-env-overrides\") pod \"ovnkube-node-m9px5\" (UID: \"8d650e41-5af5-435a-81ab-aaf3898a490c\") " pod="openshift-ovn-kubernetes/ovnkube-node-m9px5"
Jul 31 23:46:18.347408 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:18.347247    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/8d650e41-5af5-435a-81ab-aaf3898a490c-host-run-netns\") pod \"ovnkube-node-m9px5\" (UID: \"8d650e41-5af5-435a-81ab-aaf3898a490c\") " pod="openshift-ovn-kubernetes/ovnkube-node-m9px5"
Jul 31 23:46:18.347408 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:18.347273    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/51ec46bf-6903-4794-adc9-3f796e976899-serviceca\") pod \"node-ca-9hxzf\" (UID: \"51ec46bf-6903-4794-adc9-3f796e976899\") " pod="openshift-image-registry/node-ca-9hxzf"
Jul 31 23:46:18.347408 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:18.347300    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cwk2r\" (UniqueName: \"kubernetes.io/projected/51ec46bf-6903-4794-adc9-3f796e976899-kube-api-access-cwk2r\") pod \"node-ca-9hxzf\" (UID: \"51ec46bf-6903-4794-adc9-3f796e976899\") " pod="openshift-image-registry/node-ca-9hxzf"
Jul 31 23:46:18.347408 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:18.347326    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/8d650e41-5af5-435a-81ab-aaf3898a490c-host-slash\") pod \"ovnkube-node-m9px5\" (UID: \"8d650e41-5af5-435a-81ab-aaf3898a490c\") " pod="openshift-ovn-kubernetes/ovnkube-node-m9px5"
Jul 31 23:46:18.347408 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:18.347350    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/8d650e41-5af5-435a-81ab-aaf3898a490c-host-cni-bin\") pod \"ovnkube-node-m9px5\" (UID: \"8d650e41-5af5-435a-81ab-aaf3898a490c\") " pod="openshift-ovn-kubernetes/ovnkube-node-m9px5"
Jul 31 23:46:18.347408 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:18.347378    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ftdml\" (UniqueName: \"kubernetes.io/projected/8d650e41-5af5-435a-81ab-aaf3898a490c-kube-api-access-ftdml\") pod \"ovnkube-node-m9px5\" (UID: \"8d650e41-5af5-435a-81ab-aaf3898a490c\") " pod="openshift-ovn-kubernetes/ovnkube-node-m9px5"
Jul 31 23:46:18.347748 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:18.347428    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/fe1c93d5-e51c-4bea-b395-b782a588e7e0-host-slash\") pod \"iptables-alerter-xssj7\" (UID: \"fe1c93d5-e51c-4bea-b395-b782a588e7e0\") " pod="openshift-network-operator/iptables-alerter-xssj7"
Jul 31 23:46:18.347748 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:18.347462    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/8d650e41-5af5-435a-81ab-aaf3898a490c-log-socket\") pod \"ovnkube-node-m9px5\" (UID: \"8d650e41-5af5-435a-81ab-aaf3898a490c\") " pod="openshift-ovn-kubernetes/ovnkube-node-m9px5"
Jul 31 23:46:18.347748 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:18.347502    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/8d650e41-5af5-435a-81ab-aaf3898a490c-host-cni-netd\") pod \"ovnkube-node-m9px5\" (UID: \"8d650e41-5af5-435a-81ab-aaf3898a490c\") " pod="openshift-ovn-kubernetes/ovnkube-node-m9px5"
Jul 31 23:46:18.347748 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:18.347526    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/51ec46bf-6903-4794-adc9-3f796e976899-host\") pod \"node-ca-9hxzf\" (UID: \"51ec46bf-6903-4794-adc9-3f796e976899\") " pod="openshift-image-registry/node-ca-9hxzf"
Jul 31 23:46:18.347748 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:18.347558    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vq568\" (UniqueName: \"kubernetes.io/projected/fe1c93d5-e51c-4bea-b395-b782a588e7e0-kube-api-access-vq568\") pod \"iptables-alerter-xssj7\" (UID: \"fe1c93d5-e51c-4bea-b395-b782a588e7e0\") " pod="openshift-network-operator/iptables-alerter-xssj7"
Jul 31 23:46:18.347748 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:18.347621    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/8d650e41-5af5-435a-81ab-aaf3898a490c-run-openvswitch\") pod \"ovnkube-node-m9px5\" (UID: \"8d650e41-5af5-435a-81ab-aaf3898a490c\") " pod="openshift-ovn-kubernetes/ovnkube-node-m9px5"
Jul 31 23:46:18.347748 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:18.347662    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/8d650e41-5af5-435a-81ab-aaf3898a490c-run-ovn\") pod \"ovnkube-node-m9px5\" (UID: \"8d650e41-5af5-435a-81ab-aaf3898a490c\") " pod="openshift-ovn-kubernetes/ovnkube-node-m9px5"
Jul 31 23:46:18.347748 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:18.347704    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8d650e41-5af5-435a-81ab-aaf3898a490c-host-run-ovn-kubernetes\") pod \"ovnkube-node-m9px5\" (UID: \"8d650e41-5af5-435a-81ab-aaf3898a490c\") " pod="openshift-ovn-kubernetes/ovnkube-node-m9px5"
Jul 31 23:46:18.347748 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:18.347731    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/8d650e41-5af5-435a-81ab-aaf3898a490c-run-systemd\") pod \"ovnkube-node-m9px5\" (UID: \"8d650e41-5af5-435a-81ab-aaf3898a490c\") " pod="openshift-ovn-kubernetes/ovnkube-node-m9px5"
Jul 31 23:46:18.348178 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:18.347757    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qs7p7"
Jul 31 23:46:18.348178 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:18.347782    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/8d650e41-5af5-435a-81ab-aaf3898a490c-var-lib-openvswitch\") pod \"ovnkube-node-m9px5\" (UID: \"8d650e41-5af5-435a-81ab-aaf3898a490c\") " pod="openshift-ovn-kubernetes/ovnkube-node-m9px5"
Jul 31 23:46:18.348178 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:18.347806    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/8d650e41-5af5-435a-81ab-aaf3898a490c-etc-openvswitch\") pod \"ovnkube-node-m9px5\" (UID: \"8d650e41-5af5-435a-81ab-aaf3898a490c\") " pod="openshift-ovn-kubernetes/ovnkube-node-m9px5"
Jul 31 23:46:18.348178 ip-10-0-136-108 kubenswrapper[2578]: E0731 23:46:18.347830    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qs7p7" podUID="087b0a99-ae24-49e2-ba59-55651731fd60"
Jul 31 23:46:18.348178 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:18.347831    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/8d650e41-5af5-435a-81ab-aaf3898a490c-ovn-node-metrics-cert\") pod \"ovnkube-node-m9px5\" (UID: \"8d650e41-5af5-435a-81ab-aaf3898a490c\") " pod="openshift-ovn-kubernetes/ovnkube-node-m9px5"
Jul 31 23:46:18.348178 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:18.347760    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Jul 31 23:46:18.348178 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:18.347893    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Jul 31 23:46:18.348178 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:18.347903    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/8d650e41-5af5-435a-81ab-aaf3898a490c-ovnkube-script-lib\") pod \"ovnkube-node-m9px5\" (UID: \"8d650e41-5af5-435a-81ab-aaf3898a490c\") " pod="openshift-ovn-kubernetes/ovnkube-node-m9px5"
Jul 31 23:46:18.348178 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:18.347934    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/b32e6d42-e4e0-4113-b8ce-f5d5c5dc13f7-agent-certs\") pod \"konnectivity-agent-gttdm\" (UID: \"b32e6d42-e4e0-4113-b8ce-f5d5c5dc13f7\") " pod="kube-system/konnectivity-agent-gttdm"
Jul 31 23:46:18.348178 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:18.347958    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/8d650e41-5af5-435a-81ab-aaf3898a490c-systemd-units\") pod \"ovnkube-node-m9px5\" (UID: \"8d650e41-5af5-435a-81ab-aaf3898a490c\") " pod="openshift-ovn-kubernetes/ovnkube-node-m9px5"
Jul 31 23:46:18.348178 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:18.347988    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/b32e6d42-e4e0-4113-b8ce-f5d5c5dc13f7-konnectivity-ca\") pod \"konnectivity-agent-gttdm\" (UID: \"b32e6d42-e4e0-4113-b8ce-f5d5c5dc13f7\") " pod="kube-system/konnectivity-agent-gttdm"
Jul 31 23:46:18.348178 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:18.348033    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-znngt\""
Jul 31 23:46:18.349799 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:18.349776    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z94nv"
Jul 31 23:46:18.351446 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:18.351423    2578 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 31 23:46:18.351548 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:18.351431    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Jul 31 23:46:18.351548 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:18.351487    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-qfjxw\""
Jul 31 23:46:18.351548 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:18.351527    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Jul 31 23:46:18.351973 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:18.351946    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Jul 31 23:46:18.352845 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:18.352827    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-h8nq8"
Jul 31 23:46:18.354803 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:18.354646    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Jul 31 23:46:18.354803 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:18.354652    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Jul 31 23:46:18.354803 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:18.354721    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-nj6rj\""
Jul 31 23:46:18.356099 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:18.356064    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8d9rr"
Jul 31 23:46:18.356211 ip-10-0-136-108 kubenswrapper[2578]: E0731 23:46:18.356172    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8d9rr" podUID="0503a101-ddee-440a-ae03-e76fe30685e7"
Jul 31 23:46:18.356532 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:18.356511    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Jul 31 23:46:18.356765 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:18.356066    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Jul 31 23:46:18.357641 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:18.357398    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Jul 31 23:46:18.360683 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:18.360663    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-hlpzq"
Jul 31 23:46:18.363370 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:18.362934    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Jul 31 23:46:18.363370 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:18.363153    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-vrlhf\""
Jul 31 23:46:18.363370 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:18.363349    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xlv68"
Jul 31 23:46:18.363556 ip-10-0-136-108 kubenswrapper[2578]: E0731 23:46:18.363414    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xlv68" podUID="9d113c99-25bd-49a4-b8e2-f5c46f3ecc02"
Jul 31 23:46:18.380721 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:18.380697    2578 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-lhsk9"
Jul 31 23:46:18.389090 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:18.389067    2578 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-lhsk9"
Jul 31 23:46:18.412374 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:18.412348    2578 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 31 23:46:18.438178 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:18.438129    2578 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Jul 31 23:46:18.448581 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:18.448548    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/fe1c93d5-e51c-4bea-b395-b782a588e7e0-iptables-alerter-script\") pod \"iptables-alerter-xssj7\" (UID: \"fe1c93d5-e51c-4bea-b395-b782a588e7e0\") " pod="openshift-network-operator/iptables-alerter-xssj7"
Jul 31 23:46:18.448756 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:18.448593    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/8d650e41-5af5-435a-81ab-aaf3898a490c-node-log\") pod \"ovnkube-node-m9px5\" (UID: \"8d650e41-5af5-435a-81ab-aaf3898a490c\") " pod="openshift-ovn-kubernetes/ovnkube-node-m9px5"
Jul 31 23:46:18.448756 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:18.448629    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/66c4c4e9-d9f1-4a76-b937-cc3b1e16cee8-device-dir\") pod \"aws-ebs-csi-driver-node-z94nv\" (UID: \"66c4c4e9-d9f1-4a76-b937-cc3b1e16cee8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z94nv"
Jul 31 23:46:18.448756 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:18.448660    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c94dm\" (UniqueName: \"kubernetes.io/projected/66c4c4e9-d9f1-4a76-b937-cc3b1e16cee8-kube-api-access-c94dm\") pod \"aws-ebs-csi-driver-node-z94nv\" (UID: \"66c4c4e9-d9f1-4a76-b937-cc3b1e16cee8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z94nv"
Jul 31 23:46:18.448756 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:18.448688    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/9d113c99-25bd-49a4-b8e2-f5c46f3ecc02-original-pull-secret\") pod \"global-pull-secret-syncer-xlv68\" (UID: \"9d113c99-25bd-49a4-b8e2-f5c46f3ecc02\") " pod="kube-system/global-pull-secret-syncer-xlv68"
Jul 31 23:46:18.448756 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:18.448702    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/8d650e41-5af5-435a-81ab-aaf3898a490c-node-log\") pod \"ovnkube-node-m9px5\" (UID: \"8d650e41-5af5-435a-81ab-aaf3898a490c\") " pod="openshift-ovn-kubernetes/ovnkube-node-m9px5"
Jul 31 23:46:18.448756 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:18.448713    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/93fe11db-24ef-40d3-a042-7be6f62bd6f6-etc-sysconfig\") pod \"tuned-p7bwg\" (UID: \"93fe11db-24ef-40d3-a042-7be6f62bd6f6\") " pod="openshift-cluster-node-tuning-operator/tuned-p7bwg"
Jul 31 23:46:18.449069 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:18.448762    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/93fe11db-24ef-40d3-a042-7be6f62bd6f6-run\") pod \"tuned-p7bwg\" (UID: \"93fe11db-24ef-40d3-a042-7be6f62bd6f6\") " pod="openshift-cluster-node-tuning-operator/tuned-p7bwg"
Jul 31 23:46:18.449069 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:18.448802    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/a82bd3b0-bcad-4511-a3c4-4e4bdd5d6798-host-var-lib-cni-multus\") pod \"multus-hlpzq\" (UID: \"a82bd3b0-bcad-4511-a3c4-4e4bdd5d6798\") " pod="openshift-multus/multus-hlpzq"
Jul 31 23:46:18.449069 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:18.448831    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/93fe11db-24ef-40d3-a042-7be6f62bd6f6-etc-tuned\") pod \"tuned-p7bwg\" (UID: \"93fe11db-24ef-40d3-a042-7be6f62bd6f6\") " pod="openshift-cluster-node-tuning-operator/tuned-p7bwg"
Jul 31 23:46:18.449069 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:18.448857    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/a82bd3b0-bcad-4511-a3c4-4e4bdd5d6798-multus-daemon-config\") pod \"multus-hlpzq\" (UID: \"a82bd3b0-bcad-4511-a3c4-4e4bdd5d6798\") " pod="openshift-multus/multus-hlpzq"
Jul 31 23:46:18.449069 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:18.448881    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/a82bd3b0-bcad-4511-a3c4-4e4bdd5d6798-host-run-multus-certs\") pod \"multus-hlpzq\" (UID: \"a82bd3b0-bcad-4511-a3c4-4e4bdd5d6798\") " pod="openshift-multus/multus-hlpzq"
Jul 31 23:46:18.449069 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:18.448912    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t42nv\" (UniqueName: \"kubernetes.io/projected/dfb5011f-b591-4203-87ce-b1c6d6ad16ce-kube-api-access-t42nv\") pod \"multus-additional-cni-plugins-h8nq8\" (UID: \"dfb5011f-b591-4203-87ce-b1c6d6ad16ce\") " pod="openshift-multus/multus-additional-cni-plugins-h8nq8"
Jul 31 23:46:18.449069 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:18.448937    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/93fe11db-24ef-40d3-a042-7be6f62bd6f6-sys\") pod \"tuned-p7bwg\" (UID: \"93fe11db-24ef-40d3-a042-7be6f62bd6f6\") " pod="openshift-cluster-node-tuning-operator/tuned-p7bwg"
Jul 31 23:46:18.449069 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:18.448963    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-25jg7\" (UniqueName: \"kubernetes.io/projected/93fe11db-24ef-40d3-a042-7be6f62bd6f6-kube-api-access-25jg7\") pod \"tuned-p7bwg\" (UID: \"93fe11db-24ef-40d3-a042-7be6f62bd6f6\") " pod="openshift-cluster-node-tuning-operator/tuned-p7bwg"
Jul 31 23:46:18.449069 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:18.448993    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a82bd3b0-bcad-4511-a3c4-4e4bdd5d6798-host-var-lib-kubelet\") pod \"multus-hlpzq\" (UID: \"a82bd3b0-bcad-4511-a3c4-4e4bdd5d6798\") " pod="openshift-multus/multus-hlpzq"
Jul 31 23:46:18.449069 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:18.449024    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cwk2r\" (UniqueName: \"kubernetes.io/projected/51ec46bf-6903-4794-adc9-3f796e976899-kube-api-access-cwk2r\") pod \"node-ca-9hxzf\" (UID: \"51ec46bf-6903-4794-adc9-3f796e976899\") " pod="openshift-image-registry/node-ca-9hxzf"
Jul 31 23:46:18.449069 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:18.449051    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/8d650e41-5af5-435a-81ab-aaf3898a490c-host-slash\") pod \"ovnkube-node-m9px5\" (UID: \"8d650e41-5af5-435a-81ab-aaf3898a490c\") " pod="openshift-ovn-kubernetes/ovnkube-node-m9px5"
Jul 31 23:46:18.449069 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:18.449076    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/8d650e41-5af5-435a-81ab-aaf3898a490c-host-cni-bin\") pod \"ovnkube-node-m9px5\" (UID: \"8d650e41-5af5-435a-81ab-aaf3898a490c\") " pod="openshift-ovn-kubernetes/ovnkube-node-m9px5"
Jul 31 23:46:18.449688 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:18.449100    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/67d8b75c-284a-4bdb-a560-65d472fb1515-hosts-file\") pod \"node-resolver-wr2rk\" (UID: \"67d8b75c-284a-4bdb-a560-65d472fb1515\") " pod="openshift-dns/node-resolver-wr2rk"
Jul 31 23:46:18.449688 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:18.449127    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8rdb2\" (UniqueName: \"kubernetes.io/projected/67d8b75c-284a-4bdb-a560-65d472fb1515-kube-api-access-8rdb2\") pod \"node-resolver-wr2rk\" (UID: \"67d8b75c-284a-4bdb-a560-65d472fb1515\") " pod="openshift-dns/node-resolver-wr2rk"
Jul 31 23:46:18.449688 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:18.449161    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/dfb5011f-b591-4203-87ce-b1c6d6ad16ce-os-release\") pod \"multus-additional-cni-plugins-h8nq8\" (UID: \"dfb5011f-b591-4203-87ce-b1c6d6ad16ce\") " pod="openshift-multus/multus-additional-cni-plugins-h8nq8"
Jul 31 23:46:18.449688 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:18.449222    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/8d650e41-5af5-435a-81ab-aaf3898a490c-host-slash\") pod \"ovnkube-node-m9px5\" (UID: \"8d650e41-5af5-435a-81ab-aaf3898a490c\") " pod="openshift-ovn-kubernetes/ovnkube-node-m9px5"
Jul 31 23:46:18.449688 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:18.449239    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/9d113c99-25bd-49a4-b8e2-f5c46f3ecc02-dbus\") pod \"global-pull-secret-syncer-xlv68\" (UID: \"9d113c99-25bd-49a4-b8e2-f5c46f3ecc02\") " pod="kube-system/global-pull-secret-syncer-xlv68"
Jul 31 23:46:18.449688 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:18.449287    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/93fe11db-24ef-40d3-a042-7be6f62bd6f6-etc-sysctl-conf\") pod \"tuned-p7bwg\" (UID: \"93fe11db-24ef-40d3-a042-7be6f62bd6f6\") " pod="openshift-cluster-node-tuning-operator/tuned-p7bwg"
Jul 31 23:46:18.449688 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:18.449292    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/fe1c93d5-e51c-4bea-b395-b782a588e7e0-iptables-alerter-script\") pod \"iptables-alerter-xssj7\" (UID: \"fe1c93d5-e51c-4bea-b395-b782a588e7e0\") " pod="openshift-network-operator/iptables-alerter-xssj7"
Jul 31 23:46:18.449688 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:18.449317    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/fe1c93d5-e51c-4bea-b395-b782a588e7e0-host-slash\") pod \"iptables-alerter-xssj7\" (UID: \"fe1c93d5-e51c-4bea-b395-b782a588e7e0\") " pod="openshift-network-operator/iptables-alerter-xssj7"
Jul 31 23:46:18.449688 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:18.449331    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/8d650e41-5af5-435a-81ab-aaf3898a490c-host-cni-bin\") pod \"ovnkube-node-m9px5\" (UID: \"8d650e41-5af5-435a-81ab-aaf3898a490c\") " pod="openshift-ovn-kubernetes/ovnkube-node-m9px5"
Jul 31 23:46:18.449688 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:18.449345    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/8d650e41-5af5-435a-81ab-aaf3898a490c-log-socket\") pod \"ovnkube-node-m9px5\" (UID: \"8d650e41-5af5-435a-81ab-aaf3898a490c\") " pod="openshift-ovn-kubernetes/ovnkube-node-m9px5"
Jul 31 23:46:18.449688 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:18.449376    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/fe1c93d5-e51c-4bea-b395-b782a588e7e0-host-slash\") pod \"iptables-alerter-xssj7\" (UID: \"fe1c93d5-e51c-4bea-b395-b782a588e7e0\") " pod="openshift-network-operator/iptables-alerter-xssj7"
Jul 31 23:46:18.449688 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:18.449378    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/93fe11db-24ef-40d3-a042-7be6f62bd6f6-tmp\") pod \"tuned-p7bwg\" (UID: \"93fe11db-24ef-40d3-a042-7be6f62bd6f6\") " pod="openshift-cluster-node-tuning-operator/tuned-p7bwg"
Jul 31 23:46:18.449688 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:18.449402    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/8d650e41-5af5-435a-81ab-aaf3898a490c-log-socket\") pod \"ovnkube-node-m9px5\" (UID: \"8d650e41-5af5-435a-81ab-aaf3898a490c\") " pod="openshift-ovn-kubernetes/ovnkube-node-m9px5"
Jul 31 23:46:18.449688 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:18.449439    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-94n2k\" (UniqueName: \"kubernetes.io/projected/0503a101-ddee-440a-ae03-e76fe30685e7-kube-api-access-94n2k\") pod \"network-check-target-8d9rr\" (UID: \"0503a101-ddee-440a-ae03-e76fe30685e7\") " pod="openshift-network-diagnostics/network-check-target-8d9rr"
Jul 31 23:46:18.449688 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:18.449468    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a82bd3b0-bcad-4511-a3c4-4e4bdd5d6798-multus-cni-dir\") pod \"multus-hlpzq\" (UID: \"a82bd3b0-bcad-4511-a3c4-4e4bdd5d6798\") " pod="openshift-multus/multus-hlpzq"
Jul 31 23:46:18.449688 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:18.449495    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/a82bd3b0-bcad-4511-a3c4-4e4bdd5d6798-cnibin\") pod \"multus-hlpzq\" (UID: \"a82bd3b0-bcad-4511-a3c4-4e4bdd5d6798\") " pod="openshift-multus/multus-hlpzq"
Jul 31 23:46:18.449688 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:18.449537    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/a82bd3b0-bcad-4511-a3c4-4e4bdd5d6798-os-release\") pod \"multus-hlpzq\" (UID: \"a82bd3b0-bcad-4511-a3c4-4e4bdd5d6798\") " pod="openshift-multus/multus-hlpzq"
Jul 31 23:46:18.450441 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:18.449602    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/a82bd3b0-bcad-4511-a3c4-4e4bdd5d6798-multus-socket-dir-parent\") pod \"multus-hlpzq\" (UID: \"a82bd3b0-bcad-4511-a3c4-4e4bdd5d6798\") " pod="openshift-multus/multus-hlpzq"
Jul 31 23:46:18.450441 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:18.449639    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/51ec46bf-6903-4794-adc9-3f796e976899-host\") pod \"node-ca-9hxzf\" (UID: \"51ec46bf-6903-4794-adc9-3f796e976899\") " pod="openshift-image-registry/node-ca-9hxzf"
Jul 31 23:46:18.450441 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:18.449671    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vq568\" (UniqueName: \"kubernetes.io/projected/fe1c93d5-e51c-4bea-b395-b782a588e7e0-kube-api-access-vq568\") pod \"iptables-alerter-xssj7\" (UID: \"fe1c93d5-e51c-4bea-b395-b782a588e7e0\") " pod="openshift-network-operator/iptables-alerter-xssj7"
Jul 31 23:46:18.450441 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:18.449726    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/51ec46bf-6903-4794-adc9-3f796e976899-host\") pod \"node-ca-9hxzf\" (UID: \"51ec46bf-6903-4794-adc9-3f796e976899\") " pod="openshift-image-registry/node-ca-9hxzf"
Jul 31 23:46:18.450441 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:18.449726    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/8d650e41-5af5-435a-81ab-aaf3898a490c-run-ovn\") pod \"ovnkube-node-m9px5\" (UID: \"8d650e41-5af5-435a-81ab-aaf3898a490c\") " pod="openshift-ovn-kubernetes/ovnkube-node-m9px5"
Jul 31 23:46:18.450441 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:18.449767    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/8d650e41-5af5-435a-81ab-aaf3898a490c-run-ovn\") pod \"ovnkube-node-m9px5\" (UID: \"8d650e41-5af5-435a-81ab-aaf3898a490c\") " pod="openshift-ovn-kubernetes/ovnkube-node-m9px5"
Jul 31 23:46:18.450441 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:18.449873    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8d650e41-5af5-435a-81ab-aaf3898a490c-host-run-ovn-kubernetes\") pod \"ovnkube-node-m9px5\" (UID: \"8d650e41-5af5-435a-81ab-aaf3898a490c\") " pod="openshift-ovn-kubernetes/ovnkube-node-m9px5"
Jul 31 23:46:18.450441 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:18.449916    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/66c4c4e9-d9f1-4a76-b937-cc3b1e16cee8-etc-selinux\") pod \"aws-ebs-csi-driver-node-z94nv\" (UID: \"66c4c4e9-d9f1-4a76-b937-cc3b1e16cee8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z94nv"
Jul 31 23:46:18.450441 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:18.449952    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/8d650e41-5af5-435a-81ab-aaf3898a490c-run-systemd\") pod \"ovnkube-node-m9px5\" (UID: \"8d650e41-5af5-435a-81ab-aaf3898a490c\") " pod="openshift-ovn-kubernetes/ovnkube-node-m9px5"
Jul 31 23:46:18.450441 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:18.449983    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8d650e41-5af5-435a-81ab-aaf3898a490c-host-run-ovn-kubernetes\") pod \"ovnkube-node-m9px5\" (UID: \"8d650e41-5af5-435a-81ab-aaf3898a490c\") " pod="openshift-ovn-kubernetes/ovnkube-node-m9px5"
Jul 31 23:46:18.450441 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:18.450043    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/8d650e41-5af5-435a-81ab-aaf3898a490c-ovn-node-metrics-cert\") pod \"ovnkube-node-m9px5\" (UID: \"8d650e41-5af5-435a-81ab-aaf3898a490c\") " pod="openshift-ovn-kubernetes/ovnkube-node-m9px5"
Jul 31 23:46:18.450441 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:18.450080    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/8d650e41-5af5-435a-81ab-aaf3898a490c-run-systemd\") pod \"ovnkube-node-m9px5\" (UID: \"8d650e41-5af5-435a-81ab-aaf3898a490c\") " pod="openshift-ovn-kubernetes/ovnkube-node-m9px5"
Jul 31 23:46:18.450441 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:18.450116    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/8d650e41-5af5-435a-81ab-aaf3898a490c-ovnkube-script-lib\") pod \"ovnkube-node-m9px5\" (UID: \"8d650e41-5af5-435a-81ab-aaf3898a490c\") " pod="openshift-ovn-kubernetes/ovnkube-node-m9px5"
Jul 31 23:46:18.450441 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:18.450140    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/b32e6d42-e4e0-4113-b8ce-f5d5c5dc13f7-agent-certs\") pod \"konnectivity-agent-gttdm\" (UID: \"b32e6d42-e4e0-4113-b8ce-f5d5c5dc13f7\") " pod="kube-system/konnectivity-agent-gttdm"
Jul 31 23:46:18.450441 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:18.450168    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/66c4c4e9-d9f1-4a76-b937-cc3b1e16cee8-socket-dir\") pod \"aws-ebs-csi-driver-node-z94nv\" (UID: \"66c4c4e9-d9f1-4a76-b937-cc3b1e16cee8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z94nv"
Jul 31 23:46:18.450441 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:18.450226    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/dfb5011f-b591-4203-87ce-b1c6d6ad16ce-system-cni-dir\") pod \"multus-additional-cni-plugins-h8nq8\" (UID: \"dfb5011f-b591-4203-87ce-b1c6d6ad16ce\") " pod="openshift-multus/multus-additional-cni-plugins-h8nq8"
Jul 31 23:46:18.450441 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:18.450271    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/a82bd3b0-bcad-4511-a3c4-4e4bdd5d6798-cni-binary-copy\") pod \"multus-hlpzq\" (UID: \"a82bd3b0-bcad-4511-a3c4-4e4bdd5d6798\") " pod="openshift-multus/multus-hlpzq"
Jul 31 23:46:18.451033 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:18.450308    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/b32e6d42-e4e0-4113-b8ce-f5d5c5dc13f7-konnectivity-ca\") pod \"konnectivity-agent-gttdm\" (UID: \"b32e6d42-e4e0-4113-b8ce-f5d5c5dc13f7\") " pod="kube-system/konnectivity-agent-gttdm"
Jul 31 23:46:18.451033 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:18.450334    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/dfb5011f-b591-4203-87ce-b1c6d6ad16ce-cni-binary-copy\") pod \"multus-additional-cni-plugins-h8nq8\" (UID: \"dfb5011f-b591-4203-87ce-b1c6d6ad16ce\") " pod="openshift-multus/multus-additional-cni-plugins-h8nq8"
Jul 31 23:46:18.451033 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:18.450360    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/93fe11db-24ef-40d3-a042-7be6f62bd6f6-etc-modprobe-d\") pod \"tuned-p7bwg\" (UID: \"93fe11db-24ef-40d3-a042-7be6f62bd6f6\") " pod="openshift-cluster-node-tuning-operator/tuned-p7bwg"
Jul 31 23:46:18.451033 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:18.450384    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/a82bd3b0-bcad-4511-a3c4-4e4bdd5d6798-host-run-netns\") pod \"multus-hlpzq\" (UID: \"a82bd3b0-bcad-4511-a3c4-4e4bdd5d6798\") " pod="openshift-multus/multus-hlpzq"
Jul 31 23:46:18.451033 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:18.450422    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/a82bd3b0-bcad-4511-a3c4-4e4bdd5d6798-multus-conf-dir\") pod \"multus-hlpzq\" (UID: \"a82bd3b0-bcad-4511-a3c4-4e4bdd5d6798\") " pod="openshift-multus/multus-hlpzq"
Jul 31 23:46:18.451033 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:18.450470    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/8d650e41-5af5-435a-81ab-aaf3898a490c-host-kubelet\") pod \"ovnkube-node-m9px5\" (UID: \"8d650e41-5af5-435a-81ab-aaf3898a490c\") " pod="openshift-ovn-kubernetes/ovnkube-node-m9px5"
Jul 31 23:46:18.451033 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:18.450508    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8d650e41-5af5-435a-81ab-aaf3898a490c-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-m9px5\" (UID: \"8d650e41-5af5-435a-81ab-aaf3898a490c\") " pod="openshift-ovn-kubernetes/ovnkube-node-m9px5"
Jul 31 23:46:18.451033 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:18.450507    2578 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Jul 31 23:46:18.451033 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:18.450559    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/8d650e41-5af5-435a-81ab-aaf3898a490c-host-kubelet\") pod \"ovnkube-node-m9px5\" (UID: \"8d650e41-5af5-435a-81ab-aaf3898a490c\") " pod="openshift-ovn-kubernetes/ovnkube-node-m9px5"
Jul 31 23:46:18.451033 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:18.450591    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/8d650e41-5af5-435a-81ab-aaf3898a490c-ovnkube-config\") pod \"ovnkube-node-m9px5\" (UID: \"8d650e41-5af5-435a-81ab-aaf3898a490c\") " pod="openshift-ovn-kubernetes/ovnkube-node-m9px5"
Jul 31 23:46:18.451033 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:18.450650    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/8d650e41-5af5-435a-81ab-aaf3898a490c-env-overrides\") pod \"ovnkube-node-m9px5\" (UID: \"8d650e41-5af5-435a-81ab-aaf3898a490c\") " pod="openshift-ovn-kubernetes/ovnkube-node-m9px5"
Jul 31 23:46:18.451033 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:18.450712    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/a82bd3b0-bcad-4511-a3c4-4e4bdd5d6798-host-run-k8s-cni-cncf-io\") pod \"multus-hlpzq\" (UID: \"a82bd3b0-bcad-4511-a3c4-4e4bdd5d6798\") " pod="openshift-multus/multus-hlpzq"
Jul 31 23:46:18.451033 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:18.450741    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/a82bd3b0-bcad-4511-a3c4-4e4bdd5d6798-hostroot\") pod \"multus-hlpzq\" (UID: \"a82bd3b0-bcad-4511-a3c4-4e4bdd5d6798\") " pod="openshift-multus/multus-hlpzq"
Jul 31 23:46:18.451033 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:18.450764    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a82bd3b0-bcad-4511-a3c4-4e4bdd5d6798-etc-kubernetes\") pod \"multus-hlpzq\" (UID: \"a82bd3b0-bcad-4511-a3c4-4e4bdd5d6798\") " pod="openshift-multus/multus-hlpzq"
Jul 31 23:46:18.451033 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:18.450889    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/b32e6d42-e4e0-4113-b8ce-f5d5c5dc13f7-konnectivity-ca\") pod \"konnectivity-agent-gttdm\" (UID: \"b32e6d42-e4e0-4113-b8ce-f5d5c5dc13f7\") " pod="kube-system/konnectivity-agent-gttdm"
Jul 31 23:46:18.451033 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:18.450982    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8d650e41-5af5-435a-81ab-aaf3898a490c-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-m9px5\" (UID: \"8d650e41-5af5-435a-81ab-aaf3898a490c\") " pod="openshift-ovn-kubernetes/ovnkube-node-m9px5"
Jul 31 23:46:18.451756 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:18.451209    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/8d650e41-5af5-435a-81ab-aaf3898a490c-env-overrides\") pod \"ovnkube-node-m9px5\" (UID: \"8d650e41-5af5-435a-81ab-aaf3898a490c\") " pod="openshift-ovn-kubernetes/ovnkube-node-m9px5"
Jul 31 23:46:18.451756 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:18.451239    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/8d650e41-5af5-435a-81ab-aaf3898a490c-ovnkube-script-lib\") pod \"ovnkube-node-m9px5\" (UID: \"8d650e41-5af5-435a-81ab-aaf3898a490c\") " pod="openshift-ovn-kubernetes/ovnkube-node-m9px5"
Jul 31 23:46:18.451756 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:18.451308    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/8d650e41-5af5-435a-81ab-aaf3898a490c-host-run-netns\") pod \"ovnkube-node-m9px5\" (UID: \"8d650e41-5af5-435a-81ab-aaf3898a490c\") " pod="openshift-ovn-kubernetes/ovnkube-node-m9px5"
Jul 31 23:46:18.451756 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:18.451352    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/66c4c4e9-d9f1-4a76-b937-cc3b1e16cee8-kubelet-dir\") pod \"aws-ebs-csi-driver-node-z94nv\" (UID: \"66c4c4e9-d9f1-4a76-b937-cc3b1e16cee8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z94nv"
Jul 31 23:46:18.451756 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:18.451378    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/8d650e41-5af5-435a-81ab-aaf3898a490c-host-run-netns\") pod \"ovnkube-node-m9px5\" (UID: \"8d650e41-5af5-435a-81ab-aaf3898a490c\") " pod="openshift-ovn-kubernetes/ovnkube-node-m9px5"
Jul 31 23:46:18.451756 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:18.451432    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-76fwg\" (UniqueName: \"kubernetes.io/projected/a82bd3b0-bcad-4511-a3c4-4e4bdd5d6798-kube-api-access-76fwg\") pod \"multus-hlpzq\" (UID: \"a82bd3b0-bcad-4511-a3c4-4e4bdd5d6798\") " pod="openshift-multus/multus-hlpzq"
Jul 31 23:46:18.451756 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:18.451466    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/51ec46bf-6903-4794-adc9-3f796e976899-serviceca\") pod \"node-ca-9hxzf\" (UID: \"51ec46bf-6903-4794-adc9-3f796e976899\") " pod="openshift-image-registry/node-ca-9hxzf"
Jul 31 23:46:18.451756 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:18.451493    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/8d650e41-5af5-435a-81ab-aaf3898a490c-ovnkube-config\") pod \"ovnkube-node-m9px5\" (UID: \"8d650e41-5af5-435a-81ab-aaf3898a490c\") " pod="openshift-ovn-kubernetes/ovnkube-node-m9px5"
Jul 31 23:46:18.451756 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:18.451545    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/dfb5011f-b591-4203-87ce-b1c6d6ad16ce-cnibin\") pod \"multus-additional-cni-plugins-h8nq8\" (UID: \"dfb5011f-b591-4203-87ce-b1c6d6ad16ce\") " pod="openshift-multus/multus-additional-cni-plugins-h8nq8"
Jul 31 23:46:18.451756 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:18.451582    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/dfb5011f-b591-4203-87ce-b1c6d6ad16ce-tuning-conf-dir\") pod \"multus-additional-cni-plugins-h8nq8\" (UID: \"dfb5011f-b591-4203-87ce-b1c6d6ad16ce\") " pod="openshift-multus/multus-additional-cni-plugins-h8nq8"
Jul 31 23:46:18.451756 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:18.451612    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/93fe11db-24ef-40d3-a042-7be6f62bd6f6-etc-sysctl-d\") pod \"tuned-p7bwg\" (UID: \"93fe11db-24ef-40d3-a042-7be6f62bd6f6\") " pod="openshift-cluster-node-tuning-operator/tuned-p7bwg"
Jul 31 23:46:18.451756 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:18.451652    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/93fe11db-24ef-40d3-a042-7be6f62bd6f6-etc-systemd\") pod \"tuned-p7bwg\" (UID: \"93fe11db-24ef-40d3-a042-7be6f62bd6f6\") " pod="openshift-cluster-node-tuning-operator/tuned-p7bwg"
Jul 31 23:46:18.451756 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:18.451693    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ftdml\" (UniqueName: \"kubernetes.io/projected/8d650e41-5af5-435a-81ab-aaf3898a490c-kube-api-access-ftdml\") pod \"ovnkube-node-m9px5\" (UID: \"8d650e41-5af5-435a-81ab-aaf3898a490c\") " pod="openshift-ovn-kubernetes/ovnkube-node-m9px5"
Jul 31 23:46:18.451756 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:18.451729    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/67d8b75c-284a-4bdb-a560-65d472fb1515-tmp-dir\") pod \"node-resolver-wr2rk\" (UID: \"67d8b75c-284a-4bdb-a560-65d472fb1515\") " pod="openshift-dns/node-resolver-wr2rk"
Jul 31 23:46:18.451756 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:18.451767    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/087b0a99-ae24-49e2-ba59-55651731fd60-metrics-certs\") pod \"network-metrics-daemon-qs7p7\" (UID: \"087b0a99-ae24-49e2-ba59-55651731fd60\") " pod="openshift-multus/network-metrics-daemon-qs7p7"
Jul 31 23:46:18.452523 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:18.451790    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v2xtx\" (UniqueName: \"kubernetes.io/projected/087b0a99-ae24-49e2-ba59-55651731fd60-kube-api-access-v2xtx\") pod \"network-metrics-daemon-qs7p7\" (UID: \"087b0a99-ae24-49e2-ba59-55651731fd60\") " pod="openshift-multus/network-metrics-daemon-qs7p7"
Jul 31 23:46:18.452523 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:18.451812    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/66c4c4e9-d9f1-4a76-b937-cc3b1e16cee8-sys-fs\") pod \"aws-ebs-csi-driver-node-z94nv\" (UID: \"66c4c4e9-d9f1-4a76-b937-cc3b1e16cee8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z94nv"
Jul 31 23:46:18.452523 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:18.451837    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/93fe11db-24ef-40d3-a042-7be6f62bd6f6-etc-kubernetes\") pod \"tuned-p7bwg\" (UID: \"93fe11db-24ef-40d3-a042-7be6f62bd6f6\") " pod="openshift-cluster-node-tuning-operator/tuned-p7bwg"
Jul 31 23:46:18.452523 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:18.451903    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/51ec46bf-6903-4794-adc9-3f796e976899-serviceca\") pod \"node-ca-9hxzf\" (UID: \"51ec46bf-6903-4794-adc9-3f796e976899\") " pod="openshift-image-registry/node-ca-9hxzf"
Jul 31 23:46:18.452523 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:18.451964    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/9d113c99-25bd-49a4-b8e2-f5c46f3ecc02-kubelet-config\") pod \"global-pull-secret-syncer-xlv68\" (UID: \"9d113c99-25bd-49a4-b8e2-f5c46f3ecc02\") " pod="kube-system/global-pull-secret-syncer-xlv68"
Jul 31 23:46:18.452523 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:18.452017    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/8d650e41-5af5-435a-81ab-aaf3898a490c-host-cni-netd\") pod \"ovnkube-node-m9px5\" (UID: \"8d650e41-5af5-435a-81ab-aaf3898a490c\") " pod="openshift-ovn-kubernetes/ovnkube-node-m9px5"
Jul 31 23:46:18.452523 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:18.452035    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/8d650e41-5af5-435a-81ab-aaf3898a490c-run-openvswitch\") pod \"ovnkube-node-m9px5\" (UID: \"8d650e41-5af5-435a-81ab-aaf3898a490c\") " pod="openshift-ovn-kubernetes/ovnkube-node-m9px5"
Jul 31 23:46:18.452523 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:18.452052    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/dfb5011f-b591-4203-87ce-b1c6d6ad16ce-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-h8nq8\" (UID: \"dfb5011f-b591-4203-87ce-b1c6d6ad16ce\") " pod="openshift-multus/multus-additional-cni-plugins-h8nq8"
Jul 31 23:46:18.452523 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:18.452071    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/93fe11db-24ef-40d3-a042-7be6f62bd6f6-lib-modules\") pod \"tuned-p7bwg\" (UID: \"93fe11db-24ef-40d3-a042-7be6f62bd6f6\") " pod="openshift-cluster-node-tuning-operator/tuned-p7bwg"
Jul 31 23:46:18.452523 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:18.452091    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/93fe11db-24ef-40d3-a042-7be6f62bd6f6-var-lib-kubelet\") pod \"tuned-p7bwg\" (UID: \"93fe11db-24ef-40d3-a042-7be6f62bd6f6\") " pod="openshift-cluster-node-tuning-operator/tuned-p7bwg"
Jul 31 23:46:18.452523 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:18.452117    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a82bd3b0-bcad-4511-a3c4-4e4bdd5d6798-system-cni-dir\") pod \"multus-hlpzq\" (UID: \"a82bd3b0-bcad-4511-a3c4-4e4bdd5d6798\") " pod="openshift-multus/multus-hlpzq"
Jul 31 23:46:18.452523 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:18.452143    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/8d650e41-5af5-435a-81ab-aaf3898a490c-var-lib-openvswitch\") pod \"ovnkube-node-m9px5\" (UID: \"8d650e41-5af5-435a-81ab-aaf3898a490c\") " pod="openshift-ovn-kubernetes/ovnkube-node-m9px5"
Jul 31 23:46:18.452523 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:18.452208    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/8d650e41-5af5-435a-81ab-aaf3898a490c-etc-openvswitch\") pod \"ovnkube-node-m9px5\" (UID: \"8d650e41-5af5-435a-81ab-aaf3898a490c\") " pod="openshift-ovn-kubernetes/ovnkube-node-m9px5"
Jul 31 23:46:18.452523 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:18.452245    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/dfb5011f-b591-4203-87ce-b1c6d6ad16ce-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-h8nq8\" (UID: \"dfb5011f-b591-4203-87ce-b1c6d6ad16ce\") " pod="openshift-multus/multus-additional-cni-plugins-h8nq8"
Jul 31 23:46:18.452523 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:18.452281    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/8d650e41-5af5-435a-81ab-aaf3898a490c-var-lib-openvswitch\") pod \"ovnkube-node-m9px5\" (UID: \"8d650e41-5af5-435a-81ab-aaf3898a490c\") " pod="openshift-ovn-kubernetes/ovnkube-node-m9px5"
Jul 31 23:46:18.452523 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:18.452296    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/8d650e41-5af5-435a-81ab-aaf3898a490c-systemd-units\") pod \"ovnkube-node-m9px5\" (UID: \"8d650e41-5af5-435a-81ab-aaf3898a490c\") " pod="openshift-ovn-kubernetes/ovnkube-node-m9px5"
Jul 31 23:46:18.453283 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:18.452321    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/66c4c4e9-d9f1-4a76-b937-cc3b1e16cee8-registration-dir\") pod \"aws-ebs-csi-driver-node-z94nv\" (UID: \"66c4c4e9-d9f1-4a76-b937-cc3b1e16cee8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z94nv"
Jul 31 23:46:18.453283 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:18.452352    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/93fe11db-24ef-40d3-a042-7be6f62bd6f6-host\") pod \"tuned-p7bwg\" (UID: \"93fe11db-24ef-40d3-a042-7be6f62bd6f6\") " pod="openshift-cluster-node-tuning-operator/tuned-p7bwg"
Jul 31 23:46:18.453283 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:18.452377    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/a82bd3b0-bcad-4511-a3c4-4e4bdd5d6798-host-var-lib-cni-bin\") pod \"multus-hlpzq\" (UID: \"a82bd3b0-bcad-4511-a3c4-4e4bdd5d6798\") " pod="openshift-multus/multus-hlpzq"
Jul 31 23:46:18.453283 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:18.452461    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/8d650e41-5af5-435a-81ab-aaf3898a490c-host-cni-netd\") pod \"ovnkube-node-m9px5\" (UID: \"8d650e41-5af5-435a-81ab-aaf3898a490c\") " pod="openshift-ovn-kubernetes/ovnkube-node-m9px5"
Jul 31 23:46:18.453283 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:18.452481    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/8d650e41-5af5-435a-81ab-aaf3898a490c-etc-openvswitch\") pod \"ovnkube-node-m9px5\" (UID: \"8d650e41-5af5-435a-81ab-aaf3898a490c\") " pod="openshift-ovn-kubernetes/ovnkube-node-m9px5"
Jul 31 23:46:18.453283 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:18.452488    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/8d650e41-5af5-435a-81ab-aaf3898a490c-systemd-units\") pod \"ovnkube-node-m9px5\" (UID: \"8d650e41-5af5-435a-81ab-aaf3898a490c\") " pod="openshift-ovn-kubernetes/ovnkube-node-m9px5"
Jul 31 23:46:18.453283 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:18.452506    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/8d650e41-5af5-435a-81ab-aaf3898a490c-run-openvswitch\") pod \"ovnkube-node-m9px5\" (UID: \"8d650e41-5af5-435a-81ab-aaf3898a490c\") " pod="openshift-ovn-kubernetes/ovnkube-node-m9px5"
Jul 31 23:46:18.455688 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:18.455661    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/b32e6d42-e4e0-4113-b8ce-f5d5c5dc13f7-agent-certs\") pod \"konnectivity-agent-gttdm\" (UID: \"b32e6d42-e4e0-4113-b8ce-f5d5c5dc13f7\") " pod="kube-system/konnectivity-agent-gttdm"
Jul 31 23:46:18.456428 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:18.456401    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/8d650e41-5af5-435a-81ab-aaf3898a490c-ovn-node-metrics-cert\") pod \"ovnkube-node-m9px5\" (UID: \"8d650e41-5af5-435a-81ab-aaf3898a490c\") " pod="openshift-ovn-kubernetes/ovnkube-node-m9px5"
Jul 31 23:46:18.457954 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:18.457918    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vq568\" (UniqueName: \"kubernetes.io/projected/fe1c93d5-e51c-4bea-b395-b782a588e7e0-kube-api-access-vq568\") pod \"iptables-alerter-xssj7\" (UID: \"fe1c93d5-e51c-4bea-b395-b782a588e7e0\") " pod="openshift-network-operator/iptables-alerter-xssj7"
Jul 31 23:46:18.458932 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:18.458908    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cwk2r\" (UniqueName: \"kubernetes.io/projected/51ec46bf-6903-4794-adc9-3f796e976899-kube-api-access-cwk2r\") pod \"node-ca-9hxzf\" (UID: \"51ec46bf-6903-4794-adc9-3f796e976899\") " pod="openshift-image-registry/node-ca-9hxzf"
Jul 31 23:46:18.460007 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:18.459982    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ftdml\" (UniqueName: \"kubernetes.io/projected/8d650e41-5af5-435a-81ab-aaf3898a490c-kube-api-access-ftdml\") pod \"ovnkube-node-m9px5\" (UID: \"8d650e41-5af5-435a-81ab-aaf3898a490c\") " pod="openshift-ovn-kubernetes/ovnkube-node-m9px5"
Jul 31 23:46:18.489374 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:18.489317    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-108.ec2.internal" event={"ID":"739886fda1f37e5f1a9572bed84e6cb4","Type":"ContainerStarted","Data":"3d5536fad15f0660c7b4cb51f341a50aeea3a7870f5d7cdfdb16691cd42c231a"}
Jul 31 23:46:18.491150 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:18.491117    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-108.ec2.internal" event={"ID":"cc577a92ce0dd873599a996e54a72791","Type":"ContainerStarted","Data":"642ce8073770e2e16ebc1ce259523e225d2bbd63dece78f533c7d452a5dca311"}
Jul 31 23:46:18.553543 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:18.553504    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/66c4c4e9-d9f1-4a76-b937-cc3b1e16cee8-registration-dir\") pod \"aws-ebs-csi-driver-node-z94nv\" (UID: \"66c4c4e9-d9f1-4a76-b937-cc3b1e16cee8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z94nv"
Jul 31 23:46:18.553735 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:18.553551    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/93fe11db-24ef-40d3-a042-7be6f62bd6f6-host\") pod \"tuned-p7bwg\" (UID: \"93fe11db-24ef-40d3-a042-7be6f62bd6f6\") " pod="openshift-cluster-node-tuning-operator/tuned-p7bwg"
Jul 31 23:46:18.553735 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:18.553581    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/a82bd3b0-bcad-4511-a3c4-4e4bdd5d6798-host-var-lib-cni-bin\") pod \"multus-hlpzq\" (UID: \"a82bd3b0-bcad-4511-a3c4-4e4bdd5d6798\") " pod="openshift-multus/multus-hlpzq"
Jul 31 23:46:18.553735 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:18.553607    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/66c4c4e9-d9f1-4a76-b937-cc3b1e16cee8-device-dir\") pod \"aws-ebs-csi-driver-node-z94nv\" (UID: \"66c4c4e9-d9f1-4a76-b937-cc3b1e16cee8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z94nv"
Jul 31 23:46:18.553735 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:18.553638    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/66c4c4e9-d9f1-4a76-b937-cc3b1e16cee8-registration-dir\") pod \"aws-ebs-csi-driver-node-z94nv\" (UID: \"66c4c4e9-d9f1-4a76-b937-cc3b1e16cee8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z94nv"
Jul 31 23:46:18.553735 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:18.553687    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/93fe11db-24ef-40d3-a042-7be6f62bd6f6-host\") pod \"tuned-p7bwg\" (UID: \"93fe11db-24ef-40d3-a042-7be6f62bd6f6\") " pod="openshift-cluster-node-tuning-operator/tuned-p7bwg"
Jul 31 23:46:18.553735 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:18.553733    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c94dm\" (UniqueName: \"kubernetes.io/projected/66c4c4e9-d9f1-4a76-b937-cc3b1e16cee8-kube-api-access-c94dm\") pod \"aws-ebs-csi-driver-node-z94nv\" (UID: \"66c4c4e9-d9f1-4a76-b937-cc3b1e16cee8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z94nv"
Jul 31 23:46:18.554128 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:18.553757    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/9d113c99-25bd-49a4-b8e2-f5c46f3ecc02-original-pull-secret\") pod \"global-pull-secret-syncer-xlv68\" (UID: \"9d113c99-25bd-49a4-b8e2-f5c46f3ecc02\") " pod="kube-system/global-pull-secret-syncer-xlv68"
Jul 31 23:46:18.554128 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:18.553773    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/93fe11db-24ef-40d3-a042-7be6f62bd6f6-etc-sysconfig\") pod \"tuned-p7bwg\" (UID: \"93fe11db-24ef-40d3-a042-7be6f62bd6f6\") " pod="openshift-cluster-node-tuning-operator/tuned-p7bwg"
Jul 31 23:46:18.554128 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:18.553780    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/66c4c4e9-d9f1-4a76-b937-cc3b1e16cee8-device-dir\") pod \"aws-ebs-csi-driver-node-z94nv\" (UID: \"66c4c4e9-d9f1-4a76-b937-cc3b1e16cee8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z94nv"
Jul 31 23:46:18.554128 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:18.553798    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/93fe11db-24ef-40d3-a042-7be6f62bd6f6-run\") pod \"tuned-p7bwg\" (UID: \"93fe11db-24ef-40d3-a042-7be6f62bd6f6\") " pod="openshift-cluster-node-tuning-operator/tuned-p7bwg"
Jul 31 23:46:18.554128 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:18.553836    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/a82bd3b0-bcad-4511-a3c4-4e4bdd5d6798-host-var-lib-cni-multus\") pod \"multus-hlpzq\" (UID: \"a82bd3b0-bcad-4511-a3c4-4e4bdd5d6798\") " pod="openshift-multus/multus-hlpzq"
Jul 31 23:46:18.554128 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:18.553868    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/93fe11db-24ef-40d3-a042-7be6f62bd6f6-etc-tuned\") pod \"tuned-p7bwg\" (UID: \"93fe11db-24ef-40d3-a042-7be6f62bd6f6\") " pod="openshift-cluster-node-tuning-operator/tuned-p7bwg"
Jul 31 23:46:18.554128 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:18.553897    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/a82bd3b0-bcad-4511-a3c4-4e4bdd5d6798-multus-daemon-config\") pod \"multus-hlpzq\" (UID: \"a82bd3b0-bcad-4511-a3c4-4e4bdd5d6798\") " pod="openshift-multus/multus-hlpzq"
Jul 31 23:46:18.554128 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:18.553914    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/93fe11db-24ef-40d3-a042-7be6f62bd6f6-run\") pod \"tuned-p7bwg\" (UID: \"93fe11db-24ef-40d3-a042-7be6f62bd6f6\") " pod="openshift-cluster-node-tuning-operator/tuned-p7bwg"
Jul 31 23:46:18.554128 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:18.553927    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/a82bd3b0-bcad-4511-a3c4-4e4bdd5d6798-host-run-multus-certs\") pod \"multus-hlpzq\" (UID: \"a82bd3b0-bcad-4511-a3c4-4e4bdd5d6798\") " pod="openshift-multus/multus-hlpzq"
Jul 31 23:46:18.554128 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:18.553969    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t42nv\" (UniqueName: \"kubernetes.io/projected/dfb5011f-b591-4203-87ce-b1c6d6ad16ce-kube-api-access-t42nv\") pod \"multus-additional-cni-plugins-h8nq8\" (UID: \"dfb5011f-b591-4203-87ce-b1c6d6ad16ce\") " pod="openshift-multus/multus-additional-cni-plugins-h8nq8"
Jul 31 23:46:18.554128 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:18.553999    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/93fe11db-24ef-40d3-a042-7be6f62bd6f6-sys\") pod \"tuned-p7bwg\" (UID: \"93fe11db-24ef-40d3-a042-7be6f62bd6f6\") " pod="openshift-cluster-node-tuning-operator/tuned-p7bwg"
Jul 31 23:46:18.554128 ip-10-0-136-108 kubenswrapper[2578]: E0731 23:46:18.554034    2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 31 23:46:18.554128 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:18.554066    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/93fe11db-24ef-40d3-a042-7be6f62bd6f6-sys\") pod \"tuned-p7bwg\" (UID: \"93fe11db-24ef-40d3-a042-7be6f62bd6f6\") " pod="openshift-cluster-node-tuning-operator/tuned-p7bwg"
Jul 31 23:46:18.554128 ip-10-0-136-108 kubenswrapper[2578]: E0731 23:46:18.554114    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9d113c99-25bd-49a4-b8e2-f5c46f3ecc02-original-pull-secret podName:9d113c99-25bd-49a4-b8e2-f5c46f3ecc02 nodeName:}" failed. No retries permitted until 2026-07-31 23:46:19.054082108 +0000 UTC m=+3.021832851 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/9d113c99-25bd-49a4-b8e2-f5c46f3ecc02-original-pull-secret") pod "global-pull-secret-syncer-xlv68" (UID: "9d113c99-25bd-49a4-b8e2-f5c46f3ecc02") : object "kube-system"/"original-pull-secret" not registered
Jul 31 23:46:18.554853 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:18.554164    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/93fe11db-24ef-40d3-a042-7be6f62bd6f6-etc-sysconfig\") pod \"tuned-p7bwg\" (UID: \"93fe11db-24ef-40d3-a042-7be6f62bd6f6\") " pod="openshift-cluster-node-tuning-operator/tuned-p7bwg"
Jul 31 23:46:18.554853 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:18.554742    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/a82bd3b0-bcad-4511-a3c4-4e4bdd5d6798-multus-daemon-config\") pod \"multus-hlpzq\" (UID: \"a82bd3b0-bcad-4511-a3c4-4e4bdd5d6798\") " pod="openshift-multus/multus-hlpzq"
Jul 31 23:46:18.554853 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:18.554758    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/a82bd3b0-bcad-4511-a3c4-4e4bdd5d6798-host-var-lib-cni-bin\") pod \"multus-hlpzq\" (UID: \"a82bd3b0-bcad-4511-a3c4-4e4bdd5d6798\") " pod="openshift-multus/multus-hlpzq"
Jul 31 23:46:18.554853 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:18.554824    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/a82bd3b0-bcad-4511-a3c4-4e4bdd5d6798-host-run-multus-certs\") pod \"multus-hlpzq\" (UID: \"a82bd3b0-bcad-4511-a3c4-4e4bdd5d6798\") " pod="openshift-multus/multus-hlpzq"
Jul 31 23:46:18.554853 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:18.554839    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/a82bd3b0-bcad-4511-a3c4-4e4bdd5d6798-host-var-lib-cni-multus\") pod \"multus-hlpzq\" (UID: \"a82bd3b0-bcad-4511-a3c4-4e4bdd5d6798\") " pod="openshift-multus/multus-hlpzq"
Jul 31 23:46:18.554853 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:18.554855    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-25jg7\" (UniqueName: \"kubernetes.io/projected/93fe11db-24ef-40d3-a042-7be6f62bd6f6-kube-api-access-25jg7\") pod \"tuned-p7bwg\" (UID: \"93fe11db-24ef-40d3-a042-7be6f62bd6f6\") " pod="openshift-cluster-node-tuning-operator/tuned-p7bwg"
Jul 31 23:46:18.555171 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:18.554887    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a82bd3b0-bcad-4511-a3c4-4e4bdd5d6798-host-var-lib-kubelet\") pod \"multus-hlpzq\" (UID: \"a82bd3b0-bcad-4511-a3c4-4e4bdd5d6798\") " pod="openshift-multus/multus-hlpzq"
Jul 31 23:46:18.555171 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:18.554916    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/67d8b75c-284a-4bdb-a560-65d472fb1515-hosts-file\") pod \"node-resolver-wr2rk\" (UID: \"67d8b75c-284a-4bdb-a560-65d472fb1515\") " pod="openshift-dns/node-resolver-wr2rk"
Jul 31 23:46:18.555171 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:18.554959    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8rdb2\" (UniqueName: \"kubernetes.io/projected/67d8b75c-284a-4bdb-a560-65d472fb1515-kube-api-access-8rdb2\") pod \"node-resolver-wr2rk\" (UID: \"67d8b75c-284a-4bdb-a560-65d472fb1515\") " pod="openshift-dns/node-resolver-wr2rk"
Jul 31 23:46:18.555171 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:18.554987    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/dfb5011f-b591-4203-87ce-b1c6d6ad16ce-os-release\") pod \"multus-additional-cni-plugins-h8nq8\" (UID: \"dfb5011f-b591-4203-87ce-b1c6d6ad16ce\") " pod="openshift-multus/multus-additional-cni-plugins-h8nq8"
Jul 31 23:46:18.555171 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:18.555000    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a82bd3b0-bcad-4511-a3c4-4e4bdd5d6798-host-var-lib-kubelet\") pod \"multus-hlpzq\" (UID: \"a82bd3b0-bcad-4511-a3c4-4e4bdd5d6798\") " pod="openshift-multus/multus-hlpzq"
Jul 31 23:46:18.555171 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:18.555012    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/9d113c99-25bd-49a4-b8e2-f5c46f3ecc02-dbus\") pod \"global-pull-secret-syncer-xlv68\" (UID: \"9d113c99-25bd-49a4-b8e2-f5c46f3ecc02\") " pod="kube-system/global-pull-secret-syncer-xlv68"
Jul 31 23:46:18.555171 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:18.555038    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/93fe11db-24ef-40d3-a042-7be6f62bd6f6-etc-sysctl-conf\") pod \"tuned-p7bwg\" (UID: \"93fe11db-24ef-40d3-a042-7be6f62bd6f6\") " pod="openshift-cluster-node-tuning-operator/tuned-p7bwg"
Jul 31 23:46:18.555171 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:18.555064    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/93fe11db-24ef-40d3-a042-7be6f62bd6f6-tmp\") pod \"tuned-p7bwg\" (UID: \"93fe11db-24ef-40d3-a042-7be6f62bd6f6\") " pod="openshift-cluster-node-tuning-operator/tuned-p7bwg"
Jul 31 23:46:18.555171 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:18.555089    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-94n2k\" (UniqueName: \"kubernetes.io/projected/0503a101-ddee-440a-ae03-e76fe30685e7-kube-api-access-94n2k\") pod \"network-check-target-8d9rr\" (UID: \"0503a101-ddee-440a-ae03-e76fe30685e7\") " pod="openshift-network-diagnostics/network-check-target-8d9rr"
Jul 31 23:46:18.555171 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:18.555115    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a82bd3b0-bcad-4511-a3c4-4e4bdd5d6798-multus-cni-dir\") pod \"multus-hlpzq\" (UID: \"a82bd3b0-bcad-4511-a3c4-4e4bdd5d6798\") " pod="openshift-multus/multus-hlpzq"
Jul 31 23:46:18.555171 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:18.555139    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/a82bd3b0-bcad-4511-a3c4-4e4bdd5d6798-cnibin\") pod \"multus-hlpzq\" (UID: \"a82bd3b0-bcad-4511-a3c4-4e4bdd5d6798\") " pod="openshift-multus/multus-hlpzq"
Jul 31 23:46:18.555171 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:18.555140    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/dfb5011f-b591-4203-87ce-b1c6d6ad16ce-os-release\") pod \"multus-additional-cni-plugins-h8nq8\" (UID: \"dfb5011f-b591-4203-87ce-b1c6d6ad16ce\") " pod="openshift-multus/multus-additional-cni-plugins-h8nq8"
Jul 31 23:46:18.555770 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:18.555185    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/a82bd3b0-bcad-4511-a3c4-4e4bdd5d6798-os-release\") pod \"multus-hlpzq\" (UID: \"a82bd3b0-bcad-4511-a3c4-4e4bdd5d6798\") " pod="openshift-multus/multus-hlpzq"
Jul 31 23:46:18.555770 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:18.555220    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/67d8b75c-284a-4bdb-a560-65d472fb1515-hosts-file\") pod \"node-resolver-wr2rk\" (UID: \"67d8b75c-284a-4bdb-a560-65d472fb1515\") " pod="openshift-dns/node-resolver-wr2rk"
Jul 31 23:46:18.555770 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:18.555225    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/a82bd3b0-bcad-4511-a3c4-4e4bdd5d6798-multus-socket-dir-parent\") pod \"multus-hlpzq\" (UID: \"a82bd3b0-bcad-4511-a3c4-4e4bdd5d6798\") " pod="openshift-multus/multus-hlpzq"
Jul 31 23:46:18.555770 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:18.555268    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/66c4c4e9-d9f1-4a76-b937-cc3b1e16cee8-etc-selinux\") pod \"aws-ebs-csi-driver-node-z94nv\" (UID: \"66c4c4e9-d9f1-4a76-b937-cc3b1e16cee8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z94nv"
Jul 31 23:46:18.555770 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:18.555297    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/66c4c4e9-d9f1-4a76-b937-cc3b1e16cee8-socket-dir\") pod \"aws-ebs-csi-driver-node-z94nv\" (UID: \"66c4c4e9-d9f1-4a76-b937-cc3b1e16cee8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z94nv"
Jul 31 23:46:18.555770 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:18.555303    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/a82bd3b0-bcad-4511-a3c4-4e4bdd5d6798-cnibin\") pod \"multus-hlpzq\" (UID: \"a82bd3b0-bcad-4511-a3c4-4e4bdd5d6798\") " pod="openshift-multus/multus-hlpzq"
Jul 31 23:46:18.555770 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:18.555322    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/dfb5011f-b591-4203-87ce-b1c6d6ad16ce-system-cni-dir\") pod \"multus-additional-cni-plugins-h8nq8\" (UID: \"dfb5011f-b591-4203-87ce-b1c6d6ad16ce\") " pod="openshift-multus/multus-additional-cni-plugins-h8nq8"
Jul 31 23:46:18.555770 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:18.555348    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/a82bd3b0-bcad-4511-a3c4-4e4bdd5d6798-cni-binary-copy\") pod \"multus-hlpzq\" (UID: \"a82bd3b0-bcad-4511-a3c4-4e4bdd5d6798\") " pod="openshift-multus/multus-hlpzq"
Jul 31 23:46:18.555770 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:18.555359    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/93fe11db-24ef-40d3-a042-7be6f62bd6f6-etc-sysctl-conf\") pod \"tuned-p7bwg\" (UID: \"93fe11db-24ef-40d3-a042-7be6f62bd6f6\") " pod="openshift-cluster-node-tuning-operator/tuned-p7bwg"
Jul 31 23:46:18.555770 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:18.555376    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/dfb5011f-b591-4203-87ce-b1c6d6ad16ce-cni-binary-copy\") pod \"multus-additional-cni-plugins-h8nq8\" (UID: \"dfb5011f-b591-4203-87ce-b1c6d6ad16ce\") " pod="openshift-multus/multus-additional-cni-plugins-h8nq8"
Jul 31 23:46:18.555770 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:18.555460    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/a82bd3b0-bcad-4511-a3c4-4e4bdd5d6798-multus-socket-dir-parent\") pod \"multus-hlpzq\" (UID: \"a82bd3b0-bcad-4511-a3c4-4e4bdd5d6798\") " pod="openshift-multus/multus-hlpzq"
Jul 31 23:46:18.555770 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:18.555606    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a82bd3b0-bcad-4511-a3c4-4e4bdd5d6798-multus-cni-dir\") pod \"multus-hlpzq\" (UID: \"a82bd3b0-bcad-4511-a3c4-4e4bdd5d6798\") " pod="openshift-multus/multus-hlpzq"
Jul 31 23:46:18.555770 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:18.555663    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/93fe11db-24ef-40d3-a042-7be6f62bd6f6-etc-modprobe-d\") pod \"tuned-p7bwg\" (UID: \"93fe11db-24ef-40d3-a042-7be6f62bd6f6\") " pod="openshift-cluster-node-tuning-operator/tuned-p7bwg"
Jul 31 23:46:18.555770 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:18.555693    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/a82bd3b0-bcad-4511-a3c4-4e4bdd5d6798-host-run-netns\") pod \"multus-hlpzq\" (UID: \"a82bd3b0-bcad-4511-a3c4-4e4bdd5d6798\") " pod="openshift-multus/multus-hlpzq"
Jul 31 23:46:18.555770 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:18.555725    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/a82bd3b0-bcad-4511-a3c4-4e4bdd5d6798-multus-conf-dir\") pod \"multus-hlpzq\" (UID: \"a82bd3b0-bcad-4511-a3c4-4e4bdd5d6798\") " pod="openshift-multus/multus-hlpzq"
Jul 31 23:46:18.555770 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:18.555759    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/a82bd3b0-bcad-4511-a3c4-4e4bdd5d6798-host-run-k8s-cni-cncf-io\") pod \"multus-hlpzq\" (UID: \"a82bd3b0-bcad-4511-a3c4-4e4bdd5d6798\") " pod="openshift-multus/multus-hlpzq"
Jul 31 23:46:18.556646 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:18.555785    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/a82bd3b0-bcad-4511-a3c4-4e4bdd5d6798-hostroot\") pod \"multus-hlpzq\" (UID: \"a82bd3b0-bcad-4511-a3c4-4e4bdd5d6798\") " pod="openshift-multus/multus-hlpzq"
Jul 31 23:46:18.556646 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:18.555803    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/a82bd3b0-bcad-4511-a3c4-4e4bdd5d6798-os-release\") pod \"multus-hlpzq\" (UID: \"a82bd3b0-bcad-4511-a3c4-4e4bdd5d6798\") " pod="openshift-multus/multus-hlpzq"
Jul 31 23:46:18.556646 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:18.555813    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a82bd3b0-bcad-4511-a3c4-4e4bdd5d6798-etc-kubernetes\") pod \"multus-hlpzq\" (UID: \"a82bd3b0-bcad-4511-a3c4-4e4bdd5d6798\") " pod="openshift-multus/multus-hlpzq"
Jul 31 23:46:18.556646 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:18.555856    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/66c4c4e9-d9f1-4a76-b937-cc3b1e16cee8-kubelet-dir\") pod \"aws-ebs-csi-driver-node-z94nv\" (UID: \"66c4c4e9-d9f1-4a76-b937-cc3b1e16cee8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z94nv"
Jul 31 23:46:18.556646 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:18.555864    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a82bd3b0-bcad-4511-a3c4-4e4bdd5d6798-etc-kubernetes\") pod \"multus-hlpzq\" (UID: \"a82bd3b0-bcad-4511-a3c4-4e4bdd5d6798\") " pod="openshift-multus/multus-hlpzq"
Jul 31 23:46:18.556646 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:18.555886    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-76fwg\" (UniqueName: \"kubernetes.io/projected/a82bd3b0-bcad-4511-a3c4-4e4bdd5d6798-kube-api-access-76fwg\") pod \"multus-hlpzq\" (UID: \"a82bd3b0-bcad-4511-a3c4-4e4bdd5d6798\") " pod="openshift-multus/multus-hlpzq"
Jul 31 23:46:18.556646 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:18.555915    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/dfb5011f-b591-4203-87ce-b1c6d6ad16ce-cnibin\") pod \"multus-additional-cni-plugins-h8nq8\" (UID: \"dfb5011f-b591-4203-87ce-b1c6d6ad16ce\") " pod="openshift-multus/multus-additional-cni-plugins-h8nq8"
Jul 31 23:46:18.556646 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:18.555919    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/a82bd3b0-bcad-4511-a3c4-4e4bdd5d6798-cni-binary-copy\") pod \"multus-hlpzq\" (UID: \"a82bd3b0-bcad-4511-a3c4-4e4bdd5d6798\") " pod="openshift-multus/multus-hlpzq"
Jul 31 23:46:18.556646 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:18.555940    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/dfb5011f-b591-4203-87ce-b1c6d6ad16ce-tuning-conf-dir\") pod \"multus-additional-cni-plugins-h8nq8\" (UID: \"dfb5011f-b591-4203-87ce-b1c6d6ad16ce\") " pod="openshift-multus/multus-additional-cni-plugins-h8nq8"
Jul 31 23:46:18.556646 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:18.555964    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/93fe11db-24ef-40d3-a042-7be6f62bd6f6-etc-sysctl-d\") pod \"tuned-p7bwg\" (UID: \"93fe11db-24ef-40d3-a042-7be6f62bd6f6\") " pod="openshift-cluster-node-tuning-operator/tuned-p7bwg"
Jul 31 23:46:18.556646 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:18.555970    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/9d113c99-25bd-49a4-b8e2-f5c46f3ecc02-dbus\") pod \"global-pull-secret-syncer-xlv68\" (UID: \"9d113c99-25bd-49a4-b8e2-f5c46f3ecc02\") " pod="kube-system/global-pull-secret-syncer-xlv68"
Jul 31 23:46:18.556646 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:18.555986    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/93fe11db-24ef-40d3-a042-7be6f62bd6f6-etc-systemd\") pod \"tuned-p7bwg\" (UID: \"93fe11db-24ef-40d3-a042-7be6f62bd6f6\") " pod="openshift-cluster-node-tuning-operator/tuned-p7bwg"
Jul 31 23:46:18.556646 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:18.556021    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/67d8b75c-284a-4bdb-a560-65d472fb1515-tmp-dir\") pod \"node-resolver-wr2rk\" (UID: \"67d8b75c-284a-4bdb-a560-65d472fb1515\") " pod="openshift-dns/node-resolver-wr2rk"
Jul 31 23:46:18.556646 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:18.556040    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/66c4c4e9-d9f1-4a76-b937-cc3b1e16cee8-kubelet-dir\") pod \"aws-ebs-csi-driver-node-z94nv\" (UID: \"66c4c4e9-d9f1-4a76-b937-cc3b1e16cee8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z94nv"
Jul 31 23:46:18.556646 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:18.556063    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/087b0a99-ae24-49e2-ba59-55651731fd60-metrics-certs\") pod \"network-metrics-daemon-qs7p7\" (UID: \"087b0a99-ae24-49e2-ba59-55651731fd60\") " pod="openshift-multus/network-metrics-daemon-qs7p7"
Jul 31 23:46:18.556646 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:18.556089    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/93fe11db-24ef-40d3-a042-7be6f62bd6f6-etc-systemd\") pod \"tuned-p7bwg\" (UID: \"93fe11db-24ef-40d3-a042-7be6f62bd6f6\") " pod="openshift-cluster-node-tuning-operator/tuned-p7bwg"
Jul 31 23:46:18.556646 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:18.556093    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v2xtx\" (UniqueName: \"kubernetes.io/projected/087b0a99-ae24-49e2-ba59-55651731fd60-kube-api-access-v2xtx\") pod \"network-metrics-daemon-qs7p7\" (UID: \"087b0a99-ae24-49e2-ba59-55651731fd60\") " pod="openshift-multus/network-metrics-daemon-qs7p7"
Jul 31 23:46:18.556646 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:18.556187    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/66c4c4e9-d9f1-4a76-b937-cc3b1e16cee8-sys-fs\") pod \"aws-ebs-csi-driver-node-z94nv\" (UID: \"66c4c4e9-d9f1-4a76-b937-cc3b1e16cee8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z94nv"
Jul 31 23:46:18.557575 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:18.556222    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/93fe11db-24ef-40d3-a042-7be6f62bd6f6-etc-kubernetes\") pod \"tuned-p7bwg\" (UID: \"93fe11db-24ef-40d3-a042-7be6f62bd6f6\") " pod="openshift-cluster-node-tuning-operator/tuned-p7bwg"
Jul 31 23:46:18.557575 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:18.556297    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/9d113c99-25bd-49a4-b8e2-f5c46f3ecc02-kubelet-config\") pod \"global-pull-secret-syncer-xlv68\" (UID: \"9d113c99-25bd-49a4-b8e2-f5c46f3ecc02\") " pod="kube-system/global-pull-secret-syncer-xlv68"
Jul 31 23:46:18.557575 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:18.556332    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/dfb5011f-b591-4203-87ce-b1c6d6ad16ce-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-h8nq8\" (UID: \"dfb5011f-b591-4203-87ce-b1c6d6ad16ce\") " pod="openshift-multus/multus-additional-cni-plugins-h8nq8"
Jul 31 23:46:18.557575 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:18.556363    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/93fe11db-24ef-40d3-a042-7be6f62bd6f6-lib-modules\") pod \"tuned-p7bwg\" (UID: \"93fe11db-24ef-40d3-a042-7be6f62bd6f6\") " pod="openshift-cluster-node-tuning-operator/tuned-p7bwg"
Jul 31 23:46:18.557575 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:18.556376    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/dfb5011f-b591-4203-87ce-b1c6d6ad16ce-cnibin\") pod \"multus-additional-cni-plugins-h8nq8\" (UID: \"dfb5011f-b591-4203-87ce-b1c6d6ad16ce\") " pod="openshift-multus/multus-additional-cni-plugins-h8nq8"
Jul 31 23:46:18.557575 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:18.556391    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/93fe11db-24ef-40d3-a042-7be6f62bd6f6-var-lib-kubelet\") pod \"tuned-p7bwg\" (UID: \"93fe11db-24ef-40d3-a042-7be6f62bd6f6\") " pod="openshift-cluster-node-tuning-operator/tuned-p7bwg"
Jul 31 23:46:18.557575 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:18.556418    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a82bd3b0-bcad-4511-a3c4-4e4bdd5d6798-system-cni-dir\") pod \"multus-hlpzq\" (UID: \"a82bd3b0-bcad-4511-a3c4-4e4bdd5d6798\") " pod="openshift-multus/multus-hlpzq"
Jul 31 23:46:18.557575 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:18.556449    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/dfb5011f-b591-4203-87ce-b1c6d6ad16ce-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-h8nq8\" (UID: \"dfb5011f-b591-4203-87ce-b1c6d6ad16ce\") " pod="openshift-multus/multus-additional-cni-plugins-h8nq8"
Jul 31 23:46:18.557575 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:18.556537    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/dfb5011f-b591-4203-87ce-b1c6d6ad16ce-tuning-conf-dir\") pod \"multus-additional-cni-plugins-h8nq8\" (UID: \"dfb5011f-b591-4203-87ce-b1c6d6ad16ce\") " pod="openshift-multus/multus-additional-cni-plugins-h8nq8"
Jul 31 23:46:18.557575 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:18.556666    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/93fe11db-24ef-40d3-a042-7be6f62bd6f6-etc-sysctl-d\") pod \"tuned-p7bwg\" (UID: \"93fe11db-24ef-40d3-a042-7be6f62bd6f6\") " pod="openshift-cluster-node-tuning-operator/tuned-p7bwg"
Jul 31 23:46:18.557575 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:18.556674    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/66c4c4e9-d9f1-4a76-b937-cc3b1e16cee8-etc-selinux\") pod \"aws-ebs-csi-driver-node-z94nv\" (UID: \"66c4c4e9-d9f1-4a76-b937-cc3b1e16cee8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z94nv"
Jul 31 23:46:18.557575 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:18.556707    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/93fe11db-24ef-40d3-a042-7be6f62bd6f6-etc-tuned\") pod \"tuned-p7bwg\" (UID: \"93fe11db-24ef-40d3-a042-7be6f62bd6f6\") " pod="openshift-cluster-node-tuning-operator/tuned-p7bwg"
Jul 31 23:46:18.557575 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:18.556836    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/a82bd3b0-bcad-4511-a3c4-4e4bdd5d6798-multus-conf-dir\") pod \"multus-hlpzq\" (UID: \"a82bd3b0-bcad-4511-a3c4-4e4bdd5d6798\") " pod="openshift-multus/multus-hlpzq"
Jul 31 23:46:18.557575 ip-10-0-136-108 kubenswrapper[2578]: E0731 23:46:18.556856    2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 31 23:46:18.557575 ip-10-0-136-108 kubenswrapper[2578]: E0731 23:46:18.556908    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/087b0a99-ae24-49e2-ba59-55651731fd60-metrics-certs podName:087b0a99-ae24-49e2-ba59-55651731fd60 nodeName:}" failed. No retries permitted until 2026-07-31 23:46:19.056891494 +0000 UTC m=+3.024642247 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/087b0a99-ae24-49e2-ba59-55651731fd60-metrics-certs") pod "network-metrics-daemon-qs7p7" (UID: "087b0a99-ae24-49e2-ba59-55651731fd60") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 31 23:46:18.557575 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:18.556936    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/66c4c4e9-d9f1-4a76-b937-cc3b1e16cee8-socket-dir\") pod \"aws-ebs-csi-driver-node-z94nv\" (UID: \"66c4c4e9-d9f1-4a76-b937-cc3b1e16cee8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z94nv"
Jul 31 23:46:18.557575 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:18.556962    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/a82bd3b0-bcad-4511-a3c4-4e4bdd5d6798-host-run-k8s-cni-cncf-io\") pod \"multus-hlpzq\" (UID: \"a82bd3b0-bcad-4511-a3c4-4e4bdd5d6798\") " pod="openshift-multus/multus-hlpzq"
Jul 31 23:46:18.558489 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:18.557001    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/93fe11db-24ef-40d3-a042-7be6f62bd6f6-etc-modprobe-d\") pod \"tuned-p7bwg\" (UID: \"93fe11db-24ef-40d3-a042-7be6f62bd6f6\") " pod="openshift-cluster-node-tuning-operator/tuned-p7bwg"
Jul 31 23:46:18.558489 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:18.557019    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/a82bd3b0-bcad-4511-a3c4-4e4bdd5d6798-hostroot\") pod \"multus-hlpzq\" (UID: \"a82bd3b0-bcad-4511-a3c4-4e4bdd5d6798\") " pod="openshift-multus/multus-hlpzq"
Jul 31 23:46:18.558489 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:18.557028    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/a82bd3b0-bcad-4511-a3c4-4e4bdd5d6798-host-run-netns\") pod \"multus-hlpzq\" (UID: \"a82bd3b0-bcad-4511-a3c4-4e4bdd5d6798\") " pod="openshift-multus/multus-hlpzq"
Jul 31 23:46:18.558489 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:18.557053    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/dfb5011f-b591-4203-87ce-b1c6d6ad16ce-system-cni-dir\") pod \"multus-additional-cni-plugins-h8nq8\" (UID: \"dfb5011f-b591-4203-87ce-b1c6d6ad16ce\") " pod="openshift-multus/multus-additional-cni-plugins-h8nq8"
Jul 31 23:46:18.558489 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:18.557088    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/9d113c99-25bd-49a4-b8e2-f5c46f3ecc02-kubelet-config\") pod \"global-pull-secret-syncer-xlv68\" (UID: \"9d113c99-25bd-49a4-b8e2-f5c46f3ecc02\") " pod="kube-system/global-pull-secret-syncer-xlv68"
Jul 31 23:46:18.558489 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:18.557130    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/dfb5011f-b591-4203-87ce-b1c6d6ad16ce-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-h8nq8\" (UID: \"dfb5011f-b591-4203-87ce-b1c6d6ad16ce\") " pod="openshift-multus/multus-additional-cni-plugins-h8nq8"
Jul 31 23:46:18.558489 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:18.557295    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/66c4c4e9-d9f1-4a76-b937-cc3b1e16cee8-sys-fs\") pod \"aws-ebs-csi-driver-node-z94nv\" (UID: \"66c4c4e9-d9f1-4a76-b937-cc3b1e16cee8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z94nv"
Jul 31 23:46:18.558489 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:18.557332    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/93fe11db-24ef-40d3-a042-7be6f62bd6f6-etc-kubernetes\") pod \"tuned-p7bwg\" (UID: \"93fe11db-24ef-40d3-a042-7be6f62bd6f6\") " pod="openshift-cluster-node-tuning-operator/tuned-p7bwg"
Jul 31 23:46:18.558489 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:18.557359    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/67d8b75c-284a-4bdb-a560-65d472fb1515-tmp-dir\") pod \"node-resolver-wr2rk\" (UID: \"67d8b75c-284a-4bdb-a560-65d472fb1515\") " pod="openshift-dns/node-resolver-wr2rk"
Jul 31 23:46:18.558489 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:18.557366    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/93fe11db-24ef-40d3-a042-7be6f62bd6f6-var-lib-kubelet\") pod \"tuned-p7bwg\" (UID: \"93fe11db-24ef-40d3-a042-7be6f62bd6f6\") " pod="openshift-cluster-node-tuning-operator/tuned-p7bwg"
Jul 31 23:46:18.558489 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:18.557434    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/93fe11db-24ef-40d3-a042-7be6f62bd6f6-lib-modules\") pod \"tuned-p7bwg\" (UID: \"93fe11db-24ef-40d3-a042-7be6f62bd6f6\") " pod="openshift-cluster-node-tuning-operator/tuned-p7bwg"
Jul 31 23:46:18.558489 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:18.557469    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a82bd3b0-bcad-4511-a3c4-4e4bdd5d6798-system-cni-dir\") pod \"multus-hlpzq\" (UID: \"a82bd3b0-bcad-4511-a3c4-4e4bdd5d6798\") " pod="openshift-multus/multus-hlpzq"
Jul 31 23:46:18.558489 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:18.557635    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/dfb5011f-b591-4203-87ce-b1c6d6ad16ce-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-h8nq8\" (UID: \"dfb5011f-b591-4203-87ce-b1c6d6ad16ce\") " pod="openshift-multus/multus-additional-cni-plugins-h8nq8"
Jul 31 23:46:18.558489 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:18.557974    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/dfb5011f-b591-4203-87ce-b1c6d6ad16ce-cni-binary-copy\") pod \"multus-additional-cni-plugins-h8nq8\" (UID: \"dfb5011f-b591-4203-87ce-b1c6d6ad16ce\") " pod="openshift-multus/multus-additional-cni-plugins-h8nq8"
Jul 31 23:46:18.559859 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:18.559833    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/93fe11db-24ef-40d3-a042-7be6f62bd6f6-tmp\") pod \"tuned-p7bwg\" (UID: \"93fe11db-24ef-40d3-a042-7be6f62bd6f6\") " pod="openshift-cluster-node-tuning-operator/tuned-p7bwg"
Jul 31 23:46:18.562618 ip-10-0-136-108 kubenswrapper[2578]: E0731 23:46:18.562588    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 31 23:46:18.562618 ip-10-0-136-108 kubenswrapper[2578]: E0731 23:46:18.562621    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 31 23:46:18.562805 ip-10-0-136-108 kubenswrapper[2578]: E0731 23:46:18.562636    2578 projected.go:196] Error preparing data for projected volume kube-api-access-94n2k for pod openshift-network-diagnostics/network-check-target-8d9rr: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 31 23:46:18.562805 ip-10-0-136-108 kubenswrapper[2578]: E0731 23:46:18.562710    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0503a101-ddee-440a-ae03-e76fe30685e7-kube-api-access-94n2k podName:0503a101-ddee-440a-ae03-e76fe30685e7 nodeName:}" failed. No retries permitted until 2026-07-31 23:46:19.06269046 +0000 UTC m=+3.030441237 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-94n2k" (UniqueName: "kubernetes.io/projected/0503a101-ddee-440a-ae03-e76fe30685e7-kube-api-access-94n2k") pod "network-check-target-8d9rr" (UID: "0503a101-ddee-440a-ae03-e76fe30685e7") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 31 23:46:18.563365 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:18.563345    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t42nv\" (UniqueName: \"kubernetes.io/projected/dfb5011f-b591-4203-87ce-b1c6d6ad16ce-kube-api-access-t42nv\") pod \"multus-additional-cni-plugins-h8nq8\" (UID: \"dfb5011f-b591-4203-87ce-b1c6d6ad16ce\") " pod="openshift-multus/multus-additional-cni-plugins-h8nq8"
Jul 31 23:46:18.564617 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:18.564592    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-76fwg\" (UniqueName: \"kubernetes.io/projected/a82bd3b0-bcad-4511-a3c4-4e4bdd5d6798-kube-api-access-76fwg\") pod \"multus-hlpzq\" (UID: \"a82bd3b0-bcad-4511-a3c4-4e4bdd5d6798\") " pod="openshift-multus/multus-hlpzq"
Jul 31 23:46:18.564714 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:18.564649    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c94dm\" (UniqueName: \"kubernetes.io/projected/66c4c4e9-d9f1-4a76-b937-cc3b1e16cee8-kube-api-access-c94dm\") pod \"aws-ebs-csi-driver-node-z94nv\" (UID: \"66c4c4e9-d9f1-4a76-b937-cc3b1e16cee8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z94nv"
Jul 31 23:46:18.565276 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:18.565256    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v2xtx\" (UniqueName: \"kubernetes.io/projected/087b0a99-ae24-49e2-ba59-55651731fd60-kube-api-access-v2xtx\") pod \"network-metrics-daemon-qs7p7\" (UID: \"087b0a99-ae24-49e2-ba59-55651731fd60\") " pod="openshift-multus/network-metrics-daemon-qs7p7"
Jul 31 23:46:18.565584 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:18.565563    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8rdb2\" (UniqueName: \"kubernetes.io/projected/67d8b75c-284a-4bdb-a560-65d472fb1515-kube-api-access-8rdb2\") pod \"node-resolver-wr2rk\" (UID: \"67d8b75c-284a-4bdb-a560-65d472fb1515\") " pod="openshift-dns/node-resolver-wr2rk"
Jul 31 23:46:18.565693 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:18.565660    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-25jg7\" (UniqueName: \"kubernetes.io/projected/93fe11db-24ef-40d3-a042-7be6f62bd6f6-kube-api-access-25jg7\") pod \"tuned-p7bwg\" (UID: \"93fe11db-24ef-40d3-a042-7be6f62bd6f6\") " pod="openshift-cluster-node-tuning-operator/tuned-p7bwg"
Jul 31 23:46:18.643996 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:18.643884    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-9hxzf"
Jul 31 23:46:18.662864 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:18.662829    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-xssj7"
Jul 31 23:46:18.672983 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:18.672937    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-m9px5"
Jul 31 23:46:18.679042 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:18.679016    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-gttdm"
Jul 31 23:46:18.687899 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:18.687878    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-p7bwg"
Jul 31 23:46:18.695722 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:18.695700    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-wr2rk"
Jul 31 23:46:18.704592 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:18.704572    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z94nv"
Jul 31 23:46:18.712405 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:18.712388    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-h8nq8"
Jul 31 23:46:18.718343 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:18.718325    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-hlpzq"
Jul 31 23:46:19.060411 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:19.060310    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/087b0a99-ae24-49e2-ba59-55651731fd60-metrics-certs\") pod \"network-metrics-daemon-qs7p7\" (UID: \"087b0a99-ae24-49e2-ba59-55651731fd60\") " pod="openshift-multus/network-metrics-daemon-qs7p7"
Jul 31 23:46:19.060411 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:19.060376    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/9d113c99-25bd-49a4-b8e2-f5c46f3ecc02-original-pull-secret\") pod \"global-pull-secret-syncer-xlv68\" (UID: \"9d113c99-25bd-49a4-b8e2-f5c46f3ecc02\") " pod="kube-system/global-pull-secret-syncer-xlv68"
Jul 31 23:46:19.060651 ip-10-0-136-108 kubenswrapper[2578]: E0731 23:46:19.060499    2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 31 23:46:19.060651 ip-10-0-136-108 kubenswrapper[2578]: E0731 23:46:19.060523    2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 31 23:46:19.060651 ip-10-0-136-108 kubenswrapper[2578]: E0731 23:46:19.060590    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9d113c99-25bd-49a4-b8e2-f5c46f3ecc02-original-pull-secret podName:9d113c99-25bd-49a4-b8e2-f5c46f3ecc02 nodeName:}" failed. No retries permitted until 2026-07-31 23:46:20.060568069 +0000 UTC m=+4.028318807 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/9d113c99-25bd-49a4-b8e2-f5c46f3ecc02-original-pull-secret") pod "global-pull-secret-syncer-xlv68" (UID: "9d113c99-25bd-49a4-b8e2-f5c46f3ecc02") : object "kube-system"/"original-pull-secret" not registered
Jul 31 23:46:19.060651 ip-10-0-136-108 kubenswrapper[2578]: E0731 23:46:19.060610    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/087b0a99-ae24-49e2-ba59-55651731fd60-metrics-certs podName:087b0a99-ae24-49e2-ba59-55651731fd60 nodeName:}" failed. No retries permitted until 2026-07-31 23:46:20.060600204 +0000 UTC m=+4.028350946 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/087b0a99-ae24-49e2-ba59-55651731fd60-metrics-certs") pod "network-metrics-daemon-qs7p7" (UID: "087b0a99-ae24-49e2-ba59-55651731fd60") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 31 23:46:19.161642 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:19.161603    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-94n2k\" (UniqueName: \"kubernetes.io/projected/0503a101-ddee-440a-ae03-e76fe30685e7-kube-api-access-94n2k\") pod \"network-check-target-8d9rr\" (UID: \"0503a101-ddee-440a-ae03-e76fe30685e7\") " pod="openshift-network-diagnostics/network-check-target-8d9rr"
Jul 31 23:46:19.161842 ip-10-0-136-108 kubenswrapper[2578]: E0731 23:46:19.161743    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 31 23:46:19.161842 ip-10-0-136-108 kubenswrapper[2578]: E0731 23:46:19.161764    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 31 23:46:19.161842 ip-10-0-136-108 kubenswrapper[2578]: E0731 23:46:19.161776    2578 projected.go:196] Error preparing data for projected volume kube-api-access-94n2k for pod openshift-network-diagnostics/network-check-target-8d9rr: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 31 23:46:19.161958 ip-10-0-136-108 kubenswrapper[2578]: E0731 23:46:19.161850    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0503a101-ddee-440a-ae03-e76fe30685e7-kube-api-access-94n2k podName:0503a101-ddee-440a-ae03-e76fe30685e7 nodeName:}" failed. No retries permitted until 2026-07-31 23:46:20.16183425 +0000 UTC m=+4.129585013 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-94n2k" (UniqueName: "kubernetes.io/projected/0503a101-ddee-440a-ae03-e76fe30685e7-kube-api-access-94n2k") pod "network-check-target-8d9rr" (UID: "0503a101-ddee-440a-ae03-e76fe30685e7") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 31 23:46:19.392823 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:19.392731    2578 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-30 23:41:18 +0000 UTC" deadline="2028-03-20 17:57:08.039292882 +0000 UTC"
Jul 31 23:46:19.392823 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:19.392783    2578 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14346h10m48.646529614s"
Jul 31 23:46:19.483284 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:19.483253    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8d9rr"
Jul 31 23:46:19.483427 ip-10-0-136-108 kubenswrapper[2578]: E0731 23:46:19.483382    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8d9rr" podUID="0503a101-ddee-440a-ae03-e76fe30685e7"
Jul 31 23:46:19.512465 ip-10-0-136-108 kubenswrapper[2578]: W0731 23:46:19.512433    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod93fe11db_24ef_40d3_a042_7be6f62bd6f6.slice/crio-d9353c452075de2923ac01f7a3803761d005152e47c27e9630cd2570d7b688ed WatchSource:0}: Error finding container d9353c452075de2923ac01f7a3803761d005152e47c27e9630cd2570d7b688ed: Status 404 returned error can't find the container with id d9353c452075de2923ac01f7a3803761d005152e47c27e9630cd2570d7b688ed
Jul 31 23:46:19.515422 ip-10-0-136-108 kubenswrapper[2578]: W0731 23:46:19.515387    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfe1c93d5_e51c_4bea_b395_b782a588e7e0.slice/crio-1ca80726e23286fb4882aa563373a0dfb8f165e3b383885f53ad88ce4f7cd796 WatchSource:0}: Error finding container 1ca80726e23286fb4882aa563373a0dfb8f165e3b383885f53ad88ce4f7cd796: Status 404 returned error can't find the container with id 1ca80726e23286fb4882aa563373a0dfb8f165e3b383885f53ad88ce4f7cd796
Jul 31 23:46:19.515727 ip-10-0-136-108 kubenswrapper[2578]: W0731 23:46:19.515702    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda82bd3b0_bcad_4511_a3c4_4e4bdd5d6798.slice/crio-64eb958cb4b88b3e65cadbec7be2b136b4d4c31a6c2ff7a8daec694444aad2b9 WatchSource:0}: Error finding container 64eb958cb4b88b3e65cadbec7be2b136b4d4c31a6c2ff7a8daec694444aad2b9: Status 404 returned error can't find the container with id 64eb958cb4b88b3e65cadbec7be2b136b4d4c31a6c2ff7a8daec694444aad2b9
Jul 31 23:46:19.518512 ip-10-0-136-108 kubenswrapper[2578]: W0731 23:46:19.518489    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb32e6d42_e4e0_4113_b8ce_f5d5c5dc13f7.slice/crio-68a4126a2c0307951359df0a5d55f6d01b129b2cc98a40c1cf41648576533464 WatchSource:0}: Error finding container 68a4126a2c0307951359df0a5d55f6d01b129b2cc98a40c1cf41648576533464: Status 404 returned error can't find the container with id 68a4126a2c0307951359df0a5d55f6d01b129b2cc98a40c1cf41648576533464
Jul 31 23:46:19.520017 ip-10-0-136-108 kubenswrapper[2578]: W0731 23:46:19.519956    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddfb5011f_b591_4203_87ce_b1c6d6ad16ce.slice/crio-47d32f681335777046e0f32167aab81d78d485685d53fdc146a5e21c77dff226 WatchSource:0}: Error finding container 47d32f681335777046e0f32167aab81d78d485685d53fdc146a5e21c77dff226: Status 404 returned error can't find the container with id 47d32f681335777046e0f32167aab81d78d485685d53fdc146a5e21c77dff226
Jul 31 23:46:19.522766 ip-10-0-136-108 kubenswrapper[2578]: W0731 23:46:19.520881    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8d650e41_5af5_435a_81ab_aaf3898a490c.slice/crio-76a8b827343db0266133ade05fed1999f1656dc0a0674f6c2ab4de43b74771ac WatchSource:0}: Error finding container 76a8b827343db0266133ade05fed1999f1656dc0a0674f6c2ab4de43b74771ac: Status 404 returned error can't find the container with id 76a8b827343db0266133ade05fed1999f1656dc0a0674f6c2ab4de43b74771ac
Jul 31 23:46:19.522766 ip-10-0-136-108 kubenswrapper[2578]: W0731 23:46:19.522293    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod51ec46bf_6903_4794_adc9_3f796e976899.slice/crio-fb5c848bcc5563496c7b0e4b47f7864c936eada096b76de530b597d0062f480a WatchSource:0}: Error finding container fb5c848bcc5563496c7b0e4b47f7864c936eada096b76de530b597d0062f480a: Status 404 returned error can't find the container with id fb5c848bcc5563496c7b0e4b47f7864c936eada096b76de530b597d0062f480a
Jul 31 23:46:19.523723 ip-10-0-136-108 kubenswrapper[2578]: W0731 23:46:19.523691    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod67d8b75c_284a_4bdb_a560_65d472fb1515.slice/crio-194e8f4aacad116ae498d27463eb71557e4925c054d4b9587d1bbd564d4b39f5 WatchSource:0}: Error finding container 194e8f4aacad116ae498d27463eb71557e4925c054d4b9587d1bbd564d4b39f5: Status 404 returned error can't find the container with id 194e8f4aacad116ae498d27463eb71557e4925c054d4b9587d1bbd564d4b39f5
Jul 31 23:46:19.526306 ip-10-0-136-108 kubenswrapper[2578]: W0731 23:46:19.524694    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod66c4c4e9_d9f1_4a76_b937_cc3b1e16cee8.slice/crio-0f2227637d4d497986b196b93b009af00fd33662f12cde550d46b9735abc8132 WatchSource:0}: Error finding container 0f2227637d4d497986b196b93b009af00fd33662f12cde550d46b9735abc8132: Status 404 returned error can't find the container with id 0f2227637d4d497986b196b93b009af00fd33662f12cde550d46b9735abc8132
Jul 31 23:46:20.071041 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:20.070901    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/9d113c99-25bd-49a4-b8e2-f5c46f3ecc02-original-pull-secret\") pod \"global-pull-secret-syncer-xlv68\" (UID: \"9d113c99-25bd-49a4-b8e2-f5c46f3ecc02\") " pod="kube-system/global-pull-secret-syncer-xlv68"
Jul 31 23:46:20.071041 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:20.071009    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/087b0a99-ae24-49e2-ba59-55651731fd60-metrics-certs\") pod \"network-metrics-daemon-qs7p7\" (UID: \"087b0a99-ae24-49e2-ba59-55651731fd60\") " pod="openshift-multus/network-metrics-daemon-qs7p7"
Jul 31 23:46:20.071369 ip-10-0-136-108 kubenswrapper[2578]: E0731 23:46:20.071151    2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 31 23:46:20.071369 ip-10-0-136-108 kubenswrapper[2578]: E0731 23:46:20.071236    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/087b0a99-ae24-49e2-ba59-55651731fd60-metrics-certs podName:087b0a99-ae24-49e2-ba59-55651731fd60 nodeName:}" failed. No retries permitted until 2026-07-31 23:46:22.071199251 +0000 UTC m=+6.038949993 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/087b0a99-ae24-49e2-ba59-55651731fd60-metrics-certs") pod "network-metrics-daemon-qs7p7" (UID: "087b0a99-ae24-49e2-ba59-55651731fd60") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 31 23:46:20.071703 ip-10-0-136-108 kubenswrapper[2578]: E0731 23:46:20.071681    2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 31 23:46:20.071770 ip-10-0-136-108 kubenswrapper[2578]: E0731 23:46:20.071746    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9d113c99-25bd-49a4-b8e2-f5c46f3ecc02-original-pull-secret podName:9d113c99-25bd-49a4-b8e2-f5c46f3ecc02 nodeName:}" failed. No retries permitted until 2026-07-31 23:46:22.071729142 +0000 UTC m=+6.039479882 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/9d113c99-25bd-49a4-b8e2-f5c46f3ecc02-original-pull-secret") pod "global-pull-secret-syncer-xlv68" (UID: "9d113c99-25bd-49a4-b8e2-f5c46f3ecc02") : object "kube-system"/"original-pull-secret" not registered
Jul 31 23:46:20.172854 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:20.172137    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-94n2k\" (UniqueName: \"kubernetes.io/projected/0503a101-ddee-440a-ae03-e76fe30685e7-kube-api-access-94n2k\") pod \"network-check-target-8d9rr\" (UID: \"0503a101-ddee-440a-ae03-e76fe30685e7\") " pod="openshift-network-diagnostics/network-check-target-8d9rr"
Jul 31 23:46:20.172854 ip-10-0-136-108 kubenswrapper[2578]: E0731 23:46:20.172353    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 31 23:46:20.172854 ip-10-0-136-108 kubenswrapper[2578]: E0731 23:46:20.172377    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 31 23:46:20.172854 ip-10-0-136-108 kubenswrapper[2578]: E0731 23:46:20.172393    2578 projected.go:196] Error preparing data for projected volume kube-api-access-94n2k for pod openshift-network-diagnostics/network-check-target-8d9rr: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 31 23:46:20.172854 ip-10-0-136-108 kubenswrapper[2578]: E0731 23:46:20.172465    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0503a101-ddee-440a-ae03-e76fe30685e7-kube-api-access-94n2k podName:0503a101-ddee-440a-ae03-e76fe30685e7 nodeName:}" failed. No retries permitted until 2026-07-31 23:46:22.172442812 +0000 UTC m=+6.140193576 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-94n2k" (UniqueName: "kubernetes.io/projected/0503a101-ddee-440a-ae03-e76fe30685e7-kube-api-access-94n2k") pod "network-check-target-8d9rr" (UID: "0503a101-ddee-440a-ae03-e76fe30685e7") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 31 23:46:20.393582 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:20.393424    2578 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-30 23:41:18 +0000 UTC" deadline="2028-03-18 09:02:23.397790695 +0000 UTC"
Jul 31 23:46:20.393582 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:20.393491    2578 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14289h16m3.004303681s"
Jul 31 23:46:20.485171 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:20.485133    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qs7p7"
Jul 31 23:46:20.485408 ip-10-0-136-108 kubenswrapper[2578]: E0731 23:46:20.485321    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qs7p7" podUID="087b0a99-ae24-49e2-ba59-55651731fd60"
Jul 31 23:46:20.485408 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:20.485378    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xlv68"
Jul 31 23:46:20.485540 ip-10-0-136-108 kubenswrapper[2578]: E0731 23:46:20.485517    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xlv68" podUID="9d113c99-25bd-49a4-b8e2-f5c46f3ecc02"
Jul 31 23:46:20.515658 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:20.514963    2578 generic.go:356] "Generic (PLEG): container finished" podID="739886fda1f37e5f1a9572bed84e6cb4" containerID="fd2e27d157df8d814024db7eaa9dfd67721d3089605f37a1cf9b2e64afc55301" exitCode=0
Jul 31 23:46:20.515658 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:20.515052    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-108.ec2.internal" event={"ID":"739886fda1f37e5f1a9572bed84e6cb4","Type":"ContainerDied","Data":"fd2e27d157df8d814024db7eaa9dfd67721d3089605f37a1cf9b2e64afc55301"}
Jul 31 23:46:20.532215 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:20.532178    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z94nv" event={"ID":"66c4c4e9-d9f1-4a76-b937-cc3b1e16cee8","Type":"ContainerStarted","Data":"0f2227637d4d497986b196b93b009af00fd33662f12cde550d46b9735abc8132"}
Jul 31 23:46:20.538142 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:20.538106    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-wr2rk" event={"ID":"67d8b75c-284a-4bdb-a560-65d472fb1515","Type":"ContainerStarted","Data":"194e8f4aacad116ae498d27463eb71557e4925c054d4b9587d1bbd564d4b39f5"}
Jul 31 23:46:20.554045 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:20.554014    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-9hxzf" event={"ID":"51ec46bf-6903-4794-adc9-3f796e976899","Type":"ContainerStarted","Data":"fb5c848bcc5563496c7b0e4b47f7864c936eada096b76de530b597d0062f480a"}
Jul 31 23:46:20.565433 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:20.565401    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-m9px5" event={"ID":"8d650e41-5af5-435a-81ab-aaf3898a490c","Type":"ContainerStarted","Data":"76a8b827343db0266133ade05fed1999f1656dc0a0674f6c2ab4de43b74771ac"}
Jul 31 23:46:20.577712 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:20.577355    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-gttdm" event={"ID":"b32e6d42-e4e0-4113-b8ce-f5d5c5dc13f7","Type":"ContainerStarted","Data":"68a4126a2c0307951359df0a5d55f6d01b129b2cc98a40c1cf41648576533464"}
Jul 31 23:46:20.586910 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:20.586535    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-108.ec2.internal" event={"ID":"cc577a92ce0dd873599a996e54a72791","Type":"ContainerStarted","Data":"bf85707946e07ce6ceb9c7748c8fb1af2576270858e27694f0c26ee7b6b224a6"}
Jul 31 23:46:20.593402 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:20.593374    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-h8nq8" event={"ID":"dfb5011f-b591-4203-87ce-b1c6d6ad16ce","Type":"ContainerStarted","Data":"47d32f681335777046e0f32167aab81d78d485685d53fdc146a5e21c77dff226"}
Jul 31 23:46:20.605849 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:20.605813    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-hlpzq" event={"ID":"a82bd3b0-bcad-4511-a3c4-4e4bdd5d6798","Type":"ContainerStarted","Data":"64eb958cb4b88b3e65cadbec7be2b136b4d4c31a6c2ff7a8daec694444aad2b9"}
Jul 31 23:46:20.625414 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:20.625376    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-xssj7" event={"ID":"fe1c93d5-e51c-4bea-b395-b782a588e7e0","Type":"ContainerStarted","Data":"1ca80726e23286fb4882aa563373a0dfb8f165e3b383885f53ad88ce4f7cd796"}
Jul 31 23:46:20.638853 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:20.638780    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-p7bwg" event={"ID":"93fe11db-24ef-40d3-a042-7be6f62bd6f6","Type":"ContainerStarted","Data":"d9353c452075de2923ac01f7a3803761d005152e47c27e9630cd2570d7b688ed"}
Jul 31 23:46:21.483921 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:21.483871    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8d9rr"
Jul 31 23:46:21.484466 ip-10-0-136-108 kubenswrapper[2578]: E0731 23:46:21.484034    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8d9rr" podUID="0503a101-ddee-440a-ae03-e76fe30685e7"
Jul 31 23:46:21.663350 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:21.663312    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-108.ec2.internal" event={"ID":"739886fda1f37e5f1a9572bed84e6cb4","Type":"ContainerStarted","Data":"5d64db955927fc3fcfa563dddd99359c608dae188d0bfb98248655152a589d7f"}
Jul 31 23:46:21.679917 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:21.679846    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-108.ec2.internal" podStartSLOduration=3.679822581 podStartE2EDuration="3.679822581s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-31 23:46:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 23:46:20.602113444 +0000 UTC m=+4.569864207" watchObservedRunningTime="2026-07-31 23:46:21.679822581 +0000 UTC m=+5.647573343"
Jul 31 23:46:21.680118 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:21.680050    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-108.ec2.internal" podStartSLOduration=3.680042186 podStartE2EDuration="3.680042186s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-31 23:46:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 23:46:21.679200948 +0000 UTC m=+5.646951704" watchObservedRunningTime="2026-07-31 23:46:21.680042186 +0000 UTC m=+5.647792949"
Jul 31 23:46:22.091094 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:22.091036    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/9d113c99-25bd-49a4-b8e2-f5c46f3ecc02-original-pull-secret\") pod \"global-pull-secret-syncer-xlv68\" (UID: \"9d113c99-25bd-49a4-b8e2-f5c46f3ecc02\") " pod="kube-system/global-pull-secret-syncer-xlv68"
Jul 31 23:46:22.091306 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:22.091169    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/087b0a99-ae24-49e2-ba59-55651731fd60-metrics-certs\") pod \"network-metrics-daemon-qs7p7\" (UID: \"087b0a99-ae24-49e2-ba59-55651731fd60\") " pod="openshift-multus/network-metrics-daemon-qs7p7"
Jul 31 23:46:22.091391 ip-10-0-136-108 kubenswrapper[2578]: E0731 23:46:22.091329    2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 31 23:46:22.091454 ip-10-0-136-108 kubenswrapper[2578]: E0731 23:46:22.091400    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/087b0a99-ae24-49e2-ba59-55651731fd60-metrics-certs podName:087b0a99-ae24-49e2-ba59-55651731fd60 nodeName:}" failed. No retries permitted until 2026-07-31 23:46:26.091377995 +0000 UTC m=+10.059128740 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/087b0a99-ae24-49e2-ba59-55651731fd60-metrics-certs") pod "network-metrics-daemon-qs7p7" (UID: "087b0a99-ae24-49e2-ba59-55651731fd60") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 31 23:46:22.091825 ip-10-0-136-108 kubenswrapper[2578]: E0731 23:46:22.091804    2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 31 23:46:22.091903 ip-10-0-136-108 kubenswrapper[2578]: E0731 23:46:22.091858    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9d113c99-25bd-49a4-b8e2-f5c46f3ecc02-original-pull-secret podName:9d113c99-25bd-49a4-b8e2-f5c46f3ecc02 nodeName:}" failed. No retries permitted until 2026-07-31 23:46:26.091843657 +0000 UTC m=+10.059594398 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/9d113c99-25bd-49a4-b8e2-f5c46f3ecc02-original-pull-secret") pod "global-pull-secret-syncer-xlv68" (UID: "9d113c99-25bd-49a4-b8e2-f5c46f3ecc02") : object "kube-system"/"original-pull-secret" not registered
Jul 31 23:46:22.191845 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:22.191762    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-94n2k\" (UniqueName: \"kubernetes.io/projected/0503a101-ddee-440a-ae03-e76fe30685e7-kube-api-access-94n2k\") pod \"network-check-target-8d9rr\" (UID: \"0503a101-ddee-440a-ae03-e76fe30685e7\") " pod="openshift-network-diagnostics/network-check-target-8d9rr"
Jul 31 23:46:22.192118 ip-10-0-136-108 kubenswrapper[2578]: E0731 23:46:22.192054    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 31 23:46:22.192118 ip-10-0-136-108 kubenswrapper[2578]: E0731 23:46:22.192085    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 31 23:46:22.192118 ip-10-0-136-108 kubenswrapper[2578]: E0731 23:46:22.192101    2578 projected.go:196] Error preparing data for projected volume kube-api-access-94n2k for pod openshift-network-diagnostics/network-check-target-8d9rr: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 31 23:46:22.192356 ip-10-0-136-108 kubenswrapper[2578]: E0731 23:46:22.192168    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0503a101-ddee-440a-ae03-e76fe30685e7-kube-api-access-94n2k podName:0503a101-ddee-440a-ae03-e76fe30685e7 nodeName:}" failed. No retries permitted until 2026-07-31 23:46:26.192147965 +0000 UTC m=+10.159898720 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-94n2k" (UniqueName: "kubernetes.io/projected/0503a101-ddee-440a-ae03-e76fe30685e7-kube-api-access-94n2k") pod "network-check-target-8d9rr" (UID: "0503a101-ddee-440a-ae03-e76fe30685e7") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 31 23:46:22.484437 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:22.484346    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qs7p7"
Jul 31 23:46:22.484902 ip-10-0-136-108 kubenswrapper[2578]: E0731 23:46:22.484489    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qs7p7" podUID="087b0a99-ae24-49e2-ba59-55651731fd60"
Jul 31 23:46:22.487033 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:22.487008    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xlv68"
Jul 31 23:46:22.487196 ip-10-0-136-108 kubenswrapper[2578]: E0731 23:46:22.487133    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xlv68" podUID="9d113c99-25bd-49a4-b8e2-f5c46f3ecc02"
Jul 31 23:46:23.483594 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:23.483556    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8d9rr"
Jul 31 23:46:23.483796 ip-10-0-136-108 kubenswrapper[2578]: E0731 23:46:23.483695    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8d9rr" podUID="0503a101-ddee-440a-ae03-e76fe30685e7"
Jul 31 23:46:24.485831 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:24.485792    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qs7p7"
Jul 31 23:46:24.485831 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:24.485815    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xlv68"
Jul 31 23:46:24.486359 ip-10-0-136-108 kubenswrapper[2578]: E0731 23:46:24.485958    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xlv68" podUID="9d113c99-25bd-49a4-b8e2-f5c46f3ecc02"
Jul 31 23:46:24.486359 ip-10-0-136-108 kubenswrapper[2578]: E0731 23:46:24.486113    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qs7p7" podUID="087b0a99-ae24-49e2-ba59-55651731fd60"
Jul 31 23:46:25.483840 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:25.483321    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8d9rr"
Jul 31 23:46:25.483840 ip-10-0-136-108 kubenswrapper[2578]: E0731 23:46:25.483455    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8d9rr" podUID="0503a101-ddee-440a-ae03-e76fe30685e7"
Jul 31 23:46:26.127828 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:26.127619    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/9d113c99-25bd-49a4-b8e2-f5c46f3ecc02-original-pull-secret\") pod \"global-pull-secret-syncer-xlv68\" (UID: \"9d113c99-25bd-49a4-b8e2-f5c46f3ecc02\") " pod="kube-system/global-pull-secret-syncer-xlv68"
Jul 31 23:46:26.127828 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:26.127727    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/087b0a99-ae24-49e2-ba59-55651731fd60-metrics-certs\") pod \"network-metrics-daemon-qs7p7\" (UID: \"087b0a99-ae24-49e2-ba59-55651731fd60\") " pod="openshift-multus/network-metrics-daemon-qs7p7"
Jul 31 23:46:26.127828 ip-10-0-136-108 kubenswrapper[2578]: E0731 23:46:26.127804    2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 31 23:46:26.128297 ip-10-0-136-108 kubenswrapper[2578]: E0731 23:46:26.127841    2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 31 23:46:26.128297 ip-10-0-136-108 kubenswrapper[2578]: E0731 23:46:26.127882    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9d113c99-25bd-49a4-b8e2-f5c46f3ecc02-original-pull-secret podName:9d113c99-25bd-49a4-b8e2-f5c46f3ecc02 nodeName:}" failed. No retries permitted until 2026-07-31 23:46:34.127861833 +0000 UTC m=+18.095612579 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/9d113c99-25bd-49a4-b8e2-f5c46f3ecc02-original-pull-secret") pod "global-pull-secret-syncer-xlv68" (UID: "9d113c99-25bd-49a4-b8e2-f5c46f3ecc02") : object "kube-system"/"original-pull-secret" not registered
Jul 31 23:46:26.128297 ip-10-0-136-108 kubenswrapper[2578]: E0731 23:46:26.127903    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/087b0a99-ae24-49e2-ba59-55651731fd60-metrics-certs podName:087b0a99-ae24-49e2-ba59-55651731fd60 nodeName:}" failed. No retries permitted until 2026-07-31 23:46:34.12789337 +0000 UTC m=+18.095644108 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/087b0a99-ae24-49e2-ba59-55651731fd60-metrics-certs") pod "network-metrics-daemon-qs7p7" (UID: "087b0a99-ae24-49e2-ba59-55651731fd60") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 31 23:46:26.229578 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:26.228949    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-94n2k\" (UniqueName: \"kubernetes.io/projected/0503a101-ddee-440a-ae03-e76fe30685e7-kube-api-access-94n2k\") pod \"network-check-target-8d9rr\" (UID: \"0503a101-ddee-440a-ae03-e76fe30685e7\") " pod="openshift-network-diagnostics/network-check-target-8d9rr"
Jul 31 23:46:26.229578 ip-10-0-136-108 kubenswrapper[2578]: E0731 23:46:26.229121    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 31 23:46:26.229578 ip-10-0-136-108 kubenswrapper[2578]: E0731 23:46:26.229140    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 31 23:46:26.229578 ip-10-0-136-108 kubenswrapper[2578]: E0731 23:46:26.229154    2578 projected.go:196] Error preparing data for projected volume kube-api-access-94n2k for pod openshift-network-diagnostics/network-check-target-8d9rr: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 31 23:46:26.229578 ip-10-0-136-108 kubenswrapper[2578]: E0731 23:46:26.229229    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0503a101-ddee-440a-ae03-e76fe30685e7-kube-api-access-94n2k podName:0503a101-ddee-440a-ae03-e76fe30685e7 nodeName:}" failed. No retries permitted until 2026-07-31 23:46:34.229211506 +0000 UTC m=+18.196962245 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-94n2k" (UniqueName: "kubernetes.io/projected/0503a101-ddee-440a-ae03-e76fe30685e7-kube-api-access-94n2k") pod "network-check-target-8d9rr" (UID: "0503a101-ddee-440a-ae03-e76fe30685e7") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 31 23:46:26.484124 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:26.484042    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xlv68"
Jul 31 23:46:26.484289 ip-10-0-136-108 kubenswrapper[2578]: E0731 23:46:26.484200    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xlv68" podUID="9d113c99-25bd-49a4-b8e2-f5c46f3ecc02"
Jul 31 23:46:26.485680 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:26.485650    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qs7p7"
Jul 31 23:46:26.485795 ip-10-0-136-108 kubenswrapper[2578]: E0731 23:46:26.485741    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qs7p7" podUID="087b0a99-ae24-49e2-ba59-55651731fd60"
Jul 31 23:46:27.483807 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:27.483316    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8d9rr"
Jul 31 23:46:27.483807 ip-10-0-136-108 kubenswrapper[2578]: E0731 23:46:27.483440    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8d9rr" podUID="0503a101-ddee-440a-ae03-e76fe30685e7"
Jul 31 23:46:28.483555 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:28.483466    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xlv68"
Jul 31 23:46:28.483714 ip-10-0-136-108 kubenswrapper[2578]: E0731 23:46:28.483612    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xlv68" podUID="9d113c99-25bd-49a4-b8e2-f5c46f3ecc02"
Jul 31 23:46:28.483714 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:28.483677    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qs7p7"
Jul 31 23:46:28.483864 ip-10-0-136-108 kubenswrapper[2578]: E0731 23:46:28.483836    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qs7p7" podUID="087b0a99-ae24-49e2-ba59-55651731fd60"
Jul 31 23:46:29.483189 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:29.483135    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8d9rr"
Jul 31 23:46:29.483384 ip-10-0-136-108 kubenswrapper[2578]: E0731 23:46:29.483266    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8d9rr" podUID="0503a101-ddee-440a-ae03-e76fe30685e7"
Jul 31 23:46:30.483370 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:30.483326    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qs7p7"
Jul 31 23:46:30.483797 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:30.483375    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xlv68"
Jul 31 23:46:30.483797 ip-10-0-136-108 kubenswrapper[2578]: E0731 23:46:30.483470    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qs7p7" podUID="087b0a99-ae24-49e2-ba59-55651731fd60"
Jul 31 23:46:30.483797 ip-10-0-136-108 kubenswrapper[2578]: E0731 23:46:30.483584    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xlv68" podUID="9d113c99-25bd-49a4-b8e2-f5c46f3ecc02"
Jul 31 23:46:31.483253 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:31.483216    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8d9rr"
Jul 31 23:46:31.483413 ip-10-0-136-108 kubenswrapper[2578]: E0731 23:46:31.483358    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8d9rr" podUID="0503a101-ddee-440a-ae03-e76fe30685e7"
Jul 31 23:46:32.483468 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:32.483431    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qs7p7"
Jul 31 23:46:32.483980 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:32.483430    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xlv68"
Jul 31 23:46:32.483980 ip-10-0-136-108 kubenswrapper[2578]: E0731 23:46:32.483599    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qs7p7" podUID="087b0a99-ae24-49e2-ba59-55651731fd60"
Jul 31 23:46:32.483980 ip-10-0-136-108 kubenswrapper[2578]: E0731 23:46:32.483665    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xlv68" podUID="9d113c99-25bd-49a4-b8e2-f5c46f3ecc02"
Jul 31 23:46:33.483600 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:33.483565    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8d9rr"
Jul 31 23:46:33.484139 ip-10-0-136-108 kubenswrapper[2578]: E0731 23:46:33.483707    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8d9rr" podUID="0503a101-ddee-440a-ae03-e76fe30685e7"
Jul 31 23:46:34.190618 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:34.190580    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/087b0a99-ae24-49e2-ba59-55651731fd60-metrics-certs\") pod \"network-metrics-daemon-qs7p7\" (UID: \"087b0a99-ae24-49e2-ba59-55651731fd60\") " pod="openshift-multus/network-metrics-daemon-qs7p7"
Jul 31 23:46:34.190765 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:34.190636    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/9d113c99-25bd-49a4-b8e2-f5c46f3ecc02-original-pull-secret\") pod \"global-pull-secret-syncer-xlv68\" (UID: \"9d113c99-25bd-49a4-b8e2-f5c46f3ecc02\") " pod="kube-system/global-pull-secret-syncer-xlv68"
Jul 31 23:46:34.190765 ip-10-0-136-108 kubenswrapper[2578]: E0731 23:46:34.190726    2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 31 23:46:34.190765 ip-10-0-136-108 kubenswrapper[2578]: E0731 23:46:34.190739    2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 31 23:46:34.190936 ip-10-0-136-108 kubenswrapper[2578]: E0731 23:46:34.190791    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/087b0a99-ae24-49e2-ba59-55651731fd60-metrics-certs podName:087b0a99-ae24-49e2-ba59-55651731fd60 nodeName:}" failed. No retries permitted until 2026-07-31 23:46:50.190770632 +0000 UTC m=+34.158521371 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/087b0a99-ae24-49e2-ba59-55651731fd60-metrics-certs") pod "network-metrics-daemon-qs7p7" (UID: "087b0a99-ae24-49e2-ba59-55651731fd60") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 31 23:46:34.190936 ip-10-0-136-108 kubenswrapper[2578]: E0731 23:46:34.190813    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9d113c99-25bd-49a4-b8e2-f5c46f3ecc02-original-pull-secret podName:9d113c99-25bd-49a4-b8e2-f5c46f3ecc02 nodeName:}" failed. No retries permitted until 2026-07-31 23:46:50.190802452 +0000 UTC m=+34.158553191 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/9d113c99-25bd-49a4-b8e2-f5c46f3ecc02-original-pull-secret") pod "global-pull-secret-syncer-xlv68" (UID: "9d113c99-25bd-49a4-b8e2-f5c46f3ecc02") : object "kube-system"/"original-pull-secret" not registered
Jul 31 23:46:34.291787 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:34.291754    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-94n2k\" (UniqueName: \"kubernetes.io/projected/0503a101-ddee-440a-ae03-e76fe30685e7-kube-api-access-94n2k\") pod \"network-check-target-8d9rr\" (UID: \"0503a101-ddee-440a-ae03-e76fe30685e7\") " pod="openshift-network-diagnostics/network-check-target-8d9rr"
Jul 31 23:46:34.291989 ip-10-0-136-108 kubenswrapper[2578]: E0731 23:46:34.291891    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 31 23:46:34.291989 ip-10-0-136-108 kubenswrapper[2578]: E0731 23:46:34.291913    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 31 23:46:34.291989 ip-10-0-136-108 kubenswrapper[2578]: E0731 23:46:34.291924    2578 projected.go:196] Error preparing data for projected volume kube-api-access-94n2k for pod openshift-network-diagnostics/network-check-target-8d9rr: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 31 23:46:34.291989 ip-10-0-136-108 kubenswrapper[2578]: E0731 23:46:34.291974    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0503a101-ddee-440a-ae03-e76fe30685e7-kube-api-access-94n2k podName:0503a101-ddee-440a-ae03-e76fe30685e7 nodeName:}" failed. No retries permitted until 2026-07-31 23:46:50.291961041 +0000 UTC m=+34.259711777 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-94n2k" (UniqueName: "kubernetes.io/projected/0503a101-ddee-440a-ae03-e76fe30685e7-kube-api-access-94n2k") pod "network-check-target-8d9rr" (UID: "0503a101-ddee-440a-ae03-e76fe30685e7") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 31 23:46:34.483926 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:34.483833    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xlv68"
Jul 31 23:46:34.484396 ip-10-0-136-108 kubenswrapper[2578]: E0731 23:46:34.483979    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xlv68" podUID="9d113c99-25bd-49a4-b8e2-f5c46f3ecc02"
Jul 31 23:46:34.484396 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:34.484036    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qs7p7"
Jul 31 23:46:34.484396 ip-10-0-136-108 kubenswrapper[2578]: E0731 23:46:34.484182    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qs7p7" podUID="087b0a99-ae24-49e2-ba59-55651731fd60"
Jul 31 23:46:35.483364 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:35.483325    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8d9rr"
Jul 31 23:46:35.483559 ip-10-0-136-108 kubenswrapper[2578]: E0731 23:46:35.483475    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8d9rr" podUID="0503a101-ddee-440a-ae03-e76fe30685e7"
Jul 31 23:46:36.484855 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:36.484758    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qs7p7"
Jul 31 23:46:36.485344 ip-10-0-136-108 kubenswrapper[2578]: E0731 23:46:36.484905    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qs7p7" podUID="087b0a99-ae24-49e2-ba59-55651731fd60"
Jul 31 23:46:36.485344 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:36.484922    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xlv68"
Jul 31 23:46:36.485344 ip-10-0-136-108 kubenswrapper[2578]: E0731 23:46:36.485049    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xlv68" podUID="9d113c99-25bd-49a4-b8e2-f5c46f3ecc02"
Jul 31 23:46:37.483935 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:37.483888    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8d9rr"
Jul 31 23:46:37.484141 ip-10-0-136-108 kubenswrapper[2578]: E0731 23:46:37.484018    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8d9rr" podUID="0503a101-ddee-440a-ae03-e76fe30685e7"
Jul 31 23:46:38.483503 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:38.483465    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qs7p7"
Jul 31 23:46:38.483962 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:38.483516    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xlv68"
Jul 31 23:46:38.483962 ip-10-0-136-108 kubenswrapper[2578]: E0731 23:46:38.483593    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qs7p7" podUID="087b0a99-ae24-49e2-ba59-55651731fd60"
Jul 31 23:46:38.483962 ip-10-0-136-108 kubenswrapper[2578]: E0731 23:46:38.483705    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xlv68" podUID="9d113c99-25bd-49a4-b8e2-f5c46f3ecc02"
Jul 31 23:46:38.703851 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:38.703651    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-p7bwg" event={"ID":"93fe11db-24ef-40d3-a042-7be6f62bd6f6","Type":"ContainerStarted","Data":"36854bb450f4372a905b0bddcfe7c7bb4ed35d83ba92a88d99e73cda57772b7a"}
Jul 31 23:46:38.722767 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:38.722719    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-p7bwg" podStartSLOduration=2.692550308 podStartE2EDuration="21.722700304s" totalImagesPullingTime="19.030149996s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-31 23:46:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 23:46:38.722018362 +0000 UTC m=+22.689769123" watchObservedRunningTime="2026-07-31 23:46:38.722700304 +0000 UTC m=+22.690451062"
Jul 31 23:46:39.483792 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:39.483520    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8d9rr"
Jul 31 23:46:39.484733 ip-10-0-136-108 kubenswrapper[2578]: E0731 23:46:39.483855    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8d9rr" podUID="0503a101-ddee-440a-ae03-e76fe30685e7"
Jul 31 23:46:39.707283 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:39.707243    2578 generic.go:356] "Generic (PLEG): container finished" podID="dfb5011f-b591-4203-87ce-b1c6d6ad16ce" containerID="d9b2cda86dac614c005b2c84689df64189d03e86c3b9116e6d02f28bb4307275" exitCode=0
Jul 31 23:46:39.707462 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:39.707295    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-h8nq8" event={"ID":"dfb5011f-b591-4203-87ce-b1c6d6ad16ce","Type":"ContainerDied","Data":"d9b2cda86dac614c005b2c84689df64189d03e86c3b9116e6d02f28bb4307275"}
Jul 31 23:46:39.708788 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:39.708762    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-hlpzq" event={"ID":"a82bd3b0-bcad-4511-a3c4-4e4bdd5d6798","Type":"ContainerStarted","Data":"e4588cf05a14cf63187c38f5df4be93d945a47c709f019a126f1174e34b8d1ba"}
Jul 31 23:46:39.710244 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:39.710225    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z94nv" event={"ID":"66c4c4e9-d9f1-4a76-b937-cc3b1e16cee8","Type":"ContainerStarted","Data":"c795f5bc305539306290c82fed6af147402a16aef9db068968071a4f7581ea68"}
Jul 31 23:46:39.711632 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:39.711613    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-wr2rk" event={"ID":"67d8b75c-284a-4bdb-a560-65d472fb1515","Type":"ContainerStarted","Data":"7f5ff02ce4283241689e75156c2272ee5ed209a412769dcd7ef0619c5cb05a6b"}
Jul 31 23:46:39.712951 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:39.712933    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-9hxzf" event={"ID":"51ec46bf-6903-4794-adc9-3f796e976899","Type":"ContainerStarted","Data":"c3bcf3a706e9c78e2db39069dd7fa631b7db5d395e5e019dcc845d6d91a8e722"}
Jul 31 23:46:39.715698 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:39.715673    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-m9px5" event={"ID":"8d650e41-5af5-435a-81ab-aaf3898a490c","Type":"ContainerStarted","Data":"91f902b75218eacce0472954d8b93645ae6490d0ba6f8d428364847904fa6aa5"}
Jul 31 23:46:39.715789 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:39.715730    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-m9px5" event={"ID":"8d650e41-5af5-435a-81ab-aaf3898a490c","Type":"ContainerStarted","Data":"b22e5895ea326e4007e3e7fd76531ad08b467ecfafb2f428e92707da0325196c"}
Jul 31 23:46:39.715789 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:39.715761    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-m9px5" event={"ID":"8d650e41-5af5-435a-81ab-aaf3898a490c","Type":"ContainerStarted","Data":"ecf4c49ef56e126682eeea5f198660be9027b7b4fb0af5fb1c6dca35bef9e1bb"}
Jul 31 23:46:39.715789 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:39.715789    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-m9px5" event={"ID":"8d650e41-5af5-435a-81ab-aaf3898a490c","Type":"ContainerStarted","Data":"6042f7f91a054ee1629d23d35c22328659085772f97cbe4569e8e3b09a8d601e"}
Jul 31 23:46:39.715932 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:39.715821    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-m9px5" event={"ID":"8d650e41-5af5-435a-81ab-aaf3898a490c","Type":"ContainerStarted","Data":"4cbd73a3a690b31d34eca147b2a67254c5873f9cb8565a227b70f4bd0ed7ff65"}
Jul 31 23:46:39.715932 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:39.715855    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-m9px5" event={"ID":"8d650e41-5af5-435a-81ab-aaf3898a490c","Type":"ContainerStarted","Data":"918b06af6a2d50856ef67d159faf20fad98216ea7f608fdc920b596a38e5aed9"}
Jul 31 23:46:39.717120 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:39.717098    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-gttdm" event={"ID":"b32e6d42-e4e0-4113-b8ce-f5d5c5dc13f7","Type":"ContainerStarted","Data":"062898712a07b38ebdf83e2fe7185863f744ab96e45fd71abba175251c80fd0d"}
Jul 31 23:46:39.742209 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:39.742079    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-hlpzq" podStartSLOduration=3.665503253 podStartE2EDuration="22.742064757s" totalImagesPullingTime="19.076561504s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-31 23:46:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 23:46:39.74169854 +0000 UTC m=+23.709449297" watchObservedRunningTime="2026-07-31 23:46:39.742064757 +0000 UTC m=+23.709815514"
Jul 31 23:46:39.767886 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:39.767838    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-9hxzf" podStartSLOduration=3.762693119 podStartE2EDuration="22.767822091s" totalImagesPullingTime="19.005128972s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-31 23:46:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 23:46:39.754574138 +0000 UTC m=+23.722324896" watchObservedRunningTime="2026-07-31 23:46:39.767822091 +0000 UTC m=+23.735572875"
Jul 31 23:46:39.780283 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:39.780243    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-gttdm" podStartSLOduration=7.695424122 podStartE2EDuration="22.780225979s" totalImagesPullingTime="15.084801857s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 23:46:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 23:46:39.767656363 +0000 UTC m=+23.735407121" watchObservedRunningTime="2026-07-31 23:46:39.780225979 +0000 UTC m=+23.747976736"
Jul 31 23:46:39.780398 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:39.780365    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-wr2rk" podStartSLOduration=3.775354283 podStartE2EDuration="22.780360554s" totalImagesPullingTime="19.005006271s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-31 23:46:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 23:46:39.780129683 +0000 UTC m=+23.747880439" watchObservedRunningTime="2026-07-31 23:46:39.780360554 +0000 UTC m=+23.748111311"
Jul 31 23:46:39.892199 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:39.892153    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gttdm_b32e6d42-e4e0-4113-b8ce-f5d5c5dc13f7/konnectivity-agent/0.log" limit=1024
Jul 31 23:46:39.969896 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:39.969865    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gttdm_b32e6d42-e4e0-4113-b8ce-f5d5c5dc13f7/konnectivity-agent/0.log" limit=1024
Jul 31 23:46:39.998626 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:39.998569    2578 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Jul 31 23:46:40.408495 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:40.408357    2578 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-31T23:46:39.99859037Z","UUID":"ddc9d399-a012-434a-b454-89b9dea96aa5","Handler":null,"Name":"","Endpoint":""}
Jul 31 23:46:40.409777 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:40.409759    2578 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Jul 31 23:46:40.409883 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:40.409798    2578 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Jul 31 23:46:40.442626 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:40.442593    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gttdm_b32e6d42-e4e0-4113-b8ce-f5d5c5dc13f7/konnectivity-agent/0.log" limit=1024
Jul 31 23:46:40.483917 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:40.483881    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qs7p7"
Jul 31 23:46:40.484438 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:40.483881    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xlv68"
Jul 31 23:46:40.484438 ip-10-0-136-108 kubenswrapper[2578]: E0731 23:46:40.484030    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qs7p7" podUID="087b0a99-ae24-49e2-ba59-55651731fd60"
Jul 31 23:46:40.484438 ip-10-0-136-108 kubenswrapper[2578]: E0731 23:46:40.484162    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xlv68" podUID="9d113c99-25bd-49a4-b8e2-f5c46f3ecc02"
Jul 31 23:46:40.497769 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:40.497742    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-gttdm"
Jul 31 23:46:40.498410 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:40.498383    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-gttdm"
Jul 31 23:46:40.517979 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:40.517940    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gttdm_b32e6d42-e4e0-4113-b8ce-f5d5c5dc13f7/konnectivity-agent/0.log" limit=1024
Jul 31 23:46:40.722067 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:40.721979    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z94nv" event={"ID":"66c4c4e9-d9f1-4a76-b937-cc3b1e16cee8","Type":"ContainerStarted","Data":"58e42796b89c9d402c45a0f30bb1a5620e8332fc0ee40d22f6eebd53ca3cddc0"}
Jul 31 23:46:40.723849 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:40.723819    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-xssj7" event={"ID":"fe1c93d5-e51c-4bea-b395-b782a588e7e0","Type":"ContainerStarted","Data":"b5ace2fb157d0cc409c2263461b8483480911be73e6193699b42d2c0d661e57d"}
Jul 31 23:46:40.737731 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:40.737667    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-xssj7" podStartSLOduration=4.726710623 podStartE2EDuration="23.7376461s" totalImagesPullingTime="19.010935477s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-31 23:46:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 23:46:40.737567691 +0000 UTC m=+24.705318452" watchObservedRunningTime="2026-07-31 23:46:40.7376461 +0000 UTC m=+24.705396850"
Jul 31 23:46:40.858055 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:40.858015    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gttdm_b32e6d42-e4e0-4113-b8ce-f5d5c5dc13f7/konnectivity-agent/0.log" limit=1024
Jul 31 23:46:41.206254 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:41.206213    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gttdm_b32e6d42-e4e0-4113-b8ce-f5d5c5dc13f7/konnectivity-agent/0.log" limit=1024
Jul 31 23:46:41.306853 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:41.306828    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gttdm_b32e6d42-e4e0-4113-b8ce-f5d5c5dc13f7/konnectivity-agent/0.log" limit=1024
Jul 31 23:46:41.484125 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:41.484091    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8d9rr"
Jul 31 23:46:41.484604 ip-10-0-136-108 kubenswrapper[2578]: E0731 23:46:41.484254    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8d9rr" podUID="0503a101-ddee-440a-ae03-e76fe30685e7"
Jul 31 23:46:41.728707 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:41.728622    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z94nv" event={"ID":"66c4c4e9-d9f1-4a76-b937-cc3b1e16cee8","Type":"ContainerStarted","Data":"95a80f1462902f8b0c133e39cbc9516640adf2078d8a1ddb4caa3f79fa374ebf"}
Jul 31 23:46:41.732652 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:41.732623    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-m9px5" event={"ID":"8d650e41-5af5-435a-81ab-aaf3898a490c","Type":"ContainerStarted","Data":"e3cacc94968e9aceeed8603f98098ff5281dc5339dad747ed145b35356344a87"}
Jul 31 23:46:41.732809 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:41.732656    2578 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness"
Jul 31 23:46:41.747111 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:41.747061    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z94nv" podStartSLOduration=3.588178742 podStartE2EDuration="24.747041676s" totalImagesPullingTime="21.158862934s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-31 23:46:17 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 23:46:41.746456308 +0000 UTC m=+25.714207068" watchObservedRunningTime="2026-07-31 23:46:41.747041676 +0000 UTC m=+25.714792433"
Jul 31 23:46:42.483697 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:42.483487    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qs7p7"
Jul 31 23:46:42.483953 ip-10-0-136-108 kubenswrapper[2578]: E0731 23:46:42.483801    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qs7p7" podUID="087b0a99-ae24-49e2-ba59-55651731fd60"
Jul 31 23:46:42.483953 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:42.483825    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xlv68"
Jul 31 23:46:42.483953 ip-10-0-136-108 kubenswrapper[2578]: E0731 23:46:42.483913    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xlv68" podUID="9d113c99-25bd-49a4-b8e2-f5c46f3ecc02"
Jul 31 23:46:42.494418 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:42.494377    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gttdm_b32e6d42-e4e0-4113-b8ce-f5d5c5dc13f7/konnectivity-agent/0.log" limit=1024
Jul 31 23:46:43.483789 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:43.483757    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8d9rr"
Jul 31 23:46:43.484001 ip-10-0-136-108 kubenswrapper[2578]: E0731 23:46:43.483858    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8d9rr" podUID="0503a101-ddee-440a-ae03-e76fe30685e7"
Jul 31 23:46:43.691484 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:43.691392    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gttdm_b32e6d42-e4e0-4113-b8ce-f5d5c5dc13f7/konnectivity-agent/0.log" limit=1024
Jul 31 23:46:43.790103 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:43.790065    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gttdm_b32e6d42-e4e0-4113-b8ce-f5d5c5dc13f7/konnectivity-agent/0.log" limit=1024
Jul 31 23:46:43.954716 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:43.954642    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gttdm_b32e6d42-e4e0-4113-b8ce-f5d5c5dc13f7/konnectivity-agent/0.log" limit=1024
Jul 31 23:46:44.055562 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:44.055520    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gttdm_b32e6d42-e4e0-4113-b8ce-f5d5c5dc13f7/konnectivity-agent/0.log" limit=1024
Jul 31 23:46:44.483315 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:44.483106    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qs7p7"
Jul 31 23:46:44.483471 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:44.483124    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xlv68"
Jul 31 23:46:44.483471 ip-10-0-136-108 kubenswrapper[2578]: E0731 23:46:44.483408    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qs7p7" podUID="087b0a99-ae24-49e2-ba59-55651731fd60"
Jul 31 23:46:44.483544 ip-10-0-136-108 kubenswrapper[2578]: E0731 23:46:44.483485    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xlv68" podUID="9d113c99-25bd-49a4-b8e2-f5c46f3ecc02"
Jul 31 23:46:44.742276 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:44.742243    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-m9px5" event={"ID":"8d650e41-5af5-435a-81ab-aaf3898a490c","Type":"ContainerStarted","Data":"54a13df575dca6d4d32a8a71aa6c847c5e7274b620e0ed2cdfeba6b14a423d6d"}
Jul 31 23:46:44.743226 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:44.742528    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-m9px5"
Jul 31 23:46:44.743226 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:44.742602    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-m9px5"
Jul 31 23:46:44.743226 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:44.742649    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-m9px5"
Jul 31 23:46:44.744495 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:44.744467    2578 generic.go:356] "Generic (PLEG): container finished" podID="dfb5011f-b591-4203-87ce-b1c6d6ad16ce" containerID="ca3f5bf49957a47a80ee8521f18d99602707cefb2b044edec999c7486dd47dbe" exitCode=0
Jul 31 23:46:44.744585 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:44.744515    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-h8nq8" event={"ID":"dfb5011f-b591-4203-87ce-b1c6d6ad16ce","Type":"ContainerDied","Data":"ca3f5bf49957a47a80ee8521f18d99602707cefb2b044edec999c7486dd47dbe"}
Jul 31 23:46:44.757671 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:44.757644    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-m9px5"
Jul 31 23:46:44.757849 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:44.757831    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-m9px5"
Jul 31 23:46:44.759388 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:44.759341    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gttdm_b32e6d42-e4e0-4113-b8ce-f5d5c5dc13f7/konnectivity-agent/0.log" limit=1024
Jul 31 23:46:44.769327 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:44.769290    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-m9px5" podStartSLOduration=8.699528227 podStartE2EDuration="27.769276434s" totalImagesPullingTime="19.069748207s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-31 23:46:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 23:46:44.768790852 +0000 UTC m=+28.736541609" watchObservedRunningTime="2026-07-31 23:46:44.769276434 +0000 UTC m=+28.737027191"
Jul 31 23:46:44.821961 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:44.821922    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gttdm_b32e6d42-e4e0-4113-b8ce-f5d5c5dc13f7/konnectivity-agent/0.log" limit=1024
Jul 31 23:46:45.483822 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:45.483787    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8d9rr"
Jul 31 23:46:45.484095 ip-10-0-136-108 kubenswrapper[2578]: E0731 23:46:45.483909    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8d9rr" podUID="0503a101-ddee-440a-ae03-e76fe30685e7"
Jul 31 23:46:45.944195 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:45.944137    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-gttdm"
Jul 31 23:46:45.944645 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:45.944290    2578 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness"
Jul 31 23:46:45.944805 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:45.944785    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-gttdm"
Jul 31 23:46:46.006828 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:46.006797    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gttdm_b32e6d42-e4e0-4113-b8ce-f5d5c5dc13f7/konnectivity-agent/0.log" limit=1024
Jul 31 23:46:46.044372 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:46.044291    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-8d9rr"]
Jul 31 23:46:46.044533 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:46.044416    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8d9rr"
Jul 31 23:46:46.044533 ip-10-0-136-108 kubenswrapper[2578]: E0731 23:46:46.044514    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8d9rr" podUID="0503a101-ddee-440a-ae03-e76fe30685e7"
Jul 31 23:46:46.049098 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:46.049075    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-xlv68"]
Jul 31 23:46:46.049244 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:46.049192    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xlv68"
Jul 31 23:46:46.049371 ip-10-0-136-108 kubenswrapper[2578]: E0731 23:46:46.049318    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xlv68" podUID="9d113c99-25bd-49a4-b8e2-f5c46f3ecc02"
Jul 31 23:46:46.049728 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:46.049708    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-qs7p7"]
Jul 31 23:46:46.049819 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:46.049807    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qs7p7"
Jul 31 23:46:46.049909 ip-10-0-136-108 kubenswrapper[2578]: E0731 23:46:46.049893    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qs7p7" podUID="087b0a99-ae24-49e2-ba59-55651731fd60"
Jul 31 23:46:46.474004 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:46.473921    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gttdm_b32e6d42-e4e0-4113-b8ce-f5d5c5dc13f7/konnectivity-agent/0.log" limit=1024
Jul 31 23:46:46.539054 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:46.539008    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gttdm_b32e6d42-e4e0-4113-b8ce-f5d5c5dc13f7/konnectivity-agent/0.log" limit=1024
Jul 31 23:46:46.751841 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:46.751810    2578 generic.go:356] "Generic (PLEG): container finished" podID="dfb5011f-b591-4203-87ce-b1c6d6ad16ce" containerID="908a55f4bc8ea9b8a17d830208b946e9445ab22056e7ebc9ccc22379eb4f0527" exitCode=0
Jul 31 23:46:46.752004 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:46.751898    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-h8nq8" event={"ID":"dfb5011f-b591-4203-87ce-b1c6d6ad16ce","Type":"ContainerDied","Data":"908a55f4bc8ea9b8a17d830208b946e9445ab22056e7ebc9ccc22379eb4f0527"}
Jul 31 23:46:47.173729 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:47.173644    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gttdm_b32e6d42-e4e0-4113-b8ce-f5d5c5dc13f7/konnectivity-agent/0.log" limit=1024
Jul 31 23:46:47.274062 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:47.274031    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gttdm_b32e6d42-e4e0-4113-b8ce-f5d5c5dc13f7/konnectivity-agent/0.log" limit=1024
Jul 31 23:46:47.345404 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:47.345163    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gttdm_b32e6d42-e4e0-4113-b8ce-f5d5c5dc13f7/konnectivity-agent/0.log" limit=1024
Jul 31 23:46:47.483762 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:47.483707    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qs7p7"
Jul 31 23:46:47.483894 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:47.483833    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xlv68"
Jul 31 23:46:47.483894 ip-10-0-136-108 kubenswrapper[2578]: E0731 23:46:47.483840    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qs7p7" podUID="087b0a99-ae24-49e2-ba59-55651731fd60"
Jul 31 23:46:47.483894 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:47.483866    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8d9rr"
Jul 31 23:46:47.484050 ip-10-0-136-108 kubenswrapper[2578]: E0731 23:46:47.483936    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xlv68" podUID="9d113c99-25bd-49a4-b8e2-f5c46f3ecc02"
Jul 31 23:46:47.484050 ip-10-0-136-108 kubenswrapper[2578]: E0731 23:46:47.484004    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8d9rr" podUID="0503a101-ddee-440a-ae03-e76fe30685e7"
Jul 31 23:46:47.574438 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:47.574405    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gttdm_b32e6d42-e4e0-4113-b8ce-f5d5c5dc13f7/konnectivity-agent/0.log" limit=1024
Jul 31 23:46:47.748616 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:47.748591    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-wr2rk_67d8b75c-284a-4bdb-a560-65d472fb1515/dns-node-resolver/0.log"
Jul 31 23:46:47.990537 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:47.990509    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gttdm_b32e6d42-e4e0-4113-b8ce-f5d5c5dc13f7/konnectivity-agent/0.log" limit=1024
Jul 31 23:46:48.088377 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:48.088338    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gttdm_b32e6d42-e4e0-4113-b8ce-f5d5c5dc13f7/konnectivity-agent/0.log" limit=1024
Jul 31 23:46:48.265104 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:48.265020    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gttdm_b32e6d42-e4e0-4113-b8ce-f5d5c5dc13f7/konnectivity-agent/0.log" limit=1024
Jul 31 23:46:48.350398 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:48.350365    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gttdm_b32e6d42-e4e0-4113-b8ce-f5d5c5dc13f7/konnectivity-agent/0.log" limit=1024
Jul 31 23:46:48.549499 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:48.549430    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-9hxzf_51ec46bf-6903-4794-adc9-3f796e976899/node-ca/0.log"
Jul 31 23:46:48.672931 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:48.672892    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gttdm_b32e6d42-e4e0-4113-b8ce-f5d5c5dc13f7/konnectivity-agent/0.log" limit=1024
Jul 31 23:46:48.759246 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:48.759216    2578 generic.go:356] "Generic (PLEG): container finished" podID="dfb5011f-b591-4203-87ce-b1c6d6ad16ce" containerID="3295988d4d18db9fccc7ef668fc098297d48a7cdc457793abd16c4ae8a2583cb" exitCode=0
Jul 31 23:46:48.759421 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:48.759264    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-h8nq8" event={"ID":"dfb5011f-b591-4203-87ce-b1c6d6ad16ce","Type":"ContainerDied","Data":"3295988d4d18db9fccc7ef668fc098297d48a7cdc457793abd16c4ae8a2583cb"}
Jul 31 23:46:49.483736 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:49.483700    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xlv68"
Jul 31 23:46:49.483736 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:49.483733    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8d9rr"
Jul 31 23:46:49.484341 ip-10-0-136-108 kubenswrapper[2578]: E0731 23:46:49.483835    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xlv68" podUID="9d113c99-25bd-49a4-b8e2-f5c46f3ecc02"
Jul 31 23:46:49.484341 ip-10-0-136-108 kubenswrapper[2578]: E0731 23:46:49.483870    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8d9rr" podUID="0503a101-ddee-440a-ae03-e76fe30685e7"
Jul 31 23:46:49.484341 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:49.483918    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qs7p7"
Jul 31 23:46:49.484341 ip-10-0-136-108 kubenswrapper[2578]: E0731 23:46:49.483993    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qs7p7" podUID="087b0a99-ae24-49e2-ba59-55651731fd60"
Jul 31 23:46:49.911611 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:49.911571    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gttdm_b32e6d42-e4e0-4113-b8ce-f5d5c5dc13f7/konnectivity-agent/0.log" limit=1024
Jul 31 23:46:50.210920 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:50.210828    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/087b0a99-ae24-49e2-ba59-55651731fd60-metrics-certs\") pod \"network-metrics-daemon-qs7p7\" (UID: \"087b0a99-ae24-49e2-ba59-55651731fd60\") " pod="openshift-multus/network-metrics-daemon-qs7p7"
Jul 31 23:46:50.210920 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:50.210879    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/9d113c99-25bd-49a4-b8e2-f5c46f3ecc02-original-pull-secret\") pod \"global-pull-secret-syncer-xlv68\" (UID: \"9d113c99-25bd-49a4-b8e2-f5c46f3ecc02\") " pod="kube-system/global-pull-secret-syncer-xlv68"
Jul 31 23:46:50.211162 ip-10-0-136-108 kubenswrapper[2578]: E0731 23:46:50.211007    2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 31 23:46:50.211162 ip-10-0-136-108 kubenswrapper[2578]: E0731 23:46:50.211023    2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 31 23:46:50.211162 ip-10-0-136-108 kubenswrapper[2578]: E0731 23:46:50.211073    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9d113c99-25bd-49a4-b8e2-f5c46f3ecc02-original-pull-secret podName:9d113c99-25bd-49a4-b8e2-f5c46f3ecc02 nodeName:}" failed. No retries permitted until 2026-07-31 23:47:22.211055076 +0000 UTC m=+66.178805820 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/9d113c99-25bd-49a4-b8e2-f5c46f3ecc02-original-pull-secret") pod "global-pull-secret-syncer-xlv68" (UID: "9d113c99-25bd-49a4-b8e2-f5c46f3ecc02") : object "kube-system"/"original-pull-secret" not registered
Jul 31 23:46:50.211162 ip-10-0-136-108 kubenswrapper[2578]: E0731 23:46:50.211089    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/087b0a99-ae24-49e2-ba59-55651731fd60-metrics-certs podName:087b0a99-ae24-49e2-ba59-55651731fd60 nodeName:}" failed. No retries permitted until 2026-07-31 23:47:22.211082195 +0000 UTC m=+66.178832930 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/087b0a99-ae24-49e2-ba59-55651731fd60-metrics-certs") pod "network-metrics-daemon-qs7p7" (UID: "087b0a99-ae24-49e2-ba59-55651731fd60") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 31 23:46:50.311744 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:50.311709    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-94n2k\" (UniqueName: \"kubernetes.io/projected/0503a101-ddee-440a-ae03-e76fe30685e7-kube-api-access-94n2k\") pod \"network-check-target-8d9rr\" (UID: \"0503a101-ddee-440a-ae03-e76fe30685e7\") " pod="openshift-network-diagnostics/network-check-target-8d9rr"
Jul 31 23:46:50.311907 ip-10-0-136-108 kubenswrapper[2578]: E0731 23:46:50.311845    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 31 23:46:50.311907 ip-10-0-136-108 kubenswrapper[2578]: E0731 23:46:50.311860    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 31 23:46:50.311907 ip-10-0-136-108 kubenswrapper[2578]: E0731 23:46:50.311870    2578 projected.go:196] Error preparing data for projected volume kube-api-access-94n2k for pod openshift-network-diagnostics/network-check-target-8d9rr: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 31 23:46:50.312033 ip-10-0-136-108 kubenswrapper[2578]: E0731 23:46:50.311921    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0503a101-ddee-440a-ae03-e76fe30685e7-kube-api-access-94n2k podName:0503a101-ddee-440a-ae03-e76fe30685e7 nodeName:}" failed. No retries permitted until 2026-07-31 23:47:22.311908327 +0000 UTC m=+66.279659063 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-94n2k" (UniqueName: "kubernetes.io/projected/0503a101-ddee-440a-ae03-e76fe30685e7-kube-api-access-94n2k") pod "network-check-target-8d9rr" (UID: "0503a101-ddee-440a-ae03-e76fe30685e7") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 31 23:46:50.474989 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:50.474911    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gttdm_b32e6d42-e4e0-4113-b8ce-f5d5c5dc13f7/konnectivity-agent/0.log" limit=1024
Jul 31 23:46:50.575942 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:50.575856    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gttdm_b32e6d42-e4e0-4113-b8ce-f5d5c5dc13f7/konnectivity-agent/0.log" limit=1024
Jul 31 23:46:50.836770 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:50.836714    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gttdm_b32e6d42-e4e0-4113-b8ce-f5d5c5dc13f7/konnectivity-agent/0.log" limit=1024
Jul 31 23:46:51.483931 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:51.483900    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qs7p7"
Jul 31 23:46:51.484111 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:51.483944    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8d9rr"
Jul 31 23:46:51.484111 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:51.483944    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xlv68"
Jul 31 23:46:51.484111 ip-10-0-136-108 kubenswrapper[2578]: E0731 23:46:51.484011    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qs7p7" podUID="087b0a99-ae24-49e2-ba59-55651731fd60"
Jul 31 23:46:51.484252 ip-10-0-136-108 kubenswrapper[2578]: E0731 23:46:51.484127    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xlv68" podUID="9d113c99-25bd-49a4-b8e2-f5c46f3ecc02"
Jul 31 23:46:51.484252 ip-10-0-136-108 kubenswrapper[2578]: E0731 23:46:51.484220    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8d9rr" podUID="0503a101-ddee-440a-ae03-e76fe30685e7"
Jul 31 23:46:51.855063 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:51.855032    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gttdm_b32e6d42-e4e0-4113-b8ce-f5d5c5dc13f7/konnectivity-agent/0.log" limit=1024
Jul 31 23:46:52.101321 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:52.101273    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gttdm_b32e6d42-e4e0-4113-b8ce-f5d5c5dc13f7/konnectivity-agent/0.log" limit=1024
Jul 31 23:46:52.193068 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:52.192976    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gttdm_b32e6d42-e4e0-4113-b8ce-f5d5c5dc13f7/konnectivity-agent/0.log" limit=1024
Jul 31 23:46:52.294128 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:52.294096    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gttdm_b32e6d42-e4e0-4113-b8ce-f5d5c5dc13f7/konnectivity-agent/0.log" limit=1024
Jul 31 23:46:52.657890 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:52.657851    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gttdm_b32e6d42-e4e0-4113-b8ce-f5d5c5dc13f7/konnectivity-agent/0.log" limit=1024
Jul 31 23:46:53.433755 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:53.433723    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gttdm_b32e6d42-e4e0-4113-b8ce-f5d5c5dc13f7/konnectivity-agent/0.log" limit=1024
Jul 31 23:46:53.483888 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:53.483855    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xlv68"
Jul 31 23:46:53.483888 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:53.483867    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qs7p7"
Jul 31 23:46:53.484074 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:53.483952    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8d9rr"
Jul 31 23:46:53.484074 ip-10-0-136-108 kubenswrapper[2578]: E0731 23:46:53.483961    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xlv68" podUID="9d113c99-25bd-49a4-b8e2-f5c46f3ecc02"
Jul 31 23:46:53.484074 ip-10-0-136-108 kubenswrapper[2578]: E0731 23:46:53.484033    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qs7p7" podUID="087b0a99-ae24-49e2-ba59-55651731fd60"
Jul 31 23:46:53.484201 ip-10-0-136-108 kubenswrapper[2578]: E0731 23:46:53.484085    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8d9rr" podUID="0503a101-ddee-440a-ae03-e76fe30685e7"
Jul 31 23:46:53.760043 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:53.760010    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gttdm_b32e6d42-e4e0-4113-b8ce-f5d5c5dc13f7/konnectivity-agent/0.log" limit=1024
Jul 31 23:46:53.774119 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:53.774085    2578 generic.go:356] "Generic (PLEG): container finished" podID="dfb5011f-b591-4203-87ce-b1c6d6ad16ce" containerID="203a88e3fd7374d09152668786b73fe04b043a0d11256d2aa80c3e7e2ffd84e7" exitCode=0
Jul 31 23:46:53.774309 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:53.774147    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-h8nq8" event={"ID":"dfb5011f-b591-4203-87ce-b1c6d6ad16ce","Type":"ContainerDied","Data":"203a88e3fd7374d09152668786b73fe04b043a0d11256d2aa80c3e7e2ffd84e7"}
Jul 31 23:46:53.839931 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:53.839902    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gttdm_b32e6d42-e4e0-4113-b8ce-f5d5c5dc13f7/konnectivity-agent/0.log" limit=1024
Jul 31 23:46:53.953757 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:53.953720    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gttdm_b32e6d42-e4e0-4113-b8ce-f5d5c5dc13f7/konnectivity-agent/0.log" limit=1024
Jul 31 23:46:54.365708 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:54.365671    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gttdm_b32e6d42-e4e0-4113-b8ce-f5d5c5dc13f7/konnectivity-agent/0.log" limit=1024
Jul 31 23:46:54.778808 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:54.778773    2578 generic.go:356] "Generic (PLEG): container finished" podID="dfb5011f-b591-4203-87ce-b1c6d6ad16ce" containerID="b93429c46166e154bb07d9921219de9d3e195c00f8990c395c3b0affae0b4a3b" exitCode=0
Jul 31 23:46:54.779231 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:54.778828    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-h8nq8" event={"ID":"dfb5011f-b591-4203-87ce-b1c6d6ad16ce","Type":"ContainerDied","Data":"b93429c46166e154bb07d9921219de9d3e195c00f8990c395c3b0affae0b4a3b"}
Jul 31 23:46:54.934848 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:54.934654    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gttdm_b32e6d42-e4e0-4113-b8ce-f5d5c5dc13f7/konnectivity-agent/0.log" limit=1024
Jul 31 23:46:55.155037 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:55.154928    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gttdm_b32e6d42-e4e0-4113-b8ce-f5d5c5dc13f7/konnectivity-agent/0.log" limit=1024
Jul 31 23:46:55.477418 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:55.477333    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gttdm_b32e6d42-e4e0-4113-b8ce-f5d5c5dc13f7/konnectivity-agent/0.log" limit=1024
Jul 31 23:46:55.483216 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:55.483198    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8d9rr"
Jul 31 23:46:55.483216 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:55.483205    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xlv68"
Jul 31 23:46:55.483339 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:55.483287    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qs7p7"
Jul 31 23:46:55.483339 ip-10-0-136-108 kubenswrapper[2578]: E0731 23:46:55.483295    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8d9rr" podUID="0503a101-ddee-440a-ae03-e76fe30685e7"
Jul 31 23:46:55.483435 ip-10-0-136-108 kubenswrapper[2578]: E0731 23:46:55.483355    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xlv68" podUID="9d113c99-25bd-49a4-b8e2-f5c46f3ecc02"
Jul 31 23:46:55.483493 ip-10-0-136-108 kubenswrapper[2578]: E0731 23:46:55.483474    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qs7p7" podUID="087b0a99-ae24-49e2-ba59-55651731fd60"
Jul 31 23:46:55.578556 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:55.578516    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gttdm_b32e6d42-e4e0-4113-b8ce-f5d5c5dc13f7/konnectivity-agent/0.log" limit=1024
Jul 31 23:46:55.657938 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:55.657903    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gttdm_b32e6d42-e4e0-4113-b8ce-f5d5c5dc13f7/konnectivity-agent/0.log" limit=1024
Jul 31 23:46:55.760512 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:55.760477    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gttdm_b32e6d42-e4e0-4113-b8ce-f5d5c5dc13f7/konnectivity-agent/0.log" limit=1024
Jul 31 23:46:55.784443 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:55.784403    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-h8nq8" event={"ID":"dfb5011f-b591-4203-87ce-b1c6d6ad16ce","Type":"ContainerStarted","Data":"9fc141c4f870bfc26ecf68c91063d7d63e1e2f6172dff6365e8649a86e463647"}
Jul 31 23:46:55.824570 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:55.824510    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-h8nq8" podStartSLOduration=9.302605692 podStartE2EDuration="38.824495317s" totalImagesPullingTime="29.521889625s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-31 23:46:17 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 23:46:55.823571359 +0000 UTC m=+39.791322119" watchObservedRunningTime="2026-07-31 23:46:55.824495317 +0000 UTC m=+39.792246075"
Jul 31 23:46:55.852397 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:55.852367    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gttdm_b32e6d42-e4e0-4113-b8ce-f5d5c5dc13f7/konnectivity-agent/0.log" limit=1024
Jul 31 23:46:55.975715 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:55.975674    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gttdm_b32e6d42-e4e0-4113-b8ce-f5d5c5dc13f7/konnectivity-agent/0.log" limit=1024
Jul 31 23:46:56.371293 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:56.371257    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gttdm_b32e6d42-e4e0-4113-b8ce-f5d5c5dc13f7/konnectivity-agent/0.log" limit=1024
Jul 31 23:46:56.474681 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:56.474645    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gttdm_b32e6d42-e4e0-4113-b8ce-f5d5c5dc13f7/konnectivity-agent/0.log" limit=1024
Jul 31 23:46:57.151821 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:57.151789    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gttdm_b32e6d42-e4e0-4113-b8ce-f5d5c5dc13f7/konnectivity-agent/0.log" limit=1024
Jul 31 23:46:57.483773 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:57.483677    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xlv68"
Jul 31 23:46:57.483773 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:57.483720    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8d9rr"
Jul 31 23:46:57.483773 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:57.483721    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qs7p7"
Jul 31 23:46:57.484010 ip-10-0-136-108 kubenswrapper[2578]: E0731 23:46:57.483783    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xlv68" podUID="9d113c99-25bd-49a4-b8e2-f5c46f3ecc02"
Jul 31 23:46:57.484010 ip-10-0-136-108 kubenswrapper[2578]: E0731 23:46:57.483881    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qs7p7" podUID="087b0a99-ae24-49e2-ba59-55651731fd60"
Jul 31 23:46:57.484010 ip-10-0-136-108 kubenswrapper[2578]: E0731 23:46:57.483942    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8d9rr" podUID="0503a101-ddee-440a-ae03-e76fe30685e7"
Jul 31 23:46:57.495820 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:57.495791    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gttdm_b32e6d42-e4e0-4113-b8ce-f5d5c5dc13f7/konnectivity-agent/0.log" limit=1024
Jul 31 23:46:57.596018 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:57.595980    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gttdm_b32e6d42-e4e0-4113-b8ce-f5d5c5dc13f7/konnectivity-agent/0.log" limit=1024
Jul 31 23:46:57.743929 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:57.743896    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gttdm_b32e6d42-e4e0-4113-b8ce-f5d5c5dc13f7/konnectivity-agent/0.log" limit=1024
Jul 31 23:46:57.948951 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:57.948915    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gttdm_b32e6d42-e4e0-4113-b8ce-f5d5c5dc13f7/konnectivity-agent/0.log" limit=1024
Jul 31 23:46:58.135125 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:58.135030    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gttdm_b32e6d42-e4e0-4113-b8ce-f5d5c5dc13f7/konnectivity-agent/0.log" limit=1024
Jul 31 23:46:58.368691 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:58.368660    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gttdm_b32e6d42-e4e0-4113-b8ce-f5d5c5dc13f7/konnectivity-agent/0.log" limit=1024
Jul 31 23:46:58.755387 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:58.755351    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gttdm_b32e6d42-e4e0-4113-b8ce-f5d5c5dc13f7/konnectivity-agent/0.log" limit=1024
Jul 31 23:46:58.959236 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:58.959201    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gttdm_b32e6d42-e4e0-4113-b8ce-f5d5c5dc13f7/konnectivity-agent/0.log" limit=1024
Jul 31 23:46:59.026593 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:59.026507    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gttdm_b32e6d42-e4e0-4113-b8ce-f5d5c5dc13f7/konnectivity-agent/0.log" limit=1024
Jul 31 23:46:59.483918 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:59.483831    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xlv68"
Jul 31 23:46:59.484323 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:59.483832    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qs7p7"
Jul 31 23:46:59.484323 ip-10-0-136-108 kubenswrapper[2578]: E0731 23:46:59.483937    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xlv68" podUID="9d113c99-25bd-49a4-b8e2-f5c46f3ecc02"
Jul 31 23:46:59.484323 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:59.483958    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8d9rr"
Jul 31 23:46:59.484323 ip-10-0-136-108 kubenswrapper[2578]: E0731 23:46:59.484029    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qs7p7" podUID="087b0a99-ae24-49e2-ba59-55651731fd60"
Jul 31 23:46:59.484323 ip-10-0-136-108 kubenswrapper[2578]: E0731 23:46:59.484104    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8d9rr" podUID="0503a101-ddee-440a-ae03-e76fe30685e7"
Jul 31 23:46:59.550877 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:59.550841    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gttdm_b32e6d42-e4e0-4113-b8ce-f5d5c5dc13f7/konnectivity-agent/0.log" limit=1024
Jul 31 23:46:59.753293 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:59.753260    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gttdm_b32e6d42-e4e0-4113-b8ce-f5d5c5dc13f7/konnectivity-agent/0.log" limit=1024
Jul 31 23:46:59.984193 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:46:59.984148    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gttdm_b32e6d42-e4e0-4113-b8ce-f5d5c5dc13f7/konnectivity-agent/0.log" limit=1024
Jul 31 23:47:00.148582 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:47:00.148482    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gttdm_b32e6d42-e4e0-4113-b8ce-f5d5c5dc13f7/konnectivity-agent/0.log" limit=1024
Jul 31 23:47:01.150982 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:47:01.150951    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gttdm_b32e6d42-e4e0-4113-b8ce-f5d5c5dc13f7/konnectivity-agent/0.log" limit=1024
Jul 31 23:47:01.353678 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:47:01.353642    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gttdm_b32e6d42-e4e0-4113-b8ce-f5d5c5dc13f7/konnectivity-agent/0.log" limit=1024
Jul 31 23:47:01.483735 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:47:01.483650    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xlv68"
Jul 31 23:47:01.483880 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:47:01.483654    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8d9rr"
Jul 31 23:47:01.483880 ip-10-0-136-108 kubenswrapper[2578]: E0731 23:47:01.483757    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xlv68" podUID="9d113c99-25bd-49a4-b8e2-f5c46f3ecc02"
Jul 31 23:47:01.483880 ip-10-0-136-108 kubenswrapper[2578]: E0731 23:47:01.483839    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8d9rr" podUID="0503a101-ddee-440a-ae03-e76fe30685e7"
Jul 31 23:47:01.483880 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:47:01.483882    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qs7p7"
Jul 31 23:47:01.484005 ip-10-0-136-108 kubenswrapper[2578]: E0731 23:47:01.483943    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qs7p7" podUID="087b0a99-ae24-49e2-ba59-55651731fd60"
Jul 31 23:47:01.745877 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:47:01.745843    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gttdm_b32e6d42-e4e0-4113-b8ce-f5d5c5dc13f7/konnectivity-agent/0.log" limit=1024
Jul 31 23:47:01.846437 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:47:01.846408    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gttdm_b32e6d42-e4e0-4113-b8ce-f5d5c5dc13f7/konnectivity-agent/0.log" limit=1024
Jul 31 23:47:01.902794 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:47:01.902755    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gttdm_b32e6d42-e4e0-4113-b8ce-f5d5c5dc13f7/konnectivity-agent/0.log" limit=1024
Jul 31 23:47:01.965095 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:47:01.965057    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gttdm_b32e6d42-e4e0-4113-b8ce-f5d5c5dc13f7/konnectivity-agent/0.log" limit=1024
Jul 31 23:47:02.028018 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:47:02.027937    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gttdm_b32e6d42-e4e0-4113-b8ce-f5d5c5dc13f7/konnectivity-agent/0.log" limit=1024
Jul 31 23:47:02.350134 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:47:02.350045    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gttdm_b32e6d42-e4e0-4113-b8ce-f5d5c5dc13f7/konnectivity-agent/0.log" limit=1024
Jul 31 23:47:02.562662 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:47:02.562620    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gttdm_b32e6d42-e4e0-4113-b8ce-f5d5c5dc13f7/konnectivity-agent/0.log" limit=1024
Jul 31 23:47:03.483426 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:47:03.483393    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xlv68"
Jul 31 23:47:03.483806 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:47:03.483436    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8d9rr"
Jul 31 23:47:03.483806 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:47:03.483393    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qs7p7"
Jul 31 23:47:03.483806 ip-10-0-136-108 kubenswrapper[2578]: E0731 23:47:03.483514    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xlv68" podUID="9d113c99-25bd-49a4-b8e2-f5c46f3ecc02"
Jul 31 23:47:03.483806 ip-10-0-136-108 kubenswrapper[2578]: E0731 23:47:03.483572    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qs7p7" podUID="087b0a99-ae24-49e2-ba59-55651731fd60"
Jul 31 23:47:03.483806 ip-10-0-136-108 kubenswrapper[2578]: E0731 23:47:03.483621    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8d9rr" podUID="0503a101-ddee-440a-ae03-e76fe30685e7"
Jul 31 23:47:05.483743 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:47:05.483658    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qs7p7"
Jul 31 23:47:05.483743 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:47:05.483698    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xlv68"
Jul 31 23:47:05.484122 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:47:05.483659    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8d9rr"
Jul 31 23:47:05.484122 ip-10-0-136-108 kubenswrapper[2578]: E0731 23:47:05.483772    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qs7p7" podUID="087b0a99-ae24-49e2-ba59-55651731fd60"
Jul 31 23:47:05.484122 ip-10-0-136-108 kubenswrapper[2578]: E0731 23:47:05.483860    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8d9rr" podUID="0503a101-ddee-440a-ae03-e76fe30685e7"
Jul 31 23:47:05.484122 ip-10-0-136-108 kubenswrapper[2578]: E0731 23:47:05.483945    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xlv68" podUID="9d113c99-25bd-49a4-b8e2-f5c46f3ecc02"
Jul 31 23:47:05.808112 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:47:05.808082    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gttdm_b32e6d42-e4e0-4113-b8ce-f5d5c5dc13f7/konnectivity-agent/0.log" limit=1024
Jul 31 23:47:05.909881 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:47:05.909846    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gttdm_b32e6d42-e4e0-4113-b8ce-f5d5c5dc13f7/konnectivity-agent/0.log" limit=1024
Jul 31 23:47:05.965819 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:47:05.965789    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gttdm_b32e6d42-e4e0-4113-b8ce-f5d5c5dc13f7/konnectivity-agent/0.log" limit=1024
Jul 31 23:47:06.106978 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:47:06.106892    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gttdm_b32e6d42-e4e0-4113-b8ce-f5d5c5dc13f7/konnectivity-agent/0.log" limit=1024
Jul 31 23:47:06.278114 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:47:06.278082    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gttdm_b32e6d42-e4e0-4113-b8ce-f5d5c5dc13f7/konnectivity-agent/0.log" limit=1024
Jul 31 23:47:06.341326 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:47:06.341288    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gttdm_b32e6d42-e4e0-4113-b8ce-f5d5c5dc13f7/konnectivity-agent/0.log" limit=1024
Jul 31 23:47:06.481858 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:47:06.481772    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gttdm_b32e6d42-e4e0-4113-b8ce-f5d5c5dc13f7/konnectivity-agent/0.log" limit=1024
Jul 31 23:47:06.541498 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:47:06.541453    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gttdm_b32e6d42-e4e0-4113-b8ce-f5d5c5dc13f7/konnectivity-agent/0.log" limit=1024
Jul 31 23:47:06.683904 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:47:06.683869    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gttdm_b32e6d42-e4e0-4113-b8ce-f5d5c5dc13f7/konnectivity-agent/0.log" limit=1024
Jul 31 23:47:06.759229 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:47:06.759163    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gttdm_b32e6d42-e4e0-4113-b8ce-f5d5c5dc13f7/konnectivity-agent/0.log" limit=1024
Jul 31 23:47:06.893658 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:47:06.893623    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gttdm_b32e6d42-e4e0-4113-b8ce-f5d5c5dc13f7/konnectivity-agent/0.log" limit=1024
Jul 31 23:47:07.294746 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:47:07.294711    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gttdm_b32e6d42-e4e0-4113-b8ce-f5d5c5dc13f7/konnectivity-agent/0.log" limit=1024
Jul 31 23:47:07.483957 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:47:07.483919    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8d9rr"
Jul 31 23:47:07.483957 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:47:07.483964    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xlv68"
Jul 31 23:47:07.484165 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:47:07.483971    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qs7p7"
Jul 31 23:47:07.484165 ip-10-0-136-108 kubenswrapper[2578]: E0731 23:47:07.484048    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8d9rr" podUID="0503a101-ddee-440a-ae03-e76fe30685e7"
Jul 31 23:47:07.484446 ip-10-0-136-108 kubenswrapper[2578]: E0731 23:47:07.484191    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xlv68" podUID="9d113c99-25bd-49a4-b8e2-f5c46f3ecc02"
Jul 31 23:47:07.484446 ip-10-0-136-108 kubenswrapper[2578]: E0731 23:47:07.484293    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qs7p7" podUID="087b0a99-ae24-49e2-ba59-55651731fd60"
Jul 31 23:47:07.490360 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:47:07.490331    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gttdm_b32e6d42-e4e0-4113-b8ce-f5d5c5dc13f7/konnectivity-agent/0.log" limit=1024
Jul 31 23:47:07.735671 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:47:07.735587    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gttdm_b32e6d42-e4e0-4113-b8ce-f5d5c5dc13f7/konnectivity-agent/0.log" limit=1024
Jul 31 23:47:08.718269 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:47:08.718222    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gttdm_b32e6d42-e4e0-4113-b8ce-f5d5c5dc13f7/konnectivity-agent/0.log" limit=1024
Jul 31 23:47:08.815081 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:47:08.815044    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gttdm_b32e6d42-e4e0-4113-b8ce-f5d5c5dc13f7/konnectivity-agent/0.log" limit=1024
Jul 31 23:47:09.483587 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:47:09.483554    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8d9rr"
Jul 31 23:47:09.483763 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:47:09.483597    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xlv68"
Jul 31 23:47:09.483763 ip-10-0-136-108 kubenswrapper[2578]: E0731 23:47:09.483657    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8d9rr" podUID="0503a101-ddee-440a-ae03-e76fe30685e7"
Jul 31 23:47:09.483763 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:47:09.483597    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qs7p7"
Jul 31 23:47:09.483763 ip-10-0-136-108 kubenswrapper[2578]: E0731 23:47:09.483693    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xlv68" podUID="9d113c99-25bd-49a4-b8e2-f5c46f3ecc02"
Jul 31 23:47:09.483903 ip-10-0-136-108 kubenswrapper[2578]: E0731 23:47:09.483778    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qs7p7" podUID="087b0a99-ae24-49e2-ba59-55651731fd60"
Jul 31 23:47:11.483933 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:47:11.483897    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qs7p7"
Jul 31 23:47:11.484407 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:47:11.483903    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8d9rr"
Jul 31 23:47:11.484407 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:47:11.484032    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xlv68"
Jul 31 23:47:11.484407 ip-10-0-136-108 kubenswrapper[2578]: E0731 23:47:11.484094    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8d9rr" podUID="0503a101-ddee-440a-ae03-e76fe30685e7"
Jul 31 23:47:11.484407 ip-10-0-136-108 kubenswrapper[2578]: E0731 23:47:11.484004    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qs7p7" podUID="087b0a99-ae24-49e2-ba59-55651731fd60"
Jul 31 23:47:11.484407 ip-10-0-136-108 kubenswrapper[2578]: E0731 23:47:11.484200    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xlv68" podUID="9d113c99-25bd-49a4-b8e2-f5c46f3ecc02"
Jul 31 23:47:12.303613 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:47:12.303575    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gttdm_b32e6d42-e4e0-4113-b8ce-f5d5c5dc13f7/konnectivity-agent/0.log" limit=1024
Jul 31 23:47:12.415324 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:47:12.415292    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gttdm_b32e6d42-e4e0-4113-b8ce-f5d5c5dc13f7/konnectivity-agent/0.log" limit=1024
Jul 31 23:47:12.519665 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:47:12.519626    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gttdm_b32e6d42-e4e0-4113-b8ce-f5d5c5dc13f7/konnectivity-agent/0.log" limit=1024
Jul 31 23:47:12.612782 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:47:12.612703    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gttdm_b32e6d42-e4e0-4113-b8ce-f5d5c5dc13f7/konnectivity-agent/0.log" limit=1024
Jul 31 23:47:13.421005 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:47:13.420926    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-108.ec2.internal" event="NodeReady"
Jul 31 23:47:13.421147 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:47:13.421044    2578 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Jul 31 23:47:13.483661 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:47:13.483632    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8d9rr"
Jul 31 23:47:13.483844 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:47:13.483634    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qs7p7"
Jul 31 23:47:13.483844 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:47:13.483634    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xlv68"
Jul 31 23:47:13.488036 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:47:13.488017    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-7ftsc"]
Jul 31 23:47:13.488318 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:47:13.488287    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 31 23:47:13.488452 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:47:13.488403    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 31 23:47:13.488790 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:47:13.488768    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 31 23:47:13.488790 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:47:13.488776    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-4hvkj\""
Jul 31 23:47:13.488962 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:47:13.488782    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 31 23:47:13.488962 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:47:13.488768    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-bfr2g\""
Jul 31 23:47:13.490159 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:47:13.490139    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-v2wkd"]
Jul 31 23:47:13.490276 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:47:13.490258    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-7ftsc"
Jul 31 23:47:13.492143 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:47:13.492127    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Jul 31 23:47:13.492331 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:47:13.492315    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-srbwg\""
Jul 31 23:47:13.492389 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:47:13.492371    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-v2wkd"
Jul 31 23:47:13.492471 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:47:13.492381    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Jul 31 23:47:13.492523 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:47:13.492319    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Jul 31 23:47:13.494572 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:47:13.494556    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Jul 31 23:47:13.495145 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:47:13.495129    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Jul 31 23:47:13.495230 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:47:13.495140    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Jul 31 23:47:13.495405 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:47:13.495385    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Jul 31 23:47:13.495760 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:47:13.495748    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-g252s\""
Jul 31 23:47:13.525401 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:47:13.525369    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-7ftsc"]
Jul 31 23:47:13.525949 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:47:13.525929    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-v2wkd"]
Jul 31 23:47:13.526460 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:47:13.526436    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-cjvb8"]
Jul 31 23:47:13.528805 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:47:13.528779    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-cjvb8"
Jul 31 23:47:13.530598 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:47:13.530564    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-x95hq\""
Jul 31 23:47:13.530709 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:47:13.530564    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Jul 31 23:47:13.530875 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:47:13.530854    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Jul 31 23:47:13.541086 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:47:13.541037    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-cjvb8"]
Jul 31 23:47:13.570492 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:47:13.570457    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gttdm_b32e6d42-e4e0-4113-b8ce-f5d5c5dc13f7/konnectivity-agent/0.log" limit=1024
Jul 31 23:47:13.571327 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:47:13.571301    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/194c7e46-9a88-4078-a687-63112c8ccab5-data-volume\") pod \"insights-runtime-extractor-v2wkd\" (UID: \"194c7e46-9a88-4078-a687-63112c8ccab5\") " pod="openshift-insights/insights-runtime-extractor-v2wkd"
Jul 31 23:47:13.571402 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:47:13.571389    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fp8nf\" (UniqueName: \"kubernetes.io/projected/2bcd7dcd-ab90-47bd-8747-5f1fafe9960c-kube-api-access-fp8nf\") pod \"ingress-canary-7ftsc\" (UID: \"2bcd7dcd-ab90-47bd-8747-5f1fafe9960c\") " pod="openshift-ingress-canary/ingress-canary-7ftsc"
Jul 31 23:47:13.571445 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:47:13.571424    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/194c7e46-9a88-4078-a687-63112c8ccab5-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-v2wkd\" (UID: \"194c7e46-9a88-4078-a687-63112c8ccab5\") " pod="openshift-insights/insights-runtime-extractor-v2wkd"
Jul 31 23:47:13.571445 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:47:13.571442    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-txjtr\" (UniqueName: \"kubernetes.io/projected/194c7e46-9a88-4078-a687-63112c8ccab5-kube-api-access-txjtr\") pod \"insights-runtime-extractor-v2wkd\" (UID: \"194c7e46-9a88-4078-a687-63112c8ccab5\") " pod="openshift-insights/insights-runtime-extractor-v2wkd"
Jul 31 23:47:13.571524 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:47:13.571465    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/759fe27f-5715-4033-bb09-a5f5a21d7c53-tmp-dir\") pod \"dns-default-cjvb8\" (UID: \"759fe27f-5715-4033-bb09-a5f5a21d7c53\") " pod="openshift-dns/dns-default-cjvb8"
Jul 31 23:47:13.571565 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:47:13.571533    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/759fe27f-5715-4033-bb09-a5f5a21d7c53-config-volume\") pod \"dns-default-cjvb8\" (UID: \"759fe27f-5715-4033-bb09-a5f5a21d7c53\") " pod="openshift-dns/dns-default-cjvb8"
Jul 31 23:47:13.571565 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:47:13.571559    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/759fe27f-5715-4033-bb09-a5f5a21d7c53-metrics-tls\") pod \"dns-default-cjvb8\" (UID: \"759fe27f-5715-4033-bb09-a5f5a21d7c53\") " pod="openshift-dns/dns-default-cjvb8"
Jul 31 23:47:13.571631 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:47:13.571623    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/194c7e46-9a88-4078-a687-63112c8ccab5-crio-socket\") pod \"insights-runtime-extractor-v2wkd\" (UID: \"194c7e46-9a88-4078-a687-63112c8ccab5\") " pod="openshift-insights/insights-runtime-extractor-v2wkd"
Jul 31 23:47:13.571670 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:47:13.571639    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/194c7e46-9a88-4078-a687-63112c8ccab5-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-v2wkd\" (UID: \"194c7e46-9a88-4078-a687-63112c8ccab5\") " pod="openshift-insights/insights-runtime-extractor-v2wkd"
Jul 31 23:47:13.571670 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:47:13.571658    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/2bcd7dcd-ab90-47bd-8747-5f1fafe9960c-cert\") pod \"ingress-canary-7ftsc\" (UID: \"2bcd7dcd-ab90-47bd-8747-5f1fafe9960c\") " pod="openshift-ingress-canary/ingress-canary-7ftsc"
Jul 31 23:47:13.571742 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:47:13.571693    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9hpfb\" (UniqueName: \"kubernetes.io/projected/759fe27f-5715-4033-bb09-a5f5a21d7c53-kube-api-access-9hpfb\") pod \"dns-default-cjvb8\" (UID: \"759fe27f-5715-4033-bb09-a5f5a21d7c53\") " pod="openshift-dns/dns-default-cjvb8"
Jul 31 23:47:13.657569 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:47:13.657537    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gttdm_b32e6d42-e4e0-4113-b8ce-f5d5c5dc13f7/konnectivity-agent/0.log" limit=1024
Jul 31 23:47:13.672609 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:47:13.672553    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fp8nf\" (UniqueName: \"kubernetes.io/projected/2bcd7dcd-ab90-47bd-8747-5f1fafe9960c-kube-api-access-fp8nf\") pod \"ingress-canary-7ftsc\" (UID: \"2bcd7dcd-ab90-47bd-8747-5f1fafe9960c\") " pod="openshift-ingress-canary/ingress-canary-7ftsc"
Jul 31 23:47:13.672609 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:47:13.672597    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/194c7e46-9a88-4078-a687-63112c8ccab5-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-v2wkd\" (UID: \"194c7e46-9a88-4078-a687-63112c8ccab5\") " pod="openshift-insights/insights-runtime-extractor-v2wkd"
Jul 31 23:47:13.672731 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:47:13.672623    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-txjtr\" (UniqueName: \"kubernetes.io/projected/194c7e46-9a88-4078-a687-63112c8ccab5-kube-api-access-txjtr\") pod \"insights-runtime-extractor-v2wkd\" (UID: \"194c7e46-9a88-4078-a687-63112c8ccab5\") " pod="openshift-insights/insights-runtime-extractor-v2wkd"
Jul 31 23:47:13.672731 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:47:13.672647    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/759fe27f-5715-4033-bb09-a5f5a21d7c53-tmp-dir\") pod \"dns-default-cjvb8\" (UID: \"759fe27f-5715-4033-bb09-a5f5a21d7c53\") " pod="openshift-dns/dns-default-cjvb8"
Jul 31 23:47:13.672731 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:47:13.672671    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/759fe27f-5715-4033-bb09-a5f5a21d7c53-config-volume\") pod \"dns-default-cjvb8\" (UID: \"759fe27f-5715-4033-bb09-a5f5a21d7c53\") " pod="openshift-dns/dns-default-cjvb8"
Jul 31 23:47:13.672731 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:47:13.672692    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/759fe27f-5715-4033-bb09-a5f5a21d7c53-metrics-tls\") pod \"dns-default-cjvb8\" (UID: \"759fe27f-5715-4033-bb09-a5f5a21d7c53\") " pod="openshift-dns/dns-default-cjvb8"
Jul 31 23:47:13.672921 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:47:13.672733    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/194c7e46-9a88-4078-a687-63112c8ccab5-crio-socket\") pod \"insights-runtime-extractor-v2wkd\" (UID: \"194c7e46-9a88-4078-a687-63112c8ccab5\") " pod="openshift-insights/insights-runtime-extractor-v2wkd"
Jul 31 23:47:13.672921 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:47:13.672755    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/194c7e46-9a88-4078-a687-63112c8ccab5-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-v2wkd\" (UID: \"194c7e46-9a88-4078-a687-63112c8ccab5\") " pod="openshift-insights/insights-runtime-extractor-v2wkd"
Jul 31 23:47:13.672921 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:47:13.672822    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/2bcd7dcd-ab90-47bd-8747-5f1fafe9960c-cert\") pod \"ingress-canary-7ftsc\" (UID: \"2bcd7dcd-ab90-47bd-8747-5f1fafe9960c\") " pod="openshift-ingress-canary/ingress-canary-7ftsc"
Jul 31 23:47:13.672921 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:47:13.672844    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/194c7e46-9a88-4078-a687-63112c8ccab5-crio-socket\") pod \"insights-runtime-extractor-v2wkd\" (UID: \"194c7e46-9a88-4078-a687-63112c8ccab5\") " pod="openshift-insights/insights-runtime-extractor-v2wkd"
Jul 31 23:47:13.672921 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:47:13.672872    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9hpfb\" (UniqueName: \"kubernetes.io/projected/759fe27f-5715-4033-bb09-a5f5a21d7c53-kube-api-access-9hpfb\") pod \"dns-default-cjvb8\" (UID: \"759fe27f-5715-4033-bb09-a5f5a21d7c53\") " pod="openshift-dns/dns-default-cjvb8"
Jul 31 23:47:13.672921 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:47:13.672917    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/194c7e46-9a88-4078-a687-63112c8ccab5-data-volume\") pod \"insights-runtime-extractor-v2wkd\" (UID: \"194c7e46-9a88-4078-a687-63112c8ccab5\") " pod="openshift-insights/insights-runtime-extractor-v2wkd"
Jul 31 23:47:13.673235 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:47:13.673008    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/759fe27f-5715-4033-bb09-a5f5a21d7c53-tmp-dir\") pod \"dns-default-cjvb8\" (UID: \"759fe27f-5715-4033-bb09-a5f5a21d7c53\") " pod="openshift-dns/dns-default-cjvb8"
Jul 31 23:47:13.673294 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:47:13.673233    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/194c7e46-9a88-4078-a687-63112c8ccab5-data-volume\") pod \"insights-runtime-extractor-v2wkd\" (UID: \"194c7e46-9a88-4078-a687-63112c8ccab5\") " pod="openshift-insights/insights-runtime-extractor-v2wkd"
Jul 31 23:47:13.673429 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:47:13.673405    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/194c7e46-9a88-4078-a687-63112c8ccab5-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-v2wkd\" (UID: \"194c7e46-9a88-4078-a687-63112c8ccab5\") " pod="openshift-insights/insights-runtime-extractor-v2wkd"
Jul 31 23:47:13.673484 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:47:13.673414    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/759fe27f-5715-4033-bb09-a5f5a21d7c53-config-volume\") pod \"dns-default-cjvb8\" (UID: \"759fe27f-5715-4033-bb09-a5f5a21d7c53\") " pod="openshift-dns/dns-default-cjvb8"
Jul 31 23:47:13.676576 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:47:13.676554    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/194c7e46-9a88-4078-a687-63112c8ccab5-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-v2wkd\" (UID: \"194c7e46-9a88-4078-a687-63112c8ccab5\") " pod="openshift-insights/insights-runtime-extractor-v2wkd"
Jul 31 23:47:13.676677 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:47:13.676621    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/2bcd7dcd-ab90-47bd-8747-5f1fafe9960c-cert\") pod \"ingress-canary-7ftsc\" (UID: \"2bcd7dcd-ab90-47bd-8747-5f1fafe9960c\") " pod="openshift-ingress-canary/ingress-canary-7ftsc"
Jul 31 23:47:13.676677 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:47:13.676635    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/759fe27f-5715-4033-bb09-a5f5a21d7c53-metrics-tls\") pod \"dns-default-cjvb8\" (UID: \"759fe27f-5715-4033-bb09-a5f5a21d7c53\") " pod="openshift-dns/dns-default-cjvb8"
Jul 31 23:47:13.679279 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:47:13.679254    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-txjtr\" (UniqueName: \"kubernetes.io/projected/194c7e46-9a88-4078-a687-63112c8ccab5-kube-api-access-txjtr\") pod \"insights-runtime-extractor-v2wkd\" (UID: \"194c7e46-9a88-4078-a687-63112c8ccab5\") " pod="openshift-insights/insights-runtime-extractor-v2wkd"
Jul 31 23:47:13.679461 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:47:13.679434    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fp8nf\" (UniqueName: \"kubernetes.io/projected/2bcd7dcd-ab90-47bd-8747-5f1fafe9960c-kube-api-access-fp8nf\") pod \"ingress-canary-7ftsc\" (UID: \"2bcd7dcd-ab90-47bd-8747-5f1fafe9960c\") " pod="openshift-ingress-canary/ingress-canary-7ftsc"
Jul 31 23:47:13.680019 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:47:13.679996    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9hpfb\" (UniqueName: \"kubernetes.io/projected/759fe27f-5715-4033-bb09-a5f5a21d7c53-kube-api-access-9hpfb\") pod \"dns-default-cjvb8\" (UID: \"759fe27f-5715-4033-bb09-a5f5a21d7c53\") " pod="openshift-dns/dns-default-cjvb8"
Jul 31 23:47:13.711039 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:47:13.711010    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gttdm_b32e6d42-e4e0-4113-b8ce-f5d5c5dc13f7/konnectivity-agent/0.log" limit=1024
Jul 31 23:47:13.812243 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:47:13.812208    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-7ftsc"
Jul 31 23:47:13.816505 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:47:13.816486    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-v2wkd"
Jul 31 23:47:13.837690 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:47:13.837662    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-cjvb8"
Jul 31 23:47:13.955577 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:47:13.955481    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gttdm_b32e6d42-e4e0-4113-b8ce-f5d5c5dc13f7/konnectivity-agent/0.log" limit=1024
Jul 31 23:47:13.963950 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:47:13.963790    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-7ftsc"]
Jul 31 23:47:13.969560 ip-10-0-136-108 kubenswrapper[2578]: W0731 23:47:13.969535    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2bcd7dcd_ab90_47bd_8747_5f1fafe9960c.slice/crio-8e35f23de850df7d0c06e210667853a70629dddeab7878964678303175de1509 WatchSource:0}: Error finding container 8e35f23de850df7d0c06e210667853a70629dddeab7878964678303175de1509: Status 404 returned error can't find the container with id 8e35f23de850df7d0c06e210667853a70629dddeab7878964678303175de1509
Jul 31 23:47:13.978348 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:47:13.978326    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-v2wkd"]
Jul 31 23:47:13.993511 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:47:13.993481    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-cjvb8"]
Jul 31 23:47:13.996803 ip-10-0-136-108 kubenswrapper[2578]: W0731 23:47:13.996778    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod759fe27f_5715_4033_bb09_a5f5a21d7c53.slice/crio-584747c5c64848f19363851f0e563645b3f315b25d758c05387c3849cc968bec WatchSource:0}: Error finding container 584747c5c64848f19363851f0e563645b3f315b25d758c05387c3849cc968bec: Status 404 returned error can't find the container with id 584747c5c64848f19363851f0e563645b3f315b25d758c05387c3849cc968bec
Jul 31 23:47:14.027635 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:47:14.027608    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gttdm_b32e6d42-e4e0-4113-b8ce-f5d5c5dc13f7/konnectivity-agent/0.log" limit=1024
Jul 31 23:47:14.126769 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:47:14.126736    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-5d59f9cb75-g6g2x"]
Jul 31 23:47:14.129281 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:47:14.129265    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5d59f9cb75-g6g2x"
Jul 31 23:47:14.131277 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:47:14.131251    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Jul 31 23:47:14.131416 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:47:14.131275    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-kcdp7\""
Jul 31 23:47:14.131416 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:47:14.131330    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Jul 31 23:47:14.131416 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:47:14.131366    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Jul 31 23:47:14.131416 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:47:14.131373    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Jul 31 23:47:14.131604 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:47:14.131579    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Jul 31 23:47:14.131697 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:47:14.131680    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Jul 31 23:47:14.131744 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:47:14.131710    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Jul 31 23:47:14.142316 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:47:14.142293    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5d59f9cb75-g6g2x"]
Jul 31 23:47:14.176779 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:47:14.176745    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/47326e9a-7028-4a90-94e4-70d55ac3290d-console-config\") pod \"console-5d59f9cb75-g6g2x\" (UID: \"47326e9a-7028-4a90-94e4-70d55ac3290d\") " pod="openshift-console/console-5d59f9cb75-g6g2x"
Jul 31 23:47:14.176916 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:47:14.176806    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/47326e9a-7028-4a90-94e4-70d55ac3290d-console-serving-cert\") pod \"console-5d59f9cb75-g6g2x\" (UID: \"47326e9a-7028-4a90-94e4-70d55ac3290d\") " pod="openshift-console/console-5d59f9cb75-g6g2x"
Jul 31 23:47:14.176916 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:47:14.176833    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/47326e9a-7028-4a90-94e4-70d55ac3290d-service-ca\") pod \"console-5d59f9cb75-g6g2x\" (UID: \"47326e9a-7028-4a90-94e4-70d55ac3290d\") " pod="openshift-console/console-5d59f9cb75-g6g2x"
Jul 31 23:47:14.176916 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:47:14.176856    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gpsh4\" (UniqueName: \"kubernetes.io/projected/47326e9a-7028-4a90-94e4-70d55ac3290d-kube-api-access-gpsh4\") pod \"console-5d59f9cb75-g6g2x\" (UID: \"47326e9a-7028-4a90-94e4-70d55ac3290d\") " pod="openshift-console/console-5d59f9cb75-g6g2x"
Jul 31 23:47:14.177033 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:47:14.176935    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/47326e9a-7028-4a90-94e4-70d55ac3290d-console-oauth-config\") pod \"console-5d59f9cb75-g6g2x\" (UID: \"47326e9a-7028-4a90-94e4-70d55ac3290d\") " pod="openshift-console/console-5d59f9cb75-g6g2x"
Jul 31 23:47:14.177033 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:47:14.176969    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/47326e9a-7028-4a90-94e4-70d55ac3290d-oauth-serving-cert\") pod \"console-5d59f9cb75-g6g2x\" (UID: \"47326e9a-7028-4a90-94e4-70d55ac3290d\") " pod="openshift-console/console-5d59f9cb75-g6g2x"
Jul 31 23:47:14.278117 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:47:14.278088    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/47326e9a-7028-4a90-94e4-70d55ac3290d-console-serving-cert\") pod \"console-5d59f9cb75-g6g2x\" (UID: \"47326e9a-7028-4a90-94e4-70d55ac3290d\") " pod="openshift-console/console-5d59f9cb75-g6g2x"
Jul 31 23:47:14.278117 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:47:14.278122    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/47326e9a-7028-4a90-94e4-70d55ac3290d-service-ca\") pod \"console-5d59f9cb75-g6g2x\" (UID: \"47326e9a-7028-4a90-94e4-70d55ac3290d\") " pod="openshift-console/console-5d59f9cb75-g6g2x"
Jul 31 23:47:14.278358 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:47:14.278142    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gpsh4\" (UniqueName: \"kubernetes.io/projected/47326e9a-7028-4a90-94e4-70d55ac3290d-kube-api-access-gpsh4\") pod \"console-5d59f9cb75-g6g2x\" (UID: \"47326e9a-7028-4a90-94e4-70d55ac3290d\") " pod="openshift-console/console-5d59f9cb75-g6g2x"
Jul 31 23:47:14.278358 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:47:14.278206    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/47326e9a-7028-4a90-94e4-70d55ac3290d-console-oauth-config\") pod \"console-5d59f9cb75-g6g2x\" (UID: \"47326e9a-7028-4a90-94e4-70d55ac3290d\") " pod="openshift-console/console-5d59f9cb75-g6g2x"
Jul 31 23:47:14.278467 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:47:14.278399    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/47326e9a-7028-4a90-94e4-70d55ac3290d-oauth-serving-cert\") pod \"console-5d59f9cb75-g6g2x\" (UID: \"47326e9a-7028-4a90-94e4-70d55ac3290d\") " pod="openshift-console/console-5d59f9cb75-g6g2x"
Jul 31 23:47:14.278467 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:47:14.278446    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/47326e9a-7028-4a90-94e4-70d55ac3290d-console-config\") pod \"console-5d59f9cb75-g6g2x\" (UID: \"47326e9a-7028-4a90-94e4-70d55ac3290d\") " pod="openshift-console/console-5d59f9cb75-g6g2x"
Jul 31 23:47:14.278941 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:47:14.278919    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/47326e9a-7028-4a90-94e4-70d55ac3290d-service-ca\") pod \"console-5d59f9cb75-g6g2x\" (UID: \"47326e9a-7028-4a90-94e4-70d55ac3290d\") " pod="openshift-console/console-5d59f9cb75-g6g2x"
Jul 31 23:47:14.279044 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:47:14.278963    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/47326e9a-7028-4a90-94e4-70d55ac3290d-oauth-serving-cert\") pod \"console-5d59f9cb75-g6g2x\" (UID: \"47326e9a-7028-4a90-94e4-70d55ac3290d\") " pod="openshift-console/console-5d59f9cb75-g6g2x"
Jul 31 23:47:14.279044 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:47:14.279009    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/47326e9a-7028-4a90-94e4-70d55ac3290d-console-config\") pod \"console-5d59f9cb75-g6g2x\" (UID: \"47326e9a-7028-4a90-94e4-70d55ac3290d\") " pod="openshift-console/console-5d59f9cb75-g6g2x"
Jul 31 23:47:14.281114 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:47:14.281094    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/47326e9a-7028-4a90-94e4-70d55ac3290d-console-oauth-config\") pod \"console-5d59f9cb75-g6g2x\" (UID: \"47326e9a-7028-4a90-94e4-70d55ac3290d\") " pod="openshift-console/console-5d59f9cb75-g6g2x"
Jul 31 23:47:14.281212 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:47:14.281103    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/47326e9a-7028-4a90-94e4-70d55ac3290d-console-serving-cert\") pod \"console-5d59f9cb75-g6g2x\" (UID: \"47326e9a-7028-4a90-94e4-70d55ac3290d\") " pod="openshift-console/console-5d59f9cb75-g6g2x"
Jul 31 23:47:14.286225 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:47:14.286207    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gpsh4\" (UniqueName: \"kubernetes.io/projected/47326e9a-7028-4a90-94e4-70d55ac3290d-kube-api-access-gpsh4\") pod \"console-5d59f9cb75-g6g2x\" (UID: \"47326e9a-7028-4a90-94e4-70d55ac3290d\") " pod="openshift-console/console-5d59f9cb75-g6g2x"
Jul 31 23:47:14.439136 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:47:14.439094    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5d59f9cb75-g6g2x"
Jul 31 23:47:14.607063 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:47:14.607010    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5d59f9cb75-g6g2x"]
Jul 31 23:47:14.627081 ip-10-0-136-108 kubenswrapper[2578]: W0731 23:47:14.627046    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod47326e9a_7028_4a90_94e4_70d55ac3290d.slice/crio-b25c304579026de5ff906da271b4dbaec220ca22faaa9b83470a1235d811f11f WatchSource:0}: Error finding container b25c304579026de5ff906da271b4dbaec220ca22faaa9b83470a1235d811f11f: Status 404 returned error can't find the container with id b25c304579026de5ff906da271b4dbaec220ca22faaa9b83470a1235d811f11f
Jul 31 23:47:14.826743 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:47:14.826613    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5d59f9cb75-g6g2x" event={"ID":"47326e9a-7028-4a90-94e4-70d55ac3290d","Type":"ContainerStarted","Data":"b25c304579026de5ff906da271b4dbaec220ca22faaa9b83470a1235d811f11f"}
Jul 31 23:47:14.828580 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:47:14.828548    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-7ftsc" event={"ID":"2bcd7dcd-ab90-47bd-8747-5f1fafe9960c","Type":"ContainerStarted","Data":"8e35f23de850df7d0c06e210667853a70629dddeab7878964678303175de1509"}
Jul 31 23:47:14.830083 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:47:14.830056    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-cjvb8" event={"ID":"759fe27f-5715-4033-bb09-a5f5a21d7c53","Type":"ContainerStarted","Data":"584747c5c64848f19363851f0e563645b3f315b25d758c05387c3849cc968bec"}
Jul 31 23:47:14.832346 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:47:14.832317    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-v2wkd" event={"ID":"194c7e46-9a88-4078-a687-63112c8ccab5","Type":"ContainerStarted","Data":"b2ae87fa25addc4cdfb7d778afbfd091392a4c144331cec50f8d2542aa9aef8e"}
Jul 31 23:47:14.832454 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:47:14.832376    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-v2wkd" event={"ID":"194c7e46-9a88-4078-a687-63112c8ccab5","Type":"ContainerStarted","Data":"a749f9a22094a445dfe33e624ee60907586071449f22a1a935ab7ab0ae6afc8f"}
Jul 31 23:47:14.832454 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:47:14.832406    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-v2wkd" event={"ID":"194c7e46-9a88-4078-a687-63112c8ccab5","Type":"ContainerStarted","Data":"4557a2e425bed6fe8e43a2fd8568f29cd83253b0e6aad1883e6b1de4ea25598f"}
Jul 31 23:47:16.201380 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:47:16.201343    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gttdm_b32e6d42-e4e0-4113-b8ce-f5d5c5dc13f7/konnectivity-agent/0.log" limit=1024
Jul 31 23:47:16.301315 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:47:16.301266    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gttdm_b32e6d42-e4e0-4113-b8ce-f5d5c5dc13f7/konnectivity-agent/0.log" limit=1024
Jul 31 23:47:16.405293 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:47:16.405259    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gttdm_b32e6d42-e4e0-4113-b8ce-f5d5c5dc13f7/konnectivity-agent/0.log" limit=1024
Jul 31 23:47:16.493942 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:47:16.493909    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gttdm_b32e6d42-e4e0-4113-b8ce-f5d5c5dc13f7/konnectivity-agent/0.log" limit=1024
Jul 31 23:47:16.695607 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:47:16.695551    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gttdm_b32e6d42-e4e0-4113-b8ce-f5d5c5dc13f7/konnectivity-agent/0.log" limit=1024
Jul 31 23:47:16.763739 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:47:16.763666    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-m9px5"
Jul 31 23:47:16.795872 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:47:16.795839    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gttdm_b32e6d42-e4e0-4113-b8ce-f5d5c5dc13f7/konnectivity-agent/0.log" limit=1024
Jul 31 23:47:16.962034 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:47:16.961992    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gttdm_b32e6d42-e4e0-4113-b8ce-f5d5c5dc13f7/konnectivity-agent/0.log" limit=1024
Jul 31 23:47:17.031051 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:47:17.031021    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gttdm_b32e6d42-e4e0-4113-b8ce-f5d5c5dc13f7/konnectivity-agent/0.log" limit=1024
Jul 31 23:47:18.844713 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:47:18.844679    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5d59f9cb75-g6g2x" event={"ID":"47326e9a-7028-4a90-94e4-70d55ac3290d","Type":"ContainerStarted","Data":"7fd964cc7523a1de09ed649542fc42bf5bad9fb2d287b303fd9efc45c3a260e0"}
Jul 31 23:47:18.846207 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:47:18.846155    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-7ftsc" event={"ID":"2bcd7dcd-ab90-47bd-8747-5f1fafe9960c","Type":"ContainerStarted","Data":"49e45057c3beac5cf0f5be26b16d138a7820e6b929b7dc488922eb9da9462222"}
Jul 31 23:47:18.847814 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:47:18.847793    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-cjvb8" event={"ID":"759fe27f-5715-4033-bb09-a5f5a21d7c53","Type":"ContainerStarted","Data":"50a0e00741f4899605a8fea05ed91bd1e985670c86bda3dd06b128a2f64cacab"}
Jul 31 23:47:18.847927 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:47:18.847849    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-cjvb8"
Jul 31 23:47:18.847927 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:47:18.847877    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-cjvb8" event={"ID":"759fe27f-5715-4033-bb09-a5f5a21d7c53","Type":"ContainerStarted","Data":"13c3cc5843a703bd0a92cd5db4b5b2797951e1acfb562522ec03199e8ec13920"}
Jul 31 23:47:18.849720 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:47:18.849694    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-v2wkd" event={"ID":"194c7e46-9a88-4078-a687-63112c8ccab5","Type":"ContainerStarted","Data":"ae7a70cdd83d134212de12f67ada2631ed2f3d57dcf49840f7fb3c4f9a194e14"}
Jul 31 23:47:18.863417 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:47:18.863364    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-5d59f9cb75-g6g2x" podStartSLOduration=1.097083083 podStartE2EDuration="4.86334676s" totalImagesPullingTime="3.766263677s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 23:47:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 23:47:18.862011647 +0000 UTC m=+62.829762405" watchObservedRunningTime="2026-07-31 23:47:18.86334676 +0000 UTC m=+62.831097495"
Jul 31 23:47:18.883302 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:47:18.883253    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-cjvb8" podStartSLOduration=1.713258019 podStartE2EDuration="5.883238071s" totalImagesPullingTime="4.169980052s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 23:47:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 23:47:18.882075474 +0000 UTC m=+62.849826232" watchObservedRunningTime="2026-07-31 23:47:18.883238071 +0000 UTC m=+62.850988806"
Jul 31 23:47:18.899437 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:47:18.899387    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-7ftsc" podStartSLOduration=1.69421887 podStartE2EDuration="5.899371209s" totalImagesPullingTime="4.205152339s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 23:47:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 23:47:18.898188604 +0000 UTC m=+62.865939356" watchObservedRunningTime="2026-07-31 23:47:18.899371209 +0000 UTC m=+62.867121966"
Jul 31 23:47:18.917949 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:47:18.917906    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-v2wkd" podStartSLOduration=1.8840153179999999 podStartE2EDuration="5.917895823s" totalImagesPullingTime="4.033880505s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-31 23:47:13 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 23:47:18.917273447 +0000 UTC m=+62.885024227" watchObservedRunningTime="2026-07-31 23:47:18.917895823 +0000 UTC m=+62.885646608"
Jul 31 23:47:19.421865 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:47:19.421830    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gttdm_b32e6d42-e4e0-4113-b8ce-f5d5c5dc13f7/konnectivity-agent/0.log" limit=1024
Jul 31 23:47:19.458656 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:47:19.458627    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-758d7cb874-knbql"]
Jul 31 23:47:19.461249 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:47:19.461232    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-758d7cb874-knbql"
Jul 31 23:47:19.463294 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:47:19.463266    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Jul 31 23:47:19.463294 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:47:19.463278    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\""
Jul 31 23:47:19.463294 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:47:19.463266    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Jul 31 23:47:19.463549 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:47:19.463299    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\""
Jul 31 23:47:19.463549 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:47:19.463307    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-9k2p9\""
Jul 31 23:47:19.463549 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:47:19.463538    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Jul 31 23:47:19.470653 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:47:19.470632    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-758d7cb874-knbql"]
Jul 31 23:47:19.495559 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:47:19.495530    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gttdm_b32e6d42-e4e0-4113-b8ce-f5d5c5dc13f7/konnectivity-agent/0.log" limit=1024
Jul 31 23:47:19.525535 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:47:19.525506    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/c01686d9-cf94-4729-88fc-8c6894967f98-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-758d7cb874-knbql\" (UID: \"c01686d9-cf94-4729-88fc-8c6894967f98\") " pod="openshift-monitoring/prometheus-operator-758d7cb874-knbql"
Jul 31 23:47:19.525663 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:47:19.525538    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c01686d9-cf94-4729-88fc-8c6894967f98-metrics-client-ca\") pod \"prometheus-operator-758d7cb874-knbql\" (UID: \"c01686d9-cf94-4729-88fc-8c6894967f98\") " pod="openshift-monitoring/prometheus-operator-758d7cb874-knbql"
Jul 31 23:47:19.525663 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:47:19.525565    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/c01686d9-cf94-4729-88fc-8c6894967f98-prometheus-operator-tls\") pod \"prometheus-operator-758d7cb874-knbql\" (UID: \"c01686d9-cf94-4729-88fc-8c6894967f98\") " pod="openshift-monitoring/prometheus-operator-758d7cb874-knbql"
Jul 31 23:47:19.525663 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:47:19.525613    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xf6gm\" (UniqueName: \"kubernetes.io/projected/c01686d9-cf94-4729-88fc-8c6894967f98-kube-api-access-xf6gm\") pod \"prometheus-operator-758d7cb874-knbql\" (UID: \"c01686d9-cf94-4729-88fc-8c6894967f98\") " pod="openshift-monitoring/prometheus-operator-758d7cb874-knbql"
Jul 31 23:47:19.627039 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:47:19.627010    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/c01686d9-cf94-4729-88fc-8c6894967f98-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-758d7cb874-knbql\" (UID: \"c01686d9-cf94-4729-88fc-8c6894967f98\") " pod="openshift-monitoring/prometheus-operator-758d7cb874-knbql"
Jul 31 23:47:19.627160 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:47:19.627049    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c01686d9-cf94-4729-88fc-8c6894967f98-metrics-client-ca\") pod \"prometheus-operator-758d7cb874-knbql\" (UID: \"c01686d9-cf94-4729-88fc-8c6894967f98\") " pod="openshift-monitoring/prometheus-operator-758d7cb874-knbql"
Jul 31 23:47:19.627160 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:47:19.627080    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/c01686d9-cf94-4729-88fc-8c6894967f98-prometheus-operator-tls\") pod \"prometheus-operator-758d7cb874-knbql\" (UID: \"c01686d9-cf94-4729-88fc-8c6894967f98\") " pod="openshift-monitoring/prometheus-operator-758d7cb874-knbql"
Jul 31 23:47:19.627160 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:47:19.627105    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xf6gm\" (UniqueName: \"kubernetes.io/projected/c01686d9-cf94-4729-88fc-8c6894967f98-kube-api-access-xf6gm\") pod \"prometheus-operator-758d7cb874-knbql\" (UID: \"c01686d9-cf94-4729-88fc-8c6894967f98\") " pod="openshift-monitoring/prometheus-operator-758d7cb874-knbql"
Jul 31 23:47:19.627309 ip-10-0-136-108 kubenswrapper[2578]: E0731 23:47:19.627262    2578 secret.go:189] Couldn't get secret openshift-monitoring/prometheus-operator-tls: secret "prometheus-operator-tls" not found
Jul 31 23:47:19.627346 ip-10-0-136-108 kubenswrapper[2578]: E0731 23:47:19.627337    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c01686d9-cf94-4729-88fc-8c6894967f98-prometheus-operator-tls podName:c01686d9-cf94-4729-88fc-8c6894967f98 nodeName:}" failed. No retries permitted until 2026-07-31 23:47:20.127316192 +0000 UTC m=+64.095066927 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "prometheus-operator-tls" (UniqueName: "kubernetes.io/secret/c01686d9-cf94-4729-88fc-8c6894967f98-prometheus-operator-tls") pod "prometheus-operator-758d7cb874-knbql" (UID: "c01686d9-cf94-4729-88fc-8c6894967f98") : secret "prometheus-operator-tls" not found
Jul 31 23:47:19.627855 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:47:19.627830    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c01686d9-cf94-4729-88fc-8c6894967f98-metrics-client-ca\") pod \"prometheus-operator-758d7cb874-knbql\" (UID: \"c01686d9-cf94-4729-88fc-8c6894967f98\") " pod="openshift-monitoring/prometheus-operator-758d7cb874-knbql"
Jul 31 23:47:19.629433 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:47:19.629411    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/c01686d9-cf94-4729-88fc-8c6894967f98-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-758d7cb874-knbql\" (UID: \"c01686d9-cf94-4729-88fc-8c6894967f98\") " pod="openshift-monitoring/prometheus-operator-758d7cb874-knbql"
Jul 31 23:47:19.635551 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:47:19.635528    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xf6gm\" (UniqueName: \"kubernetes.io/projected/c01686d9-cf94-4729-88fc-8c6894967f98-kube-api-access-xf6gm\") pod \"prometheus-operator-758d7cb874-knbql\" (UID: \"c01686d9-cf94-4729-88fc-8c6894967f98\") " pod="openshift-monitoring/prometheus-operator-758d7cb874-knbql"
Jul 31 23:47:19.824224 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:47:19.824152    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gttdm_b32e6d42-e4e0-4113-b8ce-f5d5c5dc13f7/konnectivity-agent/0.log" limit=1024
Jul 31 23:47:20.130373 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:47:20.130278    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/c01686d9-cf94-4729-88fc-8c6894967f98-prometheus-operator-tls\") pod \"prometheus-operator-758d7cb874-knbql\" (UID: \"c01686d9-cf94-4729-88fc-8c6894967f98\") " pod="openshift-monitoring/prometheus-operator-758d7cb874-knbql"
Jul 31 23:47:20.132621 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:47:20.132601    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/c01686d9-cf94-4729-88fc-8c6894967f98-prometheus-operator-tls\") pod \"prometheus-operator-758d7cb874-knbql\" (UID: \"c01686d9-cf94-4729-88fc-8c6894967f98\") " pod="openshift-monitoring/prometheus-operator-758d7cb874-knbql"
Jul 31 23:47:20.369918 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:47:20.369880    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-758d7cb874-knbql"
Jul 31 23:47:20.490906 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:47:20.490880    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-758d7cb874-knbql"]
Jul 31 23:47:20.493999 ip-10-0-136-108 kubenswrapper[2578]: W0731 23:47:20.493967    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc01686d9_cf94_4729_88fc_8c6894967f98.slice/crio-de0db40b7c756e9d9d1ef3bafc77b6dc04b203997a91b824e66a0308d41683ce WatchSource:0}: Error finding container de0db40b7c756e9d9d1ef3bafc77b6dc04b203997a91b824e66a0308d41683ce: Status 404 returned error can't find the container with id de0db40b7c756e9d9d1ef3bafc77b6dc04b203997a91b824e66a0308d41683ce
Jul 31 23:47:20.856954 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:47:20.856911    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-758d7cb874-knbql" event={"ID":"c01686d9-cf94-4729-88fc-8c6894967f98","Type":"ContainerStarted","Data":"de0db40b7c756e9d9d1ef3bafc77b6dc04b203997a91b824e66a0308d41683ce"}
Jul 31 23:47:20.915008 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:47:20.914968    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gttdm_b32e6d42-e4e0-4113-b8ce-f5d5c5dc13f7/konnectivity-agent/0.log" limit=1024
Jul 31 23:47:20.971378 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:47:20.971333    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gttdm_b32e6d42-e4e0-4113-b8ce-f5d5c5dc13f7/konnectivity-agent/0.log" limit=1024
Jul 31 23:47:21.863073 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:47:21.863033    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-758d7cb874-knbql" event={"ID":"c01686d9-cf94-4729-88fc-8c6894967f98","Type":"ContainerStarted","Data":"38be45db5742c3ad61b104b5b0533a27429f47339a127cbbdd79a7c85d0117f7"}
Jul 31 23:47:21.863500 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:47:21.863095    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-758d7cb874-knbql" event={"ID":"c01686d9-cf94-4729-88fc-8c6894967f98","Type":"ContainerStarted","Data":"cd5b386eba903d04270148e925dc2ef4bb2e023f2e750fe29fe3e5b14ca8fd11"}
Jul 31 23:47:21.878546 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:47:21.878451    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-758d7cb874-knbql" podStartSLOduration=1.746647282 podStartE2EDuration="2.878434637s" totalImagesPullingTime="1.131787355s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 23:47:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 23:47:21.878001666 +0000 UTC m=+65.845752445" watchObservedRunningTime="2026-07-31 23:47:21.878434637 +0000 UTC m=+65.846185396"
Jul 31 23:47:21.885589 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:47:21.885556    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gttdm_b32e6d42-e4e0-4113-b8ce-f5d5c5dc13f7/konnectivity-agent/0.log" limit=1024
Jul 31 23:47:22.248026 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:47:22.247997    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/087b0a99-ae24-49e2-ba59-55651731fd60-metrics-certs\") pod \"network-metrics-daemon-qs7p7\" (UID: \"087b0a99-ae24-49e2-ba59-55651731fd60\") " pod="openshift-multus/network-metrics-daemon-qs7p7"
Jul 31 23:47:22.248228 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:47:22.248038    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/9d113c99-25bd-49a4-b8e2-f5c46f3ecc02-original-pull-secret\") pod \"global-pull-secret-syncer-xlv68\" (UID: \"9d113c99-25bd-49a4-b8e2-f5c46f3ecc02\") " pod="kube-system/global-pull-secret-syncer-xlv68"
Jul 31 23:47:22.250055 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:47:22.250038    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 31 23:47:22.250301 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:47:22.250286    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 31 23:47:22.260805 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:47:22.260783    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/087b0a99-ae24-49e2-ba59-55651731fd60-metrics-certs\") pod \"network-metrics-daemon-qs7p7\" (UID: \"087b0a99-ae24-49e2-ba59-55651731fd60\") " pod="openshift-multus/network-metrics-daemon-qs7p7"
Jul 31 23:47:22.260884 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:47:22.260805    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/9d113c99-25bd-49a4-b8e2-f5c46f3ecc02-original-pull-secret\") pod \"global-pull-secret-syncer-xlv68\" (UID: \"9d113c99-25bd-49a4-b8e2-f5c46f3ecc02\") " pod="kube-system/global-pull-secret-syncer-xlv68"
Jul 31 23:47:22.348764 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:47:22.348714    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-94n2k\" (UniqueName: \"kubernetes.io/projected/0503a101-ddee-440a-ae03-e76fe30685e7-kube-api-access-94n2k\") pod \"network-check-target-8d9rr\" (UID: \"0503a101-ddee-440a-ae03-e76fe30685e7\") " pod="openshift-network-diagnostics/network-check-target-8d9rr"
Jul 31 23:47:22.351222 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:47:22.351198    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 31 23:47:22.362102 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:47:22.362072    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 31 23:47:22.373128 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:47:22.373092    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-94n2k\" (UniqueName: \"kubernetes.io/projected/0503a101-ddee-440a-ae03-e76fe30685e7-kube-api-access-94n2k\") pod \"network-check-target-8d9rr\" (UID: \"0503a101-ddee-440a-ae03-e76fe30685e7\") " pod="openshift-network-diagnostics/network-check-target-8d9rr"
Jul 31 23:47:22.496508 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:47:22.496478    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-bfr2g\""
Jul 31 23:47:22.502795 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:47:22.502745    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-4hvkj\""
Jul 31 23:47:22.505268 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:47:22.505253    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8d9rr"
Jul 31 23:47:22.506495 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:47:22.506480    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xlv68"
Jul 31 23:47:22.512000 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:47:22.511972    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qs7p7"
Jul 31 23:47:22.647722 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:47:22.647661    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-xlv68"]
Jul 31 23:47:22.653193 ip-10-0-136-108 kubenswrapper[2578]: W0731 23:47:22.653146    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9d113c99_25bd_49a4_b8e2_f5c46f3ecc02.slice/crio-8a7caad959ad095341ed6d2b2202804608900cc949c806ac3d593a8fcf563c02 WatchSource:0}: Error finding container 8a7caad959ad095341ed6d2b2202804608900cc949c806ac3d593a8fcf563c02: Status 404 returned error can't find the container with id 8a7caad959ad095341ed6d2b2202804608900cc949c806ac3d593a8fcf563c02
Jul 31 23:47:22.657765 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:47:22.657737    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-8d9rr"]
Jul 31 23:47:22.661638 ip-10-0-136-108 kubenswrapper[2578]: W0731 23:47:22.661599    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0503a101_ddee_440a_ae03_e76fe30685e7.slice/crio-ebe0c02169339710d302f7245f8fab634beedd6bd730bbceb14361bd8b6f10ba WatchSource:0}: Error finding container ebe0c02169339710d302f7245f8fab634beedd6bd730bbceb14361bd8b6f10ba: Status 404 returned error can't find the container with id ebe0c02169339710d302f7245f8fab634beedd6bd730bbceb14361bd8b6f10ba
Jul 31 23:47:22.676836 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:47:22.676812    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-qs7p7"]
Jul 31 23:47:22.680251 ip-10-0-136-108 kubenswrapper[2578]: W0731 23:47:22.680225    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod087b0a99_ae24_49e2_ba59_55651731fd60.slice/crio-57393b27b089944c639f614545c3b33e3cfbc95f9186785688e8ad3d28be6a84 WatchSource:0}: Error finding container 57393b27b089944c639f614545c3b33e3cfbc95f9186785688e8ad3d28be6a84: Status 404 returned error can't find the container with id 57393b27b089944c639f614545c3b33e3cfbc95f9186785688e8ad3d28be6a84
Jul 31 23:47:22.766715 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:47:22.766630    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gttdm_b32e6d42-e4e0-4113-b8ce-f5d5c5dc13f7/konnectivity-agent/0.log" limit=1024
Jul 31 23:47:22.823182 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:47:22.823148    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gttdm_b32e6d42-e4e0-4113-b8ce-f5d5c5dc13f7/konnectivity-agent/0.log" limit=1024
Jul 31 23:47:22.866873 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:47:22.866837    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-xlv68" event={"ID":"9d113c99-25bd-49a4-b8e2-f5c46f3ecc02","Type":"ContainerStarted","Data":"8a7caad959ad095341ed6d2b2202804608900cc949c806ac3d593a8fcf563c02"}
Jul 31 23:47:22.867986 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:47:22.867963    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-qs7p7" event={"ID":"087b0a99-ae24-49e2-ba59-55651731fd60","Type":"ContainerStarted","Data":"57393b27b089944c639f614545c3b33e3cfbc95f9186785688e8ad3d28be6a84"}
Jul 31 23:47:22.869072 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:47:22.869053    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-8d9rr" event={"ID":"0503a101-ddee-440a-ae03-e76fe30685e7","Type":"ContainerStarted","Data":"ebe0c02169339710d302f7245f8fab634beedd6bd730bbceb14361bd8b6f10ba"}
Jul 31 23:47:22.879787 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:47:22.879756    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gttdm_b32e6d42-e4e0-4113-b8ce-f5d5c5dc13f7/konnectivity-agent/0.log" limit=1024
Jul 31 23:47:23.659613 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:47:23.659487    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gttdm_b32e6d42-e4e0-4113-b8ce-f5d5c5dc13f7/konnectivity-agent/0.log" limit=1024
Jul 31 23:47:23.875447 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:47:23.875411    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-qs7p7" event={"ID":"087b0a99-ae24-49e2-ba59-55651731fd60","Type":"ContainerStarted","Data":"7e2f91709d36fb48f254ef9684f1ab46beec41554211400b27ad7f7e2aa60fc6"}
Jul 31 23:47:23.899299 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:47:23.899248    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gttdm_b32e6d42-e4e0-4113-b8ce-f5d5c5dc13f7/konnectivity-agent/0.log" limit=1024
Jul 31 23:47:24.017600 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:47:24.017564    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-wxg27"]
Jul 31 23:47:24.023526 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:47:24.023496    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-wxg27"
Jul 31 23:47:24.026580 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:47:24.026559    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Jul 31 23:47:24.026759 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:47:24.026630    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Jul 31 23:47:24.026871 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:47:24.026654    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-fllzf\""
Jul 31 23:47:24.027143 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:47:24.027124    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Jul 31 23:47:24.038589 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:47:24.038560    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gttdm_b32e6d42-e4e0-4113-b8ce-f5d5c5dc13f7/konnectivity-agent/0.log" limit=1024
Jul 31 23:47:24.061889 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:47:24.061847    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/49955a37-a47a-48ee-826c-6aaa95715852-sys\") pod \"node-exporter-wxg27\" (UID: \"49955a37-a47a-48ee-826c-6aaa95715852\") " pod="openshift-monitoring/node-exporter-wxg27"
Jul 31 23:47:24.061889 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:47:24.061890    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/49955a37-a47a-48ee-826c-6aaa95715852-root\") pod \"node-exporter-wxg27\" (UID: \"49955a37-a47a-48ee-826c-6aaa95715852\") " pod="openshift-monitoring/node-exporter-wxg27"
Jul 31 23:47:24.062118 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:47:24.061931    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/49955a37-a47a-48ee-826c-6aaa95715852-node-exporter-accelerators-collector-config\") pod \"node-exporter-wxg27\" (UID: \"49955a37-a47a-48ee-826c-6aaa95715852\") " pod="openshift-monitoring/node-exporter-wxg27"
Jul 31 23:47:24.062118 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:47:24.061968    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/49955a37-a47a-48ee-826c-6aaa95715852-node-exporter-wtmp\") pod \"node-exporter-wxg27\" (UID: \"49955a37-a47a-48ee-826c-6aaa95715852\") " pod="openshift-monitoring/node-exporter-wxg27"
Jul 31 23:47:24.062118 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:47:24.062022    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/49955a37-a47a-48ee-826c-6aaa95715852-node-exporter-textfile\") pod \"node-exporter-wxg27\" (UID: \"49955a37-a47a-48ee-826c-6aaa95715852\") " pod="openshift-monitoring/node-exporter-wxg27"
Jul 31 23:47:24.062118 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:47:24.062047    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/49955a37-a47a-48ee-826c-6aaa95715852-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-wxg27\" (UID: \"49955a37-a47a-48ee-826c-6aaa95715852\") " pod="openshift-monitoring/node-exporter-wxg27"
Jul 31 23:47:24.062118 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:47:24.062073    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-26ms4\" (UniqueName: \"kubernetes.io/projected/49955a37-a47a-48ee-826c-6aaa95715852-kube-api-access-26ms4\") pod \"node-exporter-wxg27\" (UID: \"49955a37-a47a-48ee-826c-6aaa95715852\") " pod="openshift-monitoring/node-exporter-wxg27"
Jul 31 23:47:24.062118 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:47:24.062099    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/49955a37-a47a-48ee-826c-6aaa95715852-node-exporter-tls\") pod \"node-exporter-wxg27\" (UID: \"49955a37-a47a-48ee-826c-6aaa95715852\") " pod="openshift-monitoring/node-exporter-wxg27"
Jul 31 23:47:24.062567 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:47:24.062220    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/49955a37-a47a-48ee-826c-6aaa95715852-metrics-client-ca\") pod \"node-exporter-wxg27\" (UID: \"49955a37-a47a-48ee-826c-6aaa95715852\") " pod="openshift-monitoring/node-exporter-wxg27"
Jul 31 23:47:24.152591 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:47:24.152554    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gttdm_b32e6d42-e4e0-4113-b8ce-f5d5c5dc13f7/konnectivity-agent/0.log" limit=1024
Jul 31 23:47:24.163524 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:47:24.163490    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-26ms4\" (UniqueName: \"kubernetes.io/projected/49955a37-a47a-48ee-826c-6aaa95715852-kube-api-access-26ms4\") pod \"node-exporter-wxg27\" (UID: \"49955a37-a47a-48ee-826c-6aaa95715852\") " pod="openshift-monitoring/node-exporter-wxg27"
Jul 31 23:47:24.163723 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:47:24.163537    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/49955a37-a47a-48ee-826c-6aaa95715852-node-exporter-tls\") pod \"node-exporter-wxg27\" (UID: \"49955a37-a47a-48ee-826c-6aaa95715852\") " pod="openshift-monitoring/node-exporter-wxg27"
Jul 31 23:47:24.163723 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:47:24.163579    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/49955a37-a47a-48ee-826c-6aaa95715852-metrics-client-ca\") pod \"node-exporter-wxg27\" (UID: \"49955a37-a47a-48ee-826c-6aaa95715852\") " pod="openshift-monitoring/node-exporter-wxg27"
Jul 31 23:47:24.163723 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:47:24.163620    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/49955a37-a47a-48ee-826c-6aaa95715852-sys\") pod \"node-exporter-wxg27\" (UID: \"49955a37-a47a-48ee-826c-6aaa95715852\") " pod="openshift-monitoring/node-exporter-wxg27"
Jul 31 23:47:24.163723 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:47:24.163639    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/49955a37-a47a-48ee-826c-6aaa95715852-root\") pod \"node-exporter-wxg27\" (UID: \"49955a37-a47a-48ee-826c-6aaa95715852\") " pod="openshift-monitoring/node-exporter-wxg27"
Jul 31 23:47:24.163723 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:47:24.163682    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/49955a37-a47a-48ee-826c-6aaa95715852-node-exporter-accelerators-collector-config\") pod \"node-exporter-wxg27\" (UID: \"49955a37-a47a-48ee-826c-6aaa95715852\") " pod="openshift-monitoring/node-exporter-wxg27"
Jul 31 23:47:24.163723 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:47:24.163723    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/49955a37-a47a-48ee-826c-6aaa95715852-node-exporter-wtmp\") pod \"node-exporter-wxg27\" (UID: \"49955a37-a47a-48ee-826c-6aaa95715852\") " pod="openshift-monitoring/node-exporter-wxg27"
Jul 31 23:47:24.164048 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:47:24.163791    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/49955a37-a47a-48ee-826c-6aaa95715852-node-exporter-textfile\") pod \"node-exporter-wxg27\" (UID: \"49955a37-a47a-48ee-826c-6aaa95715852\") " pod="openshift-monitoring/node-exporter-wxg27"
Jul 31 23:47:24.164048 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:47:24.163817    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/49955a37-a47a-48ee-826c-6aaa95715852-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-wxg27\" (UID: \"49955a37-a47a-48ee-826c-6aaa95715852\") " pod="openshift-monitoring/node-exporter-wxg27"
Jul 31 23:47:24.165016 ip-10-0-136-108 kubenswrapper[2578]: E0731 23:47:24.164328    2578 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found
Jul 31 23:47:24.165016 ip-10-0-136-108 kubenswrapper[2578]: E0731 23:47:24.164404    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/49955a37-a47a-48ee-826c-6aaa95715852-node-exporter-tls podName:49955a37-a47a-48ee-826c-6aaa95715852 nodeName:}" failed. No retries permitted until 2026-07-31 23:47:24.664382309 +0000 UTC m=+68.632133067 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/49955a37-a47a-48ee-826c-6aaa95715852-node-exporter-tls") pod "node-exporter-wxg27" (UID: "49955a37-a47a-48ee-826c-6aaa95715852") : secret "node-exporter-tls" not found
Jul 31 23:47:24.165016 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:47:24.164560    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/49955a37-a47a-48ee-826c-6aaa95715852-sys\") pod \"node-exporter-wxg27\" (UID: \"49955a37-a47a-48ee-826c-6aaa95715852\") " pod="openshift-monitoring/node-exporter-wxg27"
Jul 31 23:47:24.165016 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:47:24.164635    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/49955a37-a47a-48ee-826c-6aaa95715852-root\") pod \"node-exporter-wxg27\" (UID: \"49955a37-a47a-48ee-826c-6aaa95715852\") " pod="openshift-monitoring/node-exporter-wxg27"
Jul 31 23:47:24.165016 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:47:24.164665    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/49955a37-a47a-48ee-826c-6aaa95715852-node-exporter-wtmp\") pod \"node-exporter-wxg27\" (UID: \"49955a37-a47a-48ee-826c-6aaa95715852\") " pod="openshift-monitoring/node-exporter-wxg27"
Jul 31 23:47:24.165016 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:47:24.164739    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/49955a37-a47a-48ee-826c-6aaa95715852-metrics-client-ca\") pod \"node-exporter-wxg27\" (UID: \"49955a37-a47a-48ee-826c-6aaa95715852\") " pod="openshift-monitoring/node-exporter-wxg27"
Jul 31 23:47:24.165016 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:47:24.164846    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/49955a37-a47a-48ee-826c-6aaa95715852-node-exporter-textfile\") pod \"node-exporter-wxg27\" (UID: \"49955a37-a47a-48ee-826c-6aaa95715852\") " pod="openshift-monitoring/node-exporter-wxg27"
Jul 31 23:47:24.165016 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:47:24.164972    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/49955a37-a47a-48ee-826c-6aaa95715852-node-exporter-accelerators-collector-config\") pod \"node-exporter-wxg27\" (UID: \"49955a37-a47a-48ee-826c-6aaa95715852\") " pod="openshift-monitoring/node-exporter-wxg27"
Jul 31 23:47:24.166919 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:47:24.166894    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/49955a37-a47a-48ee-826c-6aaa95715852-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-wxg27\" (UID: \"49955a37-a47a-48ee-826c-6aaa95715852\") " pod="openshift-monitoring/node-exporter-wxg27"
Jul 31 23:47:24.173922 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:47:24.173842    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-26ms4\" (UniqueName: \"kubernetes.io/projected/49955a37-a47a-48ee-826c-6aaa95715852-kube-api-access-26ms4\") pod \"node-exporter-wxg27\" (UID: \"49955a37-a47a-48ee-826c-6aaa95715852\") " pod="openshift-monitoring/node-exporter-wxg27"
Jul 31 23:47:24.256432 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:47:24.256392    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gttdm_b32e6d42-e4e0-4113-b8ce-f5d5c5dc13f7/konnectivity-agent/0.log" limit=1024
Jul 31 23:47:24.351331 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:47:24.351260    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gttdm_b32e6d42-e4e0-4113-b8ce-f5d5c5dc13f7/konnectivity-agent/0.log" limit=1024
Jul 31 23:47:24.440005 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:47:24.439902    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-5d59f9cb75-g6g2x"
Jul 31 23:47:24.440005 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:47:24.439992    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-5d59f9cb75-g6g2x"
Jul 31 23:47:24.441791 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:47:24.441495    2578 patch_prober.go:28] interesting pod/console-5d59f9cb75-g6g2x container/console namespace/openshift-console: Startup probe status=failure output="Get \"https://10.133.0.9:8443/health\": dial tcp 10.133.0.9:8443: connect: connection refused" start-of-body=
Jul 31 23:47:24.441791 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:47:24.441555    2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-console/console-5d59f9cb75-g6g2x" podUID="47326e9a-7028-4a90-94e4-70d55ac3290d" containerName="console" probeResult="failure" output="Get \"https://10.133.0.9:8443/health\": dial tcp 10.133.0.9:8443: connect: connection refused"
Jul 31 23:47:24.450617 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:47:24.450582    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gttdm_b32e6d42-e4e0-4113-b8ce-f5d5c5dc13f7/konnectivity-agent/0.log" limit=1024
Jul 31 23:47:24.581264 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:47:24.581219    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gttdm_b32e6d42-e4e0-4113-b8ce-f5d5c5dc13f7/konnectivity-agent/0.log" limit=1024
Jul 31 23:47:24.668380 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:47:24.668286    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/49955a37-a47a-48ee-826c-6aaa95715852-node-exporter-tls\") pod \"node-exporter-wxg27\" (UID: \"49955a37-a47a-48ee-826c-6aaa95715852\") " pod="openshift-monitoring/node-exporter-wxg27"
Jul 31 23:47:24.671385 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:47:24.671354    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/49955a37-a47a-48ee-826c-6aaa95715852-node-exporter-tls\") pod \"node-exporter-wxg27\" (UID: \"49955a37-a47a-48ee-826c-6aaa95715852\") " pod="openshift-monitoring/node-exporter-wxg27"
Jul 31 23:47:24.873198 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:47:24.873123    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gttdm_b32e6d42-e4e0-4113-b8ce-f5d5c5dc13f7/konnectivity-agent/0.log" limit=1024
Jul 31 23:47:24.881492 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:47:24.881410    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-qs7p7" event={"ID":"087b0a99-ae24-49e2-ba59-55651731fd60","Type":"ContainerStarted","Data":"fea0a1f3a0262dc868ad80ea9f39a3c4677af7500258c4d15e79a92fa850bb79"}
Jul 31 23:47:24.899279 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:47:24.899220    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-qs7p7" podStartSLOduration=66.879435066 podStartE2EDuration="1m7.899196953s" totalImagesPullingTime="1.019761887s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 23:46:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 23:47:24.897824069 +0000 UTC m=+68.865574819" watchObservedRunningTime="2026-07-31 23:47:24.899196953 +0000 UTC m=+68.866947711"
Jul 31 23:47:24.935830 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:47:24.935797    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-wxg27"
Jul 31 23:47:25.008139 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:47:25.008097    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gttdm_b32e6d42-e4e0-4113-b8ce-f5d5c5dc13f7/konnectivity-agent/0.log" limit=1024
Jul 31 23:47:25.012977 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:47:25.012525    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 31 23:47:25.032524 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:47:25.032313    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Jul 31 23:47:25.035240 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:47:25.034746    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 31 23:47:25.035356 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:47:25.035332    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
Jul 31 23:47:25.035424 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:47:25.035374    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
Jul 31 23:47:25.038301 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:47:25.038145    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
Jul 31 23:47:25.038301 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:47:25.038271    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
Jul 31 23:47:25.038496 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:47:25.038417    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-qdr8l\""
Jul 31 23:47:25.038496 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:47:25.038440    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
Jul 31 23:47:25.038969 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:47:25.038948    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
Jul 31 23:47:25.039382 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:47:25.039364    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
Jul 31 23:47:25.039515 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:47:25.039476    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
Jul 31 23:47:25.039670 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:47:25.039652    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
Jul 31 23:47:25.072937 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:47:25.072553    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d3918f5e-491a-4456-9438-783cf36e291e-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"d3918f5e-491a-4456-9438-783cf36e291e\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 31 23:47:25.072937 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:47:25.072608    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/d3918f5e-491a-4456-9438-783cf36e291e-tls-assets\") pod \"alertmanager-main-0\" (UID: \"d3918f5e-491a-4456-9438-783cf36e291e\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 31 23:47:25.072937 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:47:25.072657    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/d3918f5e-491a-4456-9438-783cf36e291e-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"d3918f5e-491a-4456-9438-783cf36e291e\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 31 23:47:25.072937 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:47:25.072683    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fn2w6\" (UniqueName: \"kubernetes.io/projected/d3918f5e-491a-4456-9438-783cf36e291e-kube-api-access-fn2w6\") pod \"alertmanager-main-0\" (UID: \"d3918f5e-491a-4456-9438-783cf36e291e\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 31 23:47:25.072937 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:47:25.072709    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/d3918f5e-491a-4456-9438-783cf36e291e-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"d3918f5e-491a-4456-9438-783cf36e291e\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 31 23:47:25.072937 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:47:25.072772    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/d3918f5e-491a-4456-9438-783cf36e291e-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"d3918f5e-491a-4456-9438-783cf36e291e\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 31 23:47:25.072937 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:47:25.072824    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/d3918f5e-491a-4456-9438-783cf36e291e-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"d3918f5e-491a-4456-9438-783cf36e291e\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 31 23:47:25.072937 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:47:25.072860    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/d3918f5e-491a-4456-9438-783cf36e291e-config-out\") pod \"alertmanager-main-0\" (UID: \"d3918f5e-491a-4456-9438-783cf36e291e\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 31 23:47:25.072937 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:47:25.072894    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/d3918f5e-491a-4456-9438-783cf36e291e-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"d3918f5e-491a-4456-9438-783cf36e291e\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 31 23:47:25.073534 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:47:25.073090    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/d3918f5e-491a-4456-9438-783cf36e291e-web-config\") pod \"alertmanager-main-0\" (UID: \"d3918f5e-491a-4456-9438-783cf36e291e\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 31 23:47:25.073534 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:47:25.073221    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/d3918f5e-491a-4456-9438-783cf36e291e-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"d3918f5e-491a-4456-9438-783cf36e291e\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 31 23:47:25.073534 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:47:25.073256    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d3918f5e-491a-4456-9438-783cf36e291e-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"d3918f5e-491a-4456-9438-783cf36e291e\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 31 23:47:25.073534 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:47:25.073344    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/d3918f5e-491a-4456-9438-783cf36e291e-config-volume\") pod \"alertmanager-main-0\" (UID: \"d3918f5e-491a-4456-9438-783cf36e291e\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 31 23:47:25.104572 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:47:25.104538    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gttdm_b32e6d42-e4e0-4113-b8ce-f5d5c5dc13f7/konnectivity-agent/0.log" limit=1024
Jul 31 23:47:25.174824 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:47:25.174722    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/d3918f5e-491a-4456-9438-783cf36e291e-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"d3918f5e-491a-4456-9438-783cf36e291e\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 31 23:47:25.174824 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:47:25.174777    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/d3918f5e-491a-4456-9438-783cf36e291e-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"d3918f5e-491a-4456-9438-783cf36e291e\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 31 23:47:25.174824 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:47:25.174815    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/d3918f5e-491a-4456-9438-783cf36e291e-config-out\") pod \"alertmanager-main-0\" (UID: \"d3918f5e-491a-4456-9438-783cf36e291e\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 31 23:47:25.175104 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:47:25.174842    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/d3918f5e-491a-4456-9438-783cf36e291e-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"d3918f5e-491a-4456-9438-783cf36e291e\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 31 23:47:25.175104 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:47:25.174877    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/d3918f5e-491a-4456-9438-783cf36e291e-web-config\") pod \"alertmanager-main-0\" (UID: \"d3918f5e-491a-4456-9438-783cf36e291e\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 31 23:47:25.175104 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:47:25.174900    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/d3918f5e-491a-4456-9438-783cf36e291e-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"d3918f5e-491a-4456-9438-783cf36e291e\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 31 23:47:25.175104 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:47:25.174921    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d3918f5e-491a-4456-9438-783cf36e291e-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"d3918f5e-491a-4456-9438-783cf36e291e\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 31 23:47:25.175104 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:47:25.174969    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/d3918f5e-491a-4456-9438-783cf36e291e-config-volume\") pod \"alertmanager-main-0\" (UID: \"d3918f5e-491a-4456-9438-783cf36e291e\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 31 23:47:25.175104 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:47:25.174997    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d3918f5e-491a-4456-9438-783cf36e291e-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"d3918f5e-491a-4456-9438-783cf36e291e\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 31 23:47:25.175104 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:47:25.175024    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/d3918f5e-491a-4456-9438-783cf36e291e-tls-assets\") pod \"alertmanager-main-0\" (UID: \"d3918f5e-491a-4456-9438-783cf36e291e\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 31 23:47:25.175104 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:47:25.175061    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/d3918f5e-491a-4456-9438-783cf36e291e-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"d3918f5e-491a-4456-9438-783cf36e291e\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 31 23:47:25.175104 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:47:25.175083    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fn2w6\" (UniqueName: \"kubernetes.io/projected/d3918f5e-491a-4456-9438-783cf36e291e-kube-api-access-fn2w6\") pod \"alertmanager-main-0\" (UID: \"d3918f5e-491a-4456-9438-783cf36e291e\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 31 23:47:25.175711 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:47:25.175105    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/d3918f5e-491a-4456-9438-783cf36e291e-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"d3918f5e-491a-4456-9438-783cf36e291e\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 31 23:47:25.176013 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:47:25.175988    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d3918f5e-491a-4456-9438-783cf36e291e-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"d3918f5e-491a-4456-9438-783cf36e291e\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 31 23:47:25.176663 ip-10-0-136-108 kubenswrapper[2578]: E0731 23:47:25.176154    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/d3918f5e-491a-4456-9438-783cf36e291e-alertmanager-trusted-ca-bundle podName:d3918f5e-491a-4456-9438-783cf36e291e nodeName:}" failed. No retries permitted until 2026-07-31 23:47:25.676131653 +0000 UTC m=+69.643882406 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "alertmanager-trusted-ca-bundle" (UniqueName: "kubernetes.io/configmap/d3918f5e-491a-4456-9438-783cf36e291e-alertmanager-trusted-ca-bundle") pod "alertmanager-main-0" (UID: "d3918f5e-491a-4456-9438-783cf36e291e") : configmap references non-existent config key: ca-bundle.crt
Jul 31 23:47:25.176663 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:47:25.176459    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/d3918f5e-491a-4456-9438-783cf36e291e-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"d3918f5e-491a-4456-9438-783cf36e291e\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 31 23:47:25.178610 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:47:25.178249    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/d3918f5e-491a-4456-9438-783cf36e291e-web-config\") pod \"alertmanager-main-0\" (UID: \"d3918f5e-491a-4456-9438-783cf36e291e\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 31 23:47:25.178813 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:47:25.178766    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/d3918f5e-491a-4456-9438-783cf36e291e-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"d3918f5e-491a-4456-9438-783cf36e291e\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 31 23:47:25.178921 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:47:25.178817    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/d3918f5e-491a-4456-9438-783cf36e291e-config-volume\") pod \"alertmanager-main-0\" (UID: \"d3918f5e-491a-4456-9438-783cf36e291e\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 31 23:47:25.179786 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:47:25.179733    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/d3918f5e-491a-4456-9438-783cf36e291e-tls-assets\") pod \"alertmanager-main-0\" (UID: \"d3918f5e-491a-4456-9438-783cf36e291e\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 31 23:47:25.180199 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:47:25.180162    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/d3918f5e-491a-4456-9438-783cf36e291e-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"d3918f5e-491a-4456-9438-783cf36e291e\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 31 23:47:25.180298 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:47:25.180278    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/d3918f5e-491a-4456-9438-783cf36e291e-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"d3918f5e-491a-4456-9438-783cf36e291e\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 31 23:47:25.181471 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:47:25.181422    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/d3918f5e-491a-4456-9438-783cf36e291e-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"d3918f5e-491a-4456-9438-783cf36e291e\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 31 23:47:25.182240 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:47:25.182196    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/d3918f5e-491a-4456-9438-783cf36e291e-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"d3918f5e-491a-4456-9438-783cf36e291e\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 31 23:47:25.182922 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:47:25.182902    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/d3918f5e-491a-4456-9438-783cf36e291e-config-out\") pod \"alertmanager-main-0\" (UID: \"d3918f5e-491a-4456-9438-783cf36e291e\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 31 23:47:25.191136 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:47:25.191111    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fn2w6\" (UniqueName: \"kubernetes.io/projected/d3918f5e-491a-4456-9438-783cf36e291e-kube-api-access-fn2w6\") pod \"alertmanager-main-0\" (UID: \"d3918f5e-491a-4456-9438-783cf36e291e\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 31 23:47:25.206151 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:47:25.206086    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gttdm_b32e6d42-e4e0-4113-b8ce-f5d5c5dc13f7/konnectivity-agent/0.log" limit=1024
Jul 31 23:47:25.679188 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:47:25.679126    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d3918f5e-491a-4456-9438-783cf36e291e-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"d3918f5e-491a-4456-9438-783cf36e291e\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 31 23:47:25.680010 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:47:25.679989    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d3918f5e-491a-4456-9438-783cf36e291e-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"d3918f5e-491a-4456-9438-783cf36e291e\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 31 23:47:25.725834 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:47:25.725788    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gttdm_b32e6d42-e4e0-4113-b8ce-f5d5c5dc13f7/konnectivity-agent/0.log" limit=1024
Jul 31 23:47:25.828266 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:47:25.828228    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gttdm_b32e6d42-e4e0-4113-b8ce-f5d5c5dc13f7/konnectivity-agent/0.log" limit=1024
Jul 31 23:47:25.946052 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:47:25.945972    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Jul 31 23:47:26.862787 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:47:26.862739    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gttdm_b32e6d42-e4e0-4113-b8ce-f5d5c5dc13f7/konnectivity-agent/0.log" limit=1024
Jul 31 23:47:26.964580 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:47:26.964539    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gttdm_b32e6d42-e4e0-4113-b8ce-f5d5c5dc13f7/konnectivity-agent/0.log" limit=1024
Jul 31 23:47:26.972676 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:47:26.972641    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-6fb556c5d9-8dt4b"]
Jul 31 23:47:26.976325 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:47:26.976298    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-6fb556c5d9-8dt4b"
Jul 31 23:47:26.978377 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:47:26.978351    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\""
Jul 31 23:47:26.978496 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:47:26.978375    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\""
Jul 31 23:47:26.978676 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:47:26.978655    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\""
Jul 31 23:47:26.978758 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:47:26.978666    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-a0emods4aj3pt\""
Jul 31 23:47:26.978838 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:47:26.978823    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-pv69l\""
Jul 31 23:47:26.978878 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:47:26.978865    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\""
Jul 31 23:47:26.978911 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:47:26.978888    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\""
Jul 31 23:47:26.988102 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:47:26.988076    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-6fb556c5d9-8dt4b"]
Jul 31 23:47:27.064584 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:47:27.064547    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gttdm_b32e6d42-e4e0-4113-b8ce-f5d5c5dc13f7/konnectivity-agent/0.log" limit=1024
Jul 31 23:47:27.088907 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:47:27.088867    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cdtnm\" (UniqueName: \"kubernetes.io/projected/a2d3ccdb-0424-404c-a0b6-d72f56fbbb03-kube-api-access-cdtnm\") pod \"thanos-querier-6fb556c5d9-8dt4b\" (UID: \"a2d3ccdb-0424-404c-a0b6-d72f56fbbb03\") " pod="openshift-monitoring/thanos-querier-6fb556c5d9-8dt4b"
Jul 31 23:47:27.088907 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:47:27.088912    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a2d3ccdb-0424-404c-a0b6-d72f56fbbb03-metrics-client-ca\") pod \"thanos-querier-6fb556c5d9-8dt4b\" (UID: \"a2d3ccdb-0424-404c-a0b6-d72f56fbbb03\") " pod="openshift-monitoring/thanos-querier-6fb556c5d9-8dt4b"
Jul 31 23:47:27.089164 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:47:27.088932    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/a2d3ccdb-0424-404c-a0b6-d72f56fbbb03-secret-grpc-tls\") pod \"thanos-querier-6fb556c5d9-8dt4b\" (UID: \"a2d3ccdb-0424-404c-a0b6-d72f56fbbb03\") " pod="openshift-monitoring/thanos-querier-6fb556c5d9-8dt4b"
Jul 31 23:47:27.089164 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:47:27.089079    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/a2d3ccdb-0424-404c-a0b6-d72f56fbbb03-secret-thanos-querier-tls\") pod \"thanos-querier-6fb556c5d9-8dt4b\" (UID: \"a2d3ccdb-0424-404c-a0b6-d72f56fbbb03\") " pod="openshift-monitoring/thanos-querier-6fb556c5d9-8dt4b"
Jul 31 23:47:27.089164 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:47:27.089158    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/a2d3ccdb-0424-404c-a0b6-d72f56fbbb03-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-6fb556c5d9-8dt4b\" (UID: \"a2d3ccdb-0424-404c-a0b6-d72f56fbbb03\") " pod="openshift-monitoring/thanos-querier-6fb556c5d9-8dt4b"
Jul 31 23:47:27.089334 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:47:27.089212    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/a2d3ccdb-0424-404c-a0b6-d72f56fbbb03-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-6fb556c5d9-8dt4b\" (UID: \"a2d3ccdb-0424-404c-a0b6-d72f56fbbb03\") " pod="openshift-monitoring/thanos-querier-6fb556c5d9-8dt4b"
Jul 31 23:47:27.089334 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:47:27.089242    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/a2d3ccdb-0424-404c-a0b6-d72f56fbbb03-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-6fb556c5d9-8dt4b\" (UID: \"a2d3ccdb-0424-404c-a0b6-d72f56fbbb03\") " pod="openshift-monitoring/thanos-querier-6fb556c5d9-8dt4b"
Jul 31 23:47:27.089334 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:47:27.089277    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/a2d3ccdb-0424-404c-a0b6-d72f56fbbb03-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-6fb556c5d9-8dt4b\" (UID: \"a2d3ccdb-0424-404c-a0b6-d72f56fbbb03\") " pod="openshift-monitoring/thanos-querier-6fb556c5d9-8dt4b"
Jul 31 23:47:27.134114 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:47:27.134007    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gttdm_b32e6d42-e4e0-4113-b8ce-f5d5c5dc13f7/konnectivity-agent/0.log" limit=1024
Jul 31 23:47:27.190167 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:47:27.190127    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/a2d3ccdb-0424-404c-a0b6-d72f56fbbb03-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-6fb556c5d9-8dt4b\" (UID: \"a2d3ccdb-0424-404c-a0b6-d72f56fbbb03\") " pod="openshift-monitoring/thanos-querier-6fb556c5d9-8dt4b"
Jul 31 23:47:27.190382 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:47:27.190194    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/a2d3ccdb-0424-404c-a0b6-d72f56fbbb03-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-6fb556c5d9-8dt4b\" (UID: \"a2d3ccdb-0424-404c-a0b6-d72f56fbbb03\") " pod="openshift-monitoring/thanos-querier-6fb556c5d9-8dt4b"
Jul 31 23:47:27.190382 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:47:27.190226    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/a2d3ccdb-0424-404c-a0b6-d72f56fbbb03-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-6fb556c5d9-8dt4b\" (UID: \"a2d3ccdb-0424-404c-a0b6-d72f56fbbb03\") " pod="openshift-monitoring/thanos-querier-6fb556c5d9-8dt4b"
Jul 31 23:47:27.190382 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:47:27.190260    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/a2d3ccdb-0424-404c-a0b6-d72f56fbbb03-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-6fb556c5d9-8dt4b\" (UID: \"a2d3ccdb-0424-404c-a0b6-d72f56fbbb03\") " pod="openshift-monitoring/thanos-querier-6fb556c5d9-8dt4b"
Jul 31 23:47:27.190382 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:47:27.190292    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cdtnm\" (UniqueName: \"kubernetes.io/projected/a2d3ccdb-0424-404c-a0b6-d72f56fbbb03-kube-api-access-cdtnm\") pod \"thanos-querier-6fb556c5d9-8dt4b\" (UID: \"a2d3ccdb-0424-404c-a0b6-d72f56fbbb03\") " pod="openshift-monitoring/thanos-querier-6fb556c5d9-8dt4b"
Jul 31 23:47:27.190382 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:47:27.190324    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a2d3ccdb-0424-404c-a0b6-d72f56fbbb03-metrics-client-ca\") pod \"thanos-querier-6fb556c5d9-8dt4b\" (UID: \"a2d3ccdb-0424-404c-a0b6-d72f56fbbb03\") " pod="openshift-monitoring/thanos-querier-6fb556c5d9-8dt4b"
Jul 31 23:47:27.190382 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:47:27.190348    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/a2d3ccdb-0424-404c-a0b6-d72f56fbbb03-secret-grpc-tls\") pod \"thanos-querier-6fb556c5d9-8dt4b\" (UID: \"a2d3ccdb-0424-404c-a0b6-d72f56fbbb03\") " pod="openshift-monitoring/thanos-querier-6fb556c5d9-8dt4b"
Jul 31 23:47:27.190588 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:47:27.190411    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/a2d3ccdb-0424-404c-a0b6-d72f56fbbb03-secret-thanos-querier-tls\") pod \"thanos-querier-6fb556c5d9-8dt4b\" (UID: \"a2d3ccdb-0424-404c-a0b6-d72f56fbbb03\") " pod="openshift-monitoring/thanos-querier-6fb556c5d9-8dt4b"
Jul 31 23:47:27.191591 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:47:27.191558    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a2d3ccdb-0424-404c-a0b6-d72f56fbbb03-metrics-client-ca\") pod \"thanos-querier-6fb556c5d9-8dt4b\" (UID: \"a2d3ccdb-0424-404c-a0b6-d72f56fbbb03\") " pod="openshift-monitoring/thanos-querier-6fb556c5d9-8dt4b"
Jul 31 23:47:27.193367 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:47:27.193323    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/a2d3ccdb-0424-404c-a0b6-d72f56fbbb03-secret-thanos-querier-tls\") pod \"thanos-querier-6fb556c5d9-8dt4b\" (UID: \"a2d3ccdb-0424-404c-a0b6-d72f56fbbb03\") " pod="openshift-monitoring/thanos-querier-6fb556c5d9-8dt4b"
Jul 31 23:47:27.193476 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:47:27.193454    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/a2d3ccdb-0424-404c-a0b6-d72f56fbbb03-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-6fb556c5d9-8dt4b\" (UID: \"a2d3ccdb-0424-404c-a0b6-d72f56fbbb03\") " pod="openshift-monitoring/thanos-querier-6fb556c5d9-8dt4b"
Jul 31 23:47:27.193527 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:47:27.193508    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/a2d3ccdb-0424-404c-a0b6-d72f56fbbb03-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-6fb556c5d9-8dt4b\" (UID: \"a2d3ccdb-0424-404c-a0b6-d72f56fbbb03\") " pod="openshift-monitoring/thanos-querier-6fb556c5d9-8dt4b"
Jul 31 23:47:27.193725 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:47:27.193699    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/a2d3ccdb-0424-404c-a0b6-d72f56fbbb03-secret-grpc-tls\") pod \"thanos-querier-6fb556c5d9-8dt4b\" (UID: \"a2d3ccdb-0424-404c-a0b6-d72f56fbbb03\") " pod="openshift-monitoring/thanos-querier-6fb556c5d9-8dt4b"
Jul 31 23:47:27.193804 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:47:27.193726    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/a2d3ccdb-0424-404c-a0b6-d72f56fbbb03-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-6fb556c5d9-8dt4b\" (UID: \"a2d3ccdb-0424-404c-a0b6-d72f56fbbb03\") " pod="openshift-monitoring/thanos-querier-6fb556c5d9-8dt4b"
Jul 31 23:47:27.193907 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:47:27.193888    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/a2d3ccdb-0424-404c-a0b6-d72f56fbbb03-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-6fb556c5d9-8dt4b\" (UID: \"a2d3ccdb-0424-404c-a0b6-d72f56fbbb03\") " pod="openshift-monitoring/thanos-querier-6fb556c5d9-8dt4b"
Jul 31 23:47:27.198013 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:47:27.197991    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cdtnm\" (UniqueName: \"kubernetes.io/projected/a2d3ccdb-0424-404c-a0b6-d72f56fbbb03-kube-api-access-cdtnm\") pod \"thanos-querier-6fb556c5d9-8dt4b\" (UID: \"a2d3ccdb-0424-404c-a0b6-d72f56fbbb03\") " pod="openshift-monitoring/thanos-querier-6fb556c5d9-8dt4b"
Jul 31 23:47:27.287532 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:47:27.287487    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-6fb556c5d9-8dt4b"
Jul 31 23:47:27.300146 ip-10-0-136-108 kubenswrapper[2578]: W0731 23:47:27.300111    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod49955a37_a47a_48ee_826c_6aaa95715852.slice/crio-bca0c8e3db96956bf8af33d11622c90797da23b070de1021abf5412018955d7f WatchSource:0}: Error finding container bca0c8e3db96956bf8af33d11622c90797da23b070de1021abf5412018955d7f: Status 404 returned error can't find the container with id bca0c8e3db96956bf8af33d11622c90797da23b070de1021abf5412018955d7f
Jul 31 23:47:27.464609 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:47:27.464581    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 31 23:47:27.467953 ip-10-0-136-108 kubenswrapper[2578]: W0731 23:47:27.467924    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd3918f5e_491a_4456_9438_783cf36e291e.slice/crio-a30c7c264814496c844645807409405676c8df863da78bfb4b372cf3e92dfc31 WatchSource:0}: Error finding container a30c7c264814496c844645807409405676c8df863da78bfb4b372cf3e92dfc31: Status 404 returned error can't find the container with id a30c7c264814496c844645807409405676c8df863da78bfb4b372cf3e92dfc31
Jul 31 23:47:27.473223 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:47:27.473199    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-6fb556c5d9-8dt4b"]
Jul 31 23:47:27.475918 ip-10-0-136-108 kubenswrapper[2578]: W0731 23:47:27.475896    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda2d3ccdb_0424_404c_a0b6_d72f56fbbb03.slice/crio-505c95428890e6d24ad806a4198d3dda45b059ad1e82dd6747a833d18ccf7fab WatchSource:0}: Error finding container 505c95428890e6d24ad806a4198d3dda45b059ad1e82dd6747a833d18ccf7fab: Status 404 returned error can't find the container with id 505c95428890e6d24ad806a4198d3dda45b059ad1e82dd6747a833d18ccf7fab
Jul 31 23:47:27.894333 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:47:27.894288    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-8d9rr" event={"ID":"0503a101-ddee-440a-ae03-e76fe30685e7","Type":"ContainerStarted","Data":"d5b4d5e05c2461c5a901db194c8f927a5c5fed59eb0b54e88d13d231bca3855c"}
Jul 31 23:47:27.894512 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:47:27.894472    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-8d9rr"
Jul 31 23:47:27.896037 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:47:27.895998    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-wxg27" event={"ID":"49955a37-a47a-48ee-826c-6aaa95715852","Type":"ContainerStarted","Data":"bca0c8e3db96956bf8af33d11622c90797da23b070de1021abf5412018955d7f"}
Jul 31 23:47:27.897990 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:47:27.897963    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-6fb556c5d9-8dt4b" event={"ID":"a2d3ccdb-0424-404c-a0b6-d72f56fbbb03","Type":"ContainerStarted","Data":"505c95428890e6d24ad806a4198d3dda45b059ad1e82dd6747a833d18ccf7fab"}
Jul 31 23:47:27.900432 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:47:27.900407    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-xlv68" event={"ID":"9d113c99-25bd-49a4-b8e2-f5c46f3ecc02","Type":"ContainerStarted","Data":"5a1b9d86d02c1826e01b266280d9d097492a6b9cfa265d51fc6d9b00fa0411fa"}
Jul 31 23:47:27.902941 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:47:27.902906    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"d3918f5e-491a-4456-9438-783cf36e291e","Type":"ContainerStarted","Data":"a30c7c264814496c844645807409405676c8df863da78bfb4b372cf3e92dfc31"}
Jul 31 23:47:27.912148 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:47:27.912087    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-8d9rr" podStartSLOduration=66.27156463 podStartE2EDuration="1m10.912067014s" totalImagesPullingTime="4.640502384s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 23:46:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 23:47:27.910921742 +0000 UTC m=+71.878672501" watchObservedRunningTime="2026-07-31 23:47:27.912067014 +0000 UTC m=+71.879817771"
Jul 31 23:47:27.927392 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:47:27.927343    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-xlv68" podStartSLOduration=66.270049882 podStartE2EDuration="1m10.927320084s" totalImagesPullingTime="4.657270202s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-31 23:46:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 23:47:27.926303803 +0000 UTC m=+71.894054588" watchObservedRunningTime="2026-07-31 23:47:27.927320084 +0000 UTC m=+71.895070838"
Jul 31 23:47:28.410371 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:47:28.410335    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-5b86cb7b86-269w6"]
Jul 31 23:47:28.413739 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:47:28.413714    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-5b86cb7b86-269w6"
Jul 31 23:47:28.416037 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:47:28.415759    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\""
Jul 31 23:47:28.416037 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:47:28.415901    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-wmg9h\""
Jul 31 23:47:28.416951 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:47:28.416520    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\""
Jul 31 23:47:28.416951 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:47:28.416617    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Jul 31 23:47:28.416951 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:47:28.416526    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-9uhk0dl22onsg\""
Jul 31 23:47:28.416951 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:47:28.416775    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\""
Jul 31 23:47:28.426222 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:47:28.426143    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-5b86cb7b86-269w6"]
Jul 31 23:47:28.502449 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:47:28.502416    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/8cec3464-a28f-45b2-bb1e-1464eb049d79-client-ca-bundle\") pod \"metrics-server-5b86cb7b86-269w6\" (UID: \"8cec3464-a28f-45b2-bb1e-1464eb049d79\") " pod="openshift-monitoring/metrics-server-5b86cb7b86-269w6"
Jul 31 23:47:28.502604 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:47:28.502480    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/8cec3464-a28f-45b2-bb1e-1464eb049d79-secret-metrics-server-client-certs\") pod \"metrics-server-5b86cb7b86-269w6\" (UID: \"8cec3464-a28f-45b2-bb1e-1464eb049d79\") " pod="openshift-monitoring/metrics-server-5b86cb7b86-269w6"
Jul 31 23:47:28.502604 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:47:28.502509    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/8cec3464-a28f-45b2-bb1e-1464eb049d79-secret-metrics-server-tls\") pod \"metrics-server-5b86cb7b86-269w6\" (UID: \"8cec3464-a28f-45b2-bb1e-1464eb049d79\") " pod="openshift-monitoring/metrics-server-5b86cb7b86-269w6"
Jul 31 23:47:28.502604 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:47:28.502577    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/8cec3464-a28f-45b2-bb1e-1464eb049d79-audit-log\") pod \"metrics-server-5b86cb7b86-269w6\" (UID: \"8cec3464-a28f-45b2-bb1e-1464eb049d79\") " pod="openshift-monitoring/metrics-server-5b86cb7b86-269w6"
Jul 31 23:47:28.502604 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:47:28.502604    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8tq9m\" (UniqueName: \"kubernetes.io/projected/8cec3464-a28f-45b2-bb1e-1464eb049d79-kube-api-access-8tq9m\") pod \"metrics-server-5b86cb7b86-269w6\" (UID: \"8cec3464-a28f-45b2-bb1e-1464eb049d79\") " pod="openshift-monitoring/metrics-server-5b86cb7b86-269w6"
Jul 31 23:47:28.502767 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:47:28.502629    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/8cec3464-a28f-45b2-bb1e-1464eb049d79-metrics-server-audit-profiles\") pod \"metrics-server-5b86cb7b86-269w6\" (UID: \"8cec3464-a28f-45b2-bb1e-1464eb049d79\") " pod="openshift-monitoring/metrics-server-5b86cb7b86-269w6"
Jul 31 23:47:28.502767 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:47:28.502667    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8cec3464-a28f-45b2-bb1e-1464eb049d79-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-5b86cb7b86-269w6\" (UID: \"8cec3464-a28f-45b2-bb1e-1464eb049d79\") " pod="openshift-monitoring/metrics-server-5b86cb7b86-269w6"
Jul 31 23:47:28.603797 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:47:28.603628    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/8cec3464-a28f-45b2-bb1e-1464eb049d79-client-ca-bundle\") pod \"metrics-server-5b86cb7b86-269w6\" (UID: \"8cec3464-a28f-45b2-bb1e-1464eb049d79\") " pod="openshift-monitoring/metrics-server-5b86cb7b86-269w6"
Jul 31 23:47:28.603797 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:47:28.603696    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/8cec3464-a28f-45b2-bb1e-1464eb049d79-secret-metrics-server-client-certs\") pod \"metrics-server-5b86cb7b86-269w6\" (UID: \"8cec3464-a28f-45b2-bb1e-1464eb049d79\") " pod="openshift-monitoring/metrics-server-5b86cb7b86-269w6"
Jul 31 23:47:28.603797 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:47:28.603729    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/8cec3464-a28f-45b2-bb1e-1464eb049d79-secret-metrics-server-tls\") pod \"metrics-server-5b86cb7b86-269w6\" (UID: \"8cec3464-a28f-45b2-bb1e-1464eb049d79\") " pod="openshift-monitoring/metrics-server-5b86cb7b86-269w6"
Jul 31 23:47:28.603797 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:47:28.603775    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/8cec3464-a28f-45b2-bb1e-1464eb049d79-audit-log\") pod \"metrics-server-5b86cb7b86-269w6\" (UID: \"8cec3464-a28f-45b2-bb1e-1464eb049d79\") " pod="openshift-monitoring/metrics-server-5b86cb7b86-269w6"
Jul 31 23:47:28.604739 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:47:28.604712    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8tq9m\" (UniqueName: \"kubernetes.io/projected/8cec3464-a28f-45b2-bb1e-1464eb049d79-kube-api-access-8tq9m\") pod \"metrics-server-5b86cb7b86-269w6\" (UID: \"8cec3464-a28f-45b2-bb1e-1464eb049d79\") " pod="openshift-monitoring/metrics-server-5b86cb7b86-269w6"
Jul 31 23:47:28.604854 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:47:28.604732    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/8cec3464-a28f-45b2-bb1e-1464eb049d79-audit-log\") pod \"metrics-server-5b86cb7b86-269w6\" (UID: \"8cec3464-a28f-45b2-bb1e-1464eb049d79\") " pod="openshift-monitoring/metrics-server-5b86cb7b86-269w6"
Jul 31 23:47:28.604854 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:47:28.604774    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/8cec3464-a28f-45b2-bb1e-1464eb049d79-metrics-server-audit-profiles\") pod \"metrics-server-5b86cb7b86-269w6\" (UID: \"8cec3464-a28f-45b2-bb1e-1464eb049d79\") " pod="openshift-monitoring/metrics-server-5b86cb7b86-269w6"
Jul 31 23:47:28.604854 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:47:28.604836    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8cec3464-a28f-45b2-bb1e-1464eb049d79-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-5b86cb7b86-269w6\" (UID: \"8cec3464-a28f-45b2-bb1e-1464eb049d79\") " pod="openshift-monitoring/metrics-server-5b86cb7b86-269w6"
Jul 31 23:47:28.605682 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:47:28.605656    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8cec3464-a28f-45b2-bb1e-1464eb049d79-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-5b86cb7b86-269w6\" (UID: \"8cec3464-a28f-45b2-bb1e-1464eb049d79\") " pod="openshift-monitoring/metrics-server-5b86cb7b86-269w6"
Jul 31 23:47:28.606004 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:47:28.605977    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/8cec3464-a28f-45b2-bb1e-1464eb049d79-metrics-server-audit-profiles\") pod \"metrics-server-5b86cb7b86-269w6\" (UID: \"8cec3464-a28f-45b2-bb1e-1464eb049d79\") " pod="openshift-monitoring/metrics-server-5b86cb7b86-269w6"
Jul 31 23:47:28.606875 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:47:28.606821    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/8cec3464-a28f-45b2-bb1e-1464eb049d79-secret-metrics-server-client-certs\") pod \"metrics-server-5b86cb7b86-269w6\" (UID: \"8cec3464-a28f-45b2-bb1e-1464eb049d79\") " pod="openshift-monitoring/metrics-server-5b86cb7b86-269w6"
Jul 31 23:47:28.608551 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:47:28.608501    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/8cec3464-a28f-45b2-bb1e-1464eb049d79-secret-metrics-server-tls\") pod \"metrics-server-5b86cb7b86-269w6\" (UID: \"8cec3464-a28f-45b2-bb1e-1464eb049d79\") " pod="openshift-monitoring/metrics-server-5b86cb7b86-269w6"
Jul 31 23:47:28.611316 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:47:28.611249    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/8cec3464-a28f-45b2-bb1e-1464eb049d79-client-ca-bundle\") pod \"metrics-server-5b86cb7b86-269w6\" (UID: \"8cec3464-a28f-45b2-bb1e-1464eb049d79\") " pod="openshift-monitoring/metrics-server-5b86cb7b86-269w6"
Jul 31 23:47:28.613693 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:47:28.613648    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8tq9m\" (UniqueName: \"kubernetes.io/projected/8cec3464-a28f-45b2-bb1e-1464eb049d79-kube-api-access-8tq9m\") pod \"metrics-server-5b86cb7b86-269w6\" (UID: \"8cec3464-a28f-45b2-bb1e-1464eb049d79\") " pod="openshift-monitoring/metrics-server-5b86cb7b86-269w6"
Jul 31 23:47:28.631111 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:47:28.631026    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gttdm_b32e6d42-e4e0-4113-b8ce-f5d5c5dc13f7/konnectivity-agent/0.log" limit=1024
Jul 31 23:47:28.641453 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:47:28.640970    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-6dffbbd67-lzqqq"]
Jul 31 23:47:28.644273 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:47:28.644251    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-6dffbbd67-lzqqq"
Jul 31 23:47:28.646714 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:47:28.646406    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\""
Jul 31 23:47:28.646714 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:47:28.646406    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-6zxwh\""
Jul 31 23:47:28.654322 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:47:28.654258    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-6dffbbd67-lzqqq"]
Jul 31 23:47:28.705863 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:47:28.705833    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kqhgv\" (UniqueName: \"kubernetes.io/projected/8a2dfa37-6798-4b86-be3b-166e942af34d-kube-api-access-kqhgv\") pod \"monitoring-plugin-6dffbbd67-lzqqq\" (UID: \"8a2dfa37-6798-4b86-be3b-166e942af34d\") " pod="openshift-monitoring/monitoring-plugin-6dffbbd67-lzqqq"
Jul 31 23:47:28.706027 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:47:28.705885    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/8a2dfa37-6798-4b86-be3b-166e942af34d-monitoring-plugin-cert\") pod \"monitoring-plugin-6dffbbd67-lzqqq\" (UID: \"8a2dfa37-6798-4b86-be3b-166e942af34d\") " pod="openshift-monitoring/monitoring-plugin-6dffbbd67-lzqqq"
Jul 31 23:47:28.726907 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:47:28.726698    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gttdm_b32e6d42-e4e0-4113-b8ce-f5d5c5dc13f7/konnectivity-agent/0.log" limit=1024
Jul 31 23:47:28.727042 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:47:28.727002    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-5b86cb7b86-269w6"
Jul 31 23:47:28.807296 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:47:28.807259    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kqhgv\" (UniqueName: \"kubernetes.io/projected/8a2dfa37-6798-4b86-be3b-166e942af34d-kube-api-access-kqhgv\") pod \"monitoring-plugin-6dffbbd67-lzqqq\" (UID: \"8a2dfa37-6798-4b86-be3b-166e942af34d\") " pod="openshift-monitoring/monitoring-plugin-6dffbbd67-lzqqq"
Jul 31 23:47:28.807491 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:47:28.807330    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/8a2dfa37-6798-4b86-be3b-166e942af34d-monitoring-plugin-cert\") pod \"monitoring-plugin-6dffbbd67-lzqqq\" (UID: \"8a2dfa37-6798-4b86-be3b-166e942af34d\") " pod="openshift-monitoring/monitoring-plugin-6dffbbd67-lzqqq"
Jul 31 23:47:28.810004 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:47:28.809975    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/8a2dfa37-6798-4b86-be3b-166e942af34d-monitoring-plugin-cert\") pod \"monitoring-plugin-6dffbbd67-lzqqq\" (UID: \"8a2dfa37-6798-4b86-be3b-166e942af34d\") " pod="openshift-monitoring/monitoring-plugin-6dffbbd67-lzqqq"
Jul 31 23:47:28.815380 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:47:28.815355    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kqhgv\" (UniqueName: \"kubernetes.io/projected/8a2dfa37-6798-4b86-be3b-166e942af34d-kube-api-access-kqhgv\") pod \"monitoring-plugin-6dffbbd67-lzqqq\" (UID: \"8a2dfa37-6798-4b86-be3b-166e942af34d\") " pod="openshift-monitoring/monitoring-plugin-6dffbbd67-lzqqq"
Jul 31 23:47:28.827091 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:47:28.827062    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gttdm_b32e6d42-e4e0-4113-b8ce-f5d5c5dc13f7/konnectivity-agent/0.log" limit=1024
Jul 31 23:47:28.855770 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:47:28.855741    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-cjvb8"
Jul 31 23:47:28.909347 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:47:28.909299    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"d3918f5e-491a-4456-9438-783cf36e291e","Type":"ContainerStarted","Data":"48d1d67f9cf995293f6001e4aebc50f9caf343c754f307fa8077ac10440c2083"}
Jul 31 23:47:28.911210 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:47:28.911140    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-wxg27" event={"ID":"49955a37-a47a-48ee-826c-6aaa95715852","Type":"ContainerStarted","Data":"2a9d82bd6576e6f20ab7e9bb3225fc2601f85db3dd659a5f7c2c39ff8cb3332e"}
Jul 31 23:47:28.924890 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:47:28.924819    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5d59f9cb75-g6g2x"]
Jul 31 23:47:28.958252 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:47:28.958218    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-6dffbbd67-lzqqq"
Jul 31 23:47:28.970285 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:47:28.970259    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-56655dc5bf-zbmx4"]
Jul 31 23:47:28.973475 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:47:28.973455    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-56655dc5bf-zbmx4"
Jul 31 23:47:28.986057 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:47:28.986034    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\""
Jul 31 23:47:29.002075 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:47:29.002047    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-56655dc5bf-zbmx4"]
Jul 31 23:47:29.008942 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:47:29.008889    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a6aa6fe2-3453-4af0-a7e4-fe4c6930abb9-service-ca\") pod \"console-56655dc5bf-zbmx4\" (UID: \"a6aa6fe2-3453-4af0-a7e4-fe4c6930abb9\") " pod="openshift-console/console-56655dc5bf-zbmx4"
Jul 31 23:47:29.008942 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:47:29.008932    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a6aa6fe2-3453-4af0-a7e4-fe4c6930abb9-oauth-serving-cert\") pod \"console-56655dc5bf-zbmx4\" (UID: \"a6aa6fe2-3453-4af0-a7e4-fe4c6930abb9\") " pod="openshift-console/console-56655dc5bf-zbmx4"
Jul 31 23:47:29.009138 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:47:29.009011    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a6aa6fe2-3453-4af0-a7e4-fe4c6930abb9-console-serving-cert\") pod \"console-56655dc5bf-zbmx4\" (UID: \"a6aa6fe2-3453-4af0-a7e4-fe4c6930abb9\") " pod="openshift-console/console-56655dc5bf-zbmx4"
Jul 31 23:47:29.009138 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:47:29.009065    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/a6aa6fe2-3453-4af0-a7e4-fe4c6930abb9-console-config\") pod \"console-56655dc5bf-zbmx4\" (UID: \"a6aa6fe2-3453-4af0-a7e4-fe4c6930abb9\") " pod="openshift-console/console-56655dc5bf-zbmx4"
Jul 31 23:47:29.009770 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:47:29.009744    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a6aa6fe2-3453-4af0-a7e4-fe4c6930abb9-trusted-ca-bundle\") pod \"console-56655dc5bf-zbmx4\" (UID: \"a6aa6fe2-3453-4af0-a7e4-fe4c6930abb9\") " pod="openshift-console/console-56655dc5bf-zbmx4"
Jul 31 23:47:29.009894 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:47:29.009804    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r6n29\" (UniqueName: \"kubernetes.io/projected/a6aa6fe2-3453-4af0-a7e4-fe4c6930abb9-kube-api-access-r6n29\") pod \"console-56655dc5bf-zbmx4\" (UID: \"a6aa6fe2-3453-4af0-a7e4-fe4c6930abb9\") " pod="openshift-console/console-56655dc5bf-zbmx4"
Jul 31 23:47:29.009894 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:47:29.009859    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a6aa6fe2-3453-4af0-a7e4-fe4c6930abb9-console-oauth-config\") pod \"console-56655dc5bf-zbmx4\" (UID: \"a6aa6fe2-3453-4af0-a7e4-fe4c6930abb9\") " pod="openshift-console/console-56655dc5bf-zbmx4"
Jul 31 23:47:29.111309 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:47:29.111264    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a6aa6fe2-3453-4af0-a7e4-fe4c6930abb9-trusted-ca-bundle\") pod \"console-56655dc5bf-zbmx4\" (UID: \"a6aa6fe2-3453-4af0-a7e4-fe4c6930abb9\") " pod="openshift-console/console-56655dc5bf-zbmx4"
Jul 31 23:47:29.111309 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:47:29.111314    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r6n29\" (UniqueName: \"kubernetes.io/projected/a6aa6fe2-3453-4af0-a7e4-fe4c6930abb9-kube-api-access-r6n29\") pod \"console-56655dc5bf-zbmx4\" (UID: \"a6aa6fe2-3453-4af0-a7e4-fe4c6930abb9\") " pod="openshift-console/console-56655dc5bf-zbmx4"
Jul 31 23:47:29.111580 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:47:29.111516    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a6aa6fe2-3453-4af0-a7e4-fe4c6930abb9-console-oauth-config\") pod \"console-56655dc5bf-zbmx4\" (UID: \"a6aa6fe2-3453-4af0-a7e4-fe4c6930abb9\") " pod="openshift-console/console-56655dc5bf-zbmx4"
Jul 31 23:47:29.111580 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:47:29.111576    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a6aa6fe2-3453-4af0-a7e4-fe4c6930abb9-service-ca\") pod \"console-56655dc5bf-zbmx4\" (UID: \"a6aa6fe2-3453-4af0-a7e4-fe4c6930abb9\") " pod="openshift-console/console-56655dc5bf-zbmx4"
Jul 31 23:47:29.111672 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:47:29.111603    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a6aa6fe2-3453-4af0-a7e4-fe4c6930abb9-oauth-serving-cert\") pod \"console-56655dc5bf-zbmx4\" (UID: \"a6aa6fe2-3453-4af0-a7e4-fe4c6930abb9\") " pod="openshift-console/console-56655dc5bf-zbmx4"
Jul 31 23:47:29.111706 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:47:29.111675    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a6aa6fe2-3453-4af0-a7e4-fe4c6930abb9-console-serving-cert\") pod \"console-56655dc5bf-zbmx4\" (UID: \"a6aa6fe2-3453-4af0-a7e4-fe4c6930abb9\") " pod="openshift-console/console-56655dc5bf-zbmx4"
Jul 31 23:47:29.111811 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:47:29.111716    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/a6aa6fe2-3453-4af0-a7e4-fe4c6930abb9-console-config\") pod \"console-56655dc5bf-zbmx4\" (UID: \"a6aa6fe2-3453-4af0-a7e4-fe4c6930abb9\") " pod="openshift-console/console-56655dc5bf-zbmx4"
Jul 31 23:47:29.112350 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:47:29.112300    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a6aa6fe2-3453-4af0-a7e4-fe4c6930abb9-trusted-ca-bundle\") pod \"console-56655dc5bf-zbmx4\" (UID: \"a6aa6fe2-3453-4af0-a7e4-fe4c6930abb9\") " pod="openshift-console/console-56655dc5bf-zbmx4"
Jul 31 23:47:29.112473 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:47:29.112391    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/a6aa6fe2-3453-4af0-a7e4-fe4c6930abb9-console-config\") pod \"console-56655dc5bf-zbmx4\" (UID: \"a6aa6fe2-3453-4af0-a7e4-fe4c6930abb9\") " pod="openshift-console/console-56655dc5bf-zbmx4"
Jul 31 23:47:29.112638 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:47:29.112618    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a6aa6fe2-3453-4af0-a7e4-fe4c6930abb9-oauth-serving-cert\") pod \"console-56655dc5bf-zbmx4\" (UID: \"a6aa6fe2-3453-4af0-a7e4-fe4c6930abb9\") " pod="openshift-console/console-56655dc5bf-zbmx4"
Jul 31 23:47:29.112970 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:47:29.112940    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a6aa6fe2-3453-4af0-a7e4-fe4c6930abb9-service-ca\") pod \"console-56655dc5bf-zbmx4\" (UID: \"a6aa6fe2-3453-4af0-a7e4-fe4c6930abb9\") " pod="openshift-console/console-56655dc5bf-zbmx4"
Jul 31 23:47:29.114442 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:47:29.114417    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a6aa6fe2-3453-4af0-a7e4-fe4c6930abb9-console-oauth-config\") pod \"console-56655dc5bf-zbmx4\" (UID: \"a6aa6fe2-3453-4af0-a7e4-fe4c6930abb9\") " pod="openshift-console/console-56655dc5bf-zbmx4"
Jul 31 23:47:29.114668 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:47:29.114647    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a6aa6fe2-3453-4af0-a7e4-fe4c6930abb9-console-serving-cert\") pod \"console-56655dc5bf-zbmx4\" (UID: \"a6aa6fe2-3453-4af0-a7e4-fe4c6930abb9\") " pod="openshift-console/console-56655dc5bf-zbmx4"
Jul 31 23:47:29.120561 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:47:29.120535    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r6n29\" (UniqueName: \"kubernetes.io/projected/a6aa6fe2-3453-4af0-a7e4-fe4c6930abb9-kube-api-access-r6n29\") pod \"console-56655dc5bf-zbmx4\" (UID: \"a6aa6fe2-3453-4af0-a7e4-fe4c6930abb9\") " pod="openshift-console/console-56655dc5bf-zbmx4"
Jul 31 23:47:29.283619 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:47:29.283589    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-56655dc5bf-zbmx4"
Jul 31 23:47:29.362666 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:47:29.362520    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-6dffbbd67-lzqqq"]
Jul 31 23:47:29.365482 ip-10-0-136-108 kubenswrapper[2578]: W0731 23:47:29.365444    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8a2dfa37_6798_4b86_be3b_166e942af34d.slice/crio-fd6505e498ec26dea8b12c9e1c9a00247ba7805d9763c9bcdde93afef8a09472 WatchSource:0}: Error finding container fd6505e498ec26dea8b12c9e1c9a00247ba7805d9763c9bcdde93afef8a09472: Status 404 returned error can't find the container with id fd6505e498ec26dea8b12c9e1c9a00247ba7805d9763c9bcdde93afef8a09472
Jul 31 23:47:29.381588 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:47:29.381519    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-5b86cb7b86-269w6"]
Jul 31 23:47:29.385340 ip-10-0-136-108 kubenswrapper[2578]: W0731 23:47:29.385313    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8cec3464_a28f_45b2_bb1e_1464eb049d79.slice/crio-8c69013d708bd11e00c9c6dcd1609fd9e434c6659e6c102c498b8315d509845b WatchSource:0}: Error finding container 8c69013d708bd11e00c9c6dcd1609fd9e434c6659e6c102c498b8315d509845b: Status 404 returned error can't find the container with id 8c69013d708bd11e00c9c6dcd1609fd9e434c6659e6c102c498b8315d509845b
Jul 31 23:47:29.444816 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:47:29.444551    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-56655dc5bf-zbmx4"]
Jul 31 23:47:29.450482 ip-10-0-136-108 kubenswrapper[2578]: W0731 23:47:29.450454    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda6aa6fe2_3453_4af0_a7e4_fe4c6930abb9.slice/crio-9ff2d27d2f62d7e99ddf83574f5270e5e5008603c906a80dd19f477b061935dc WatchSource:0}: Error finding container 9ff2d27d2f62d7e99ddf83574f5270e5e5008603c906a80dd19f477b061935dc: Status 404 returned error can't find the container with id 9ff2d27d2f62d7e99ddf83574f5270e5e5008603c906a80dd19f477b061935dc
Jul 31 23:47:29.910488 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:47:29.910447    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gttdm_b32e6d42-e4e0-4113-b8ce-f5d5c5dc13f7/konnectivity-agent/0.log" limit=1024
Jul 31 23:47:29.921706 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:47:29.921666    2578 generic.go:356] "Generic (PLEG): container finished" podID="49955a37-a47a-48ee-826c-6aaa95715852" containerID="2a9d82bd6576e6f20ab7e9bb3225fc2601f85db3dd659a5f7c2c39ff8cb3332e" exitCode=0
Jul 31 23:47:29.922080 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:47:29.922005    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-wxg27" event={"ID":"49955a37-a47a-48ee-826c-6aaa95715852","Type":"ContainerDied","Data":"2a9d82bd6576e6f20ab7e9bb3225fc2601f85db3dd659a5f7c2c39ff8cb3332e"}
Jul 31 23:47:29.926942 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:47:29.926882    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-6fb556c5d9-8dt4b" event={"ID":"a2d3ccdb-0424-404c-a0b6-d72f56fbbb03","Type":"ContainerStarted","Data":"fdbd11567eaa62d23a3d618408361411b0fe40ab7a9dd18fb3f61ffde4b624db"}
Jul 31 23:47:29.927078 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:47:29.926973    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-6fb556c5d9-8dt4b" event={"ID":"a2d3ccdb-0424-404c-a0b6-d72f56fbbb03","Type":"ContainerStarted","Data":"fe9ca930f3a8294168be254bb49d9647ede392ae67bbfd1e6b45beca94ce5ad0"}
Jul 31 23:47:29.927078 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:47:29.927016    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-6fb556c5d9-8dt4b" event={"ID":"a2d3ccdb-0424-404c-a0b6-d72f56fbbb03","Type":"ContainerStarted","Data":"440f65d7c8d2c270376b7ee31dd41467ba38733d96f8700d2ca8fc904e3b23af"}
Jul 31 23:47:29.931311 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:47:29.931280    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-56655dc5bf-zbmx4" event={"ID":"a6aa6fe2-3453-4af0-a7e4-fe4c6930abb9","Type":"ContainerStarted","Data":"a633810c87c86cdc80207564d0ee5aa6a8d25c86cad50aa1c5d55b327417d7d3"}
Jul 31 23:47:29.931425 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:47:29.931337    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-56655dc5bf-zbmx4" event={"ID":"a6aa6fe2-3453-4af0-a7e4-fe4c6930abb9","Type":"ContainerStarted","Data":"9ff2d27d2f62d7e99ddf83574f5270e5e5008603c906a80dd19f477b061935dc"}
Jul 31 23:47:29.934267 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:47:29.934199    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-5b86cb7b86-269w6" event={"ID":"8cec3464-a28f-45b2-bb1e-1464eb049d79","Type":"ContainerStarted","Data":"8c69013d708bd11e00c9c6dcd1609fd9e434c6659e6c102c498b8315d509845b"}
Jul 31 23:47:29.941822 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:47:29.939681    2578 generic.go:356] "Generic (PLEG): container finished" podID="d3918f5e-491a-4456-9438-783cf36e291e" containerID="48d1d67f9cf995293f6001e4aebc50f9caf343c754f307fa8077ac10440c2083" exitCode=0
Jul 31 23:47:29.941822 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:47:29.939775    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"d3918f5e-491a-4456-9438-783cf36e291e","Type":"ContainerDied","Data":"48d1d67f9cf995293f6001e4aebc50f9caf343c754f307fa8077ac10440c2083"}
Jul 31 23:47:29.943121 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:47:29.943093    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-6dffbbd67-lzqqq" event={"ID":"8a2dfa37-6798-4b86-be3b-166e942af34d","Type":"ContainerStarted","Data":"fd6505e498ec26dea8b12c9e1c9a00247ba7805d9763c9bcdde93afef8a09472"}
Jul 31 23:47:29.976205 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:47:29.975889    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-56655dc5bf-zbmx4" podStartSLOduration=1.975870852 podStartE2EDuration="1.975870852s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 23:47:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 23:47:29.974382998 +0000 UTC m=+73.942133757" watchObservedRunningTime="2026-07-31 23:47:29.975870852 +0000 UTC m=+73.943621611"
Jul 31 23:47:30.007335 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:47:30.007086    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gttdm_b32e6d42-e4e0-4113-b8ce-f5d5c5dc13f7/konnectivity-agent/0.log" limit=1024
Jul 31 23:47:30.262186 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:47:30.262099    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gttdm_b32e6d42-e4e0-4113-b8ce-f5d5c5dc13f7/konnectivity-agent/0.log" limit=1024
Jul 31 23:47:30.951031 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:47:30.950993    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-wxg27" event={"ID":"49955a37-a47a-48ee-826c-6aaa95715852","Type":"ContainerStarted","Data":"5386ac957e8c6898a9a7a74fdf6d574b45c5c253610d9ccf82491da9711090d1"}
Jul 31 23:47:30.951505 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:47:30.951051    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-wxg27" event={"ID":"49955a37-a47a-48ee-826c-6aaa95715852","Type":"ContainerStarted","Data":"e8d8faaeb5c0f9eb0b8ba1cee6ac9d222ba9fd105aef2d882deb5c2076bfb198"}
Jul 31 23:47:30.969342 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:47:30.969116    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-wxg27" podStartSLOduration=6.645728288 podStartE2EDuration="7.969094757s" totalImagesPullingTime="1.255646086s" totalInitContainerRuntime="67.720383ms" isStatefulPod=true podCreationTimestamp="2026-07-31 23:47:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 23:47:30.968797869 +0000 UTC m=+74.936548629" watchObservedRunningTime="2026-07-31 23:47:30.969094757 +0000 UTC m=+74.936845515"
Jul 31 23:47:31.961383 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:47:31.961349    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-6fb556c5d9-8dt4b" event={"ID":"a2d3ccdb-0424-404c-a0b6-d72f56fbbb03","Type":"ContainerStarted","Data":"10f0a1686cb0534fa7c74a158dbe38043e8a1f545e63fe1d4a88356034fa0984"}
Jul 31 23:47:31.961780 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:47:31.961463    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-6fb556c5d9-8dt4b" event={"ID":"a2d3ccdb-0424-404c-a0b6-d72f56fbbb03","Type":"ContainerStarted","Data":"454519031d49dd955062fde6ded59366a4900fc8fd59e031b295c2b9d680d1c3"}
Jul 31 23:47:31.961780 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:47:31.961504    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-6fb556c5d9-8dt4b" event={"ID":"a2d3ccdb-0424-404c-a0b6-d72f56fbbb03","Type":"ContainerStarted","Data":"207b747eb79a89a32a12837022a056846482b58d3c3d68eee7dfb4e7ca58d68b"}
Jul 31 23:47:31.961780 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:47:31.961549    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-6fb556c5d9-8dt4b"
Jul 31 23:47:31.964008 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:47:31.963679    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-5b86cb7b86-269w6" event={"ID":"8cec3464-a28f-45b2-bb1e-1464eb049d79","Type":"ContainerStarted","Data":"71e3998229421536092357739f4ed44bdf8c2b71cbc0b922b313aafc42099389"}
Jul 31 23:47:31.967965 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:47:31.967723    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"d3918f5e-491a-4456-9438-783cf36e291e","Type":"ContainerStarted","Data":"f8fbdf05b608876f9399b90e6aa9ee177182583f1c1134a39f1110346e0e3820"}
Jul 31 23:47:31.967965 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:47:31.967797    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"d3918f5e-491a-4456-9438-783cf36e291e","Type":"ContainerStarted","Data":"f7738fb6145523c591795780526fbe6c448b21cc036a385c9e2ee8c1c06ad6c4"}
Jul 31 23:47:31.967965 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:47:31.967877    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"d3918f5e-491a-4456-9438-783cf36e291e","Type":"ContainerStarted","Data":"cde19505eac5230dd61492a4b657f4e2a34885c37aa9e45e1fe576ea950db884"}
Jul 31 23:47:31.967965 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:47:31.967919    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"d3918f5e-491a-4456-9438-783cf36e291e","Type":"ContainerStarted","Data":"0f2ef69ee7d06e3ad3e03b0deda51ea3b8063bb3ff755efe9ce710d5e2f6a467"}
Jul 31 23:47:31.970695 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:47:31.970255    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-6dffbbd67-lzqqq" event={"ID":"8a2dfa37-6798-4b86-be3b-166e942af34d","Type":"ContainerStarted","Data":"a942e51d2782cf6466eb1b0e93877024c91802f0ae38f603154eaa9ad7c08f1c"}
Jul 31 23:47:31.970695 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:47:31.970655    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-6dffbbd67-lzqqq"
Jul 31 23:47:31.976843 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:47:31.976804    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-6dffbbd67-lzqqq"
Jul 31 23:47:31.992576 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:47:31.992526    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-6fb556c5d9-8dt4b" podStartSLOduration=2.081593166 podStartE2EDuration="5.992508941s" totalImagesPullingTime="3.910915775s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 23:47:26 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 23:47:31.992286631 +0000 UTC m=+75.960037390" watchObservedRunningTime="2026-07-31 23:47:31.992508941 +0000 UTC m=+75.960259700"
Jul 31 23:47:32.016724 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:47:32.016672    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-6dffbbd67-lzqqq" podStartSLOduration=1.753354657 podStartE2EDuration="4.016649765s" totalImagesPullingTime="2.263295108s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 23:47:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 23:47:32.015577791 +0000 UTC m=+75.983328550" watchObservedRunningTime="2026-07-31 23:47:32.016649765 +0000 UTC m=+75.984400528"
Jul 31 23:47:32.038775 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:47:32.038718    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-5b86cb7b86-269w6" podStartSLOduration=1.795747768 podStartE2EDuration="4.038692801s" totalImagesPullingTime="2.242945033s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 23:47:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 23:47:32.036620134 +0000 UTC m=+76.004370889" watchObservedRunningTime="2026-07-31 23:47:32.038692801 +0000 UTC m=+76.006443560"
Jul 31 23:47:32.419610 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:47:32.419579    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gttdm_b32e6d42-e4e0-4113-b8ce-f5d5c5dc13f7/konnectivity-agent/0.log" limit=1024
Jul 31 23:47:32.517002 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:47:32.516964    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gttdm_b32e6d42-e4e0-4113-b8ce-f5d5c5dc13f7/konnectivity-agent/0.log" limit=1024
Jul 31 23:47:32.651294 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:47:32.651249    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gttdm_b32e6d42-e4e0-4113-b8ce-f5d5c5dc13f7/konnectivity-agent/0.log" limit=1024
Jul 31 23:47:32.720887 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:47:32.720800    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gttdm_b32e6d42-e4e0-4113-b8ce-f5d5c5dc13f7/konnectivity-agent/0.log" limit=1024
Jul 31 23:47:32.977480 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:47:32.977394    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"d3918f5e-491a-4456-9438-783cf36e291e","Type":"ContainerStarted","Data":"b7c2c5e442ff1470c107e0f40de29d412726156bcf2ac63e9a58000aa0c3cad0"}
Jul 31 23:47:32.977892 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:47:32.977487    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"d3918f5e-491a-4456-9438-783cf36e291e","Type":"ContainerStarted","Data":"4d0924d706ed807692ce84f7c417ef68191e4f361047697e6450b1e9c6ca8904"}
Jul 31 23:47:33.004991 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:47:33.004937    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=6.147418532 podStartE2EDuration="9.004919012s" totalImagesPullingTime="2.842108956s" totalInitContainerRuntime="15.391524ms" isStatefulPod=false podCreationTimestamp="2026-07-31 23:47:24 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 23:47:33.003082141 +0000 UTC m=+76.970832912" watchObservedRunningTime="2026-07-31 23:47:33.004919012 +0000 UTC m=+76.972669770"
Jul 31 23:47:35.651490 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:47:35.651452    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gttdm_b32e6d42-e4e0-4113-b8ce-f5d5c5dc13f7/konnectivity-agent/0.log" limit=1024
Jul 31 23:47:35.739029 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:47:35.738987    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gttdm_b32e6d42-e4e0-4113-b8ce-f5d5c5dc13f7/konnectivity-agent/0.log" limit=1024
Jul 31 23:47:35.865366 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:47:35.865333    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gttdm_b32e6d42-e4e0-4113-b8ce-f5d5c5dc13f7/konnectivity-agent/0.log" limit=1024
Jul 31 23:47:35.964749 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:47:35.964655    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gttdm_b32e6d42-e4e0-4113-b8ce-f5d5c5dc13f7/konnectivity-agent/0.log" limit=1024
Jul 31 23:47:36.377128 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:47:36.377094    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gttdm_b32e6d42-e4e0-4113-b8ce-f5d5c5dc13f7/konnectivity-agent/0.log" limit=1024
Jul 31 23:47:36.478602 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:47:36.478567    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gttdm_b32e6d42-e4e0-4113-b8ce-f5d5c5dc13f7/konnectivity-agent/0.log" limit=1024
Jul 31 23:47:36.547791 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:47:36.547758    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gttdm_b32e6d42-e4e0-4113-b8ce-f5d5c5dc13f7/konnectivity-agent/0.log" limit=1024
Jul 31 23:47:37.985652 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:47:37.985625    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-6fb556c5d9-8dt4b"
Jul 31 23:47:39.232961 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:47:39.232920    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gttdm_b32e6d42-e4e0-4113-b8ce-f5d5c5dc13f7/konnectivity-agent/0.log" limit=1024
Jul 31 23:47:39.284111 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:47:39.284077    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-56655dc5bf-zbmx4"
Jul 31 23:47:39.284429 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:47:39.284407    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-56655dc5bf-zbmx4"
Jul 31 23:47:39.289240 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:47:39.289216    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-56655dc5bf-zbmx4"
Jul 31 23:47:39.310027 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:47:39.309999    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gttdm_b32e6d42-e4e0-4113-b8ce-f5d5c5dc13f7/konnectivity-agent/0.log" limit=1024
Jul 31 23:47:40.009822 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:47:40.009797    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-56655dc5bf-zbmx4"
Jul 31 23:47:40.165261 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:47:40.165226    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gttdm_b32e6d42-e4e0-4113-b8ce-f5d5c5dc13f7/konnectivity-agent/0.log" limit=1024
Jul 31 23:47:40.266523 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:47:40.266418    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gttdm_b32e6d42-e4e0-4113-b8ce-f5d5c5dc13f7/konnectivity-agent/0.log" limit=1024
Jul 31 23:47:40.326782 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:47:40.326749    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gttdm_b32e6d42-e4e0-4113-b8ce-f5d5c5dc13f7/konnectivity-agent/0.log" limit=1024
Jul 31 23:47:41.135828 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:47:41.135791    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gttdm_b32e6d42-e4e0-4113-b8ce-f5d5c5dc13f7/konnectivity-agent/0.log" limit=1024
Jul 31 23:47:41.235956 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:47:41.235916    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gttdm_b32e6d42-e4e0-4113-b8ce-f5d5c5dc13f7/konnectivity-agent/0.log" limit=1024
Jul 31 23:47:43.961121 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:47:43.961083    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gttdm_b32e6d42-e4e0-4113-b8ce-f5d5c5dc13f7/konnectivity-agent/0.log" limit=1024
Jul 31 23:47:44.061905 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:47:44.061869    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gttdm_b32e6d42-e4e0-4113-b8ce-f5d5c5dc13f7/konnectivity-agent/0.log" limit=1024
Jul 31 23:47:44.635443 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:47:44.635412    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gttdm_b32e6d42-e4e0-4113-b8ce-f5d5c5dc13f7/konnectivity-agent/0.log" limit=1024
Jul 31 23:47:44.740614 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:47:44.740578    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gttdm_b32e6d42-e4e0-4113-b8ce-f5d5c5dc13f7/konnectivity-agent/0.log" limit=1024
Jul 31 23:47:48.304557 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:47:48.304515    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gttdm_b32e6d42-e4e0-4113-b8ce-f5d5c5dc13f7/konnectivity-agent/0.log" limit=1024
Jul 31 23:47:48.404642 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:47:48.404603    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gttdm_b32e6d42-e4e0-4113-b8ce-f5d5c5dc13f7/konnectivity-agent/0.log" limit=1024
Jul 31 23:47:48.727468 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:47:48.727388    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/metrics-server-5b86cb7b86-269w6"
Jul 31 23:47:48.727468 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:47:48.727453    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/metrics-server-5b86cb7b86-269w6"
Jul 31 23:47:53.946294 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:47:53.946248    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-5d59f9cb75-g6g2x" podUID="47326e9a-7028-4a90-94e4-70d55ac3290d" containerName="console" containerID="cri-o://7fd964cc7523a1de09ed649542fc42bf5bad9fb2d287b303fd9efc45c3a260e0" gracePeriod=15
Jul 31 23:47:54.193426 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:47:54.193402    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5d59f9cb75-g6g2x"
Jul 31 23:47:54.331248 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:47:54.331211    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gpsh4\" (UniqueName: \"kubernetes.io/projected/47326e9a-7028-4a90-94e4-70d55ac3290d-kube-api-access-gpsh4\") pod \"47326e9a-7028-4a90-94e4-70d55ac3290d\" (UID: \"47326e9a-7028-4a90-94e4-70d55ac3290d\") "
Jul 31 23:47:54.331248 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:47:54.331257    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/47326e9a-7028-4a90-94e4-70d55ac3290d-console-oauth-config\") pod \"47326e9a-7028-4a90-94e4-70d55ac3290d\" (UID: \"47326e9a-7028-4a90-94e4-70d55ac3290d\") "
Jul 31 23:47:54.331507 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:47:54.331286    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/47326e9a-7028-4a90-94e4-70d55ac3290d-service-ca\") pod \"47326e9a-7028-4a90-94e4-70d55ac3290d\" (UID: \"47326e9a-7028-4a90-94e4-70d55ac3290d\") "
Jul 31 23:47:54.331507 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:47:54.331316    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/47326e9a-7028-4a90-94e4-70d55ac3290d-console-config\") pod \"47326e9a-7028-4a90-94e4-70d55ac3290d\" (UID: \"47326e9a-7028-4a90-94e4-70d55ac3290d\") "
Jul 31 23:47:54.331507 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:47:54.331334    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/47326e9a-7028-4a90-94e4-70d55ac3290d-console-serving-cert\") pod \"47326e9a-7028-4a90-94e4-70d55ac3290d\" (UID: \"47326e9a-7028-4a90-94e4-70d55ac3290d\") "
Jul 31 23:47:54.331507 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:47:54.331350    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/47326e9a-7028-4a90-94e4-70d55ac3290d-oauth-serving-cert\") pod \"47326e9a-7028-4a90-94e4-70d55ac3290d\" (UID: \"47326e9a-7028-4a90-94e4-70d55ac3290d\") "
Jul 31 23:47:54.331798 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:47:54.331772    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/47326e9a-7028-4a90-94e4-70d55ac3290d-service-ca" (OuterVolumeSpecName: "service-ca") pod "47326e9a-7028-4a90-94e4-70d55ac3290d" (UID: "47326e9a-7028-4a90-94e4-70d55ac3290d"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 31 23:47:54.331860 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:47:54.331791    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/47326e9a-7028-4a90-94e4-70d55ac3290d-console-config" (OuterVolumeSpecName: "console-config") pod "47326e9a-7028-4a90-94e4-70d55ac3290d" (UID: "47326e9a-7028-4a90-94e4-70d55ac3290d"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 31 23:47:54.331860 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:47:54.331834    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/47326e9a-7028-4a90-94e4-70d55ac3290d-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "47326e9a-7028-4a90-94e4-70d55ac3290d" (UID: "47326e9a-7028-4a90-94e4-70d55ac3290d"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 31 23:47:54.333831 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:47:54.333808    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/47326e9a-7028-4a90-94e4-70d55ac3290d-kube-api-access-gpsh4" (OuterVolumeSpecName: "kube-api-access-gpsh4") pod "47326e9a-7028-4a90-94e4-70d55ac3290d" (UID: "47326e9a-7028-4a90-94e4-70d55ac3290d"). InnerVolumeSpecName "kube-api-access-gpsh4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 31 23:47:54.333831 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:47:54.333817    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/47326e9a-7028-4a90-94e4-70d55ac3290d-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "47326e9a-7028-4a90-94e4-70d55ac3290d" (UID: "47326e9a-7028-4a90-94e4-70d55ac3290d"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 31 23:47:54.333952 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:47:54.333887    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/47326e9a-7028-4a90-94e4-70d55ac3290d-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "47326e9a-7028-4a90-94e4-70d55ac3290d" (UID: "47326e9a-7028-4a90-94e4-70d55ac3290d"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 31 23:47:54.432154 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:47:54.432111    2578 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/47326e9a-7028-4a90-94e4-70d55ac3290d-service-ca\") on node \"ip-10-0-136-108.ec2.internal\" DevicePath \"\""
Jul 31 23:47:54.432154 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:47:54.432145    2578 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/47326e9a-7028-4a90-94e4-70d55ac3290d-console-config\") on node \"ip-10-0-136-108.ec2.internal\" DevicePath \"\""
Jul 31 23:47:54.432154 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:47:54.432154    2578 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/47326e9a-7028-4a90-94e4-70d55ac3290d-console-serving-cert\") on node \"ip-10-0-136-108.ec2.internal\" DevicePath \"\""
Jul 31 23:47:54.432154 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:47:54.432164    2578 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/47326e9a-7028-4a90-94e4-70d55ac3290d-oauth-serving-cert\") on node \"ip-10-0-136-108.ec2.internal\" DevicePath \"\""
Jul 31 23:47:54.432442 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:47:54.432203    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gpsh4\" (UniqueName: \"kubernetes.io/projected/47326e9a-7028-4a90-94e4-70d55ac3290d-kube-api-access-gpsh4\") on node \"ip-10-0-136-108.ec2.internal\" DevicePath \"\""
Jul 31 23:47:54.432442 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:47:54.432213    2578 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/47326e9a-7028-4a90-94e4-70d55ac3290d-console-oauth-config\") on node \"ip-10-0-136-108.ec2.internal\" DevicePath \"\""
Jul 31 23:47:55.061608 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:47:55.061571    2578 generic.go:356] "Generic (PLEG): container finished" podID="47326e9a-7028-4a90-94e4-70d55ac3290d" containerID="7fd964cc7523a1de09ed649542fc42bf5bad9fb2d287b303fd9efc45c3a260e0" exitCode=0
Jul 31 23:47:55.062010 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:47:55.061614    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5d59f9cb75-g6g2x" event={"ID":"47326e9a-7028-4a90-94e4-70d55ac3290d","Type":"ContainerDied","Data":"7fd964cc7523a1de09ed649542fc42bf5bad9fb2d287b303fd9efc45c3a260e0"}
Jul 31 23:47:55.062010 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:47:55.061620    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5d59f9cb75-g6g2x"
Jul 31 23:47:55.062010 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:47:55.061660    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5d59f9cb75-g6g2x" event={"ID":"47326e9a-7028-4a90-94e4-70d55ac3290d","Type":"ContainerDied","Data":"b25c304579026de5ff906da271b4dbaec220ca22faaa9b83470a1235d811f11f"}
Jul 31 23:47:55.062010 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:47:55.061683    2578 scope.go:117] "RemoveContainer" containerID="7fd964cc7523a1de09ed649542fc42bf5bad9fb2d287b303fd9efc45c3a260e0"
Jul 31 23:47:55.073720 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:47:55.073699    2578 scope.go:117] "RemoveContainer" containerID="7fd964cc7523a1de09ed649542fc42bf5bad9fb2d287b303fd9efc45c3a260e0"
Jul 31 23:47:55.073986 ip-10-0-136-108 kubenswrapper[2578]: E0731 23:47:55.073962    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7fd964cc7523a1de09ed649542fc42bf5bad9fb2d287b303fd9efc45c3a260e0\": container with ID starting with 7fd964cc7523a1de09ed649542fc42bf5bad9fb2d287b303fd9efc45c3a260e0 not found: ID does not exist" containerID="7fd964cc7523a1de09ed649542fc42bf5bad9fb2d287b303fd9efc45c3a260e0"
Jul 31 23:47:55.074058 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:47:55.073999    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7fd964cc7523a1de09ed649542fc42bf5bad9fb2d287b303fd9efc45c3a260e0"} err="failed to get container status \"7fd964cc7523a1de09ed649542fc42bf5bad9fb2d287b303fd9efc45c3a260e0\": rpc error: code = NotFound desc = could not find container \"7fd964cc7523a1de09ed649542fc42bf5bad9fb2d287b303fd9efc45c3a260e0\": container with ID starting with 7fd964cc7523a1de09ed649542fc42bf5bad9fb2d287b303fd9efc45c3a260e0 not found: ID does not exist"
Jul 31 23:47:55.078378 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:47:55.078355    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5d59f9cb75-g6g2x"]
Jul 31 23:47:55.081717 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:47:55.081696    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-5d59f9cb75-g6g2x"]
Jul 31 23:47:55.426814 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:47:55.426704    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gttdm_b32e6d42-e4e0-4113-b8ce-f5d5c5dc13f7/konnectivity-agent/0.log" limit=1024
Jul 31 23:47:56.488579 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:47:56.488536    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="47326e9a-7028-4a90-94e4-70d55ac3290d" path="/var/lib/kubelet/pods/47326e9a-7028-4a90-94e4-70d55ac3290d/volumes"
Jul 31 23:47:58.913602 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:47:58.913569    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-8d9rr"
Jul 31 23:47:58.954957 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:47:58.954916    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gttdm_b32e6d42-e4e0-4113-b8ce-f5d5c5dc13f7/konnectivity-agent/0.log" limit=1024
Jul 31 23:47:59.017699 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:47:59.017657    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gttdm_b32e6d42-e4e0-4113-b8ce-f5d5c5dc13f7/konnectivity-agent/0.log" limit=1024
Jul 31 23:48:08.733655 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:48:08.733624    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/metrics-server-5b86cb7b86-269w6"
Jul 31 23:48:08.738139 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:48:08.738116    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-5b86cb7b86-269w6"
Jul 31 23:48:17.248395 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:48:17.248351    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gttdm_b32e6d42-e4e0-4113-b8ce-f5d5c5dc13f7/konnectivity-agent/0.log" limit=1024
Jul 31 23:48:17.300129 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:48:17.300093    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gttdm_b32e6d42-e4e0-4113-b8ce-f5d5c5dc13f7/konnectivity-agent/0.log" limit=1024
Jul 31 23:48:43.151115 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:48:43.151075    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gttdm_b32e6d42-e4e0-4113-b8ce-f5d5c5dc13f7/konnectivity-agent/0.log" limit=1024
Jul 31 23:48:43.365944 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:48:43.365911    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gttdm_b32e6d42-e4e0-4113-b8ce-f5d5c5dc13f7/konnectivity-agent/0.log" limit=1024
Jul 31 23:48:43.845323 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:48:43.845275    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gttdm_b32e6d42-e4e0-4113-b8ce-f5d5c5dc13f7/konnectivity-agent/0.log" limit=1024
Jul 31 23:48:48.241247 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:48:48.241212    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-697d9cdbc4-srjhx"]
Jul 31 23:48:48.242142 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:48:48.242118    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="47326e9a-7028-4a90-94e4-70d55ac3290d" containerName="console"
Jul 31 23:48:48.242306 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:48:48.242293    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="47326e9a-7028-4a90-94e4-70d55ac3290d" containerName="console"
Jul 31 23:48:48.242487 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:48:48.242468    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="47326e9a-7028-4a90-94e4-70d55ac3290d" containerName="console"
Jul 31 23:48:48.245728 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:48:48.245703    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-697d9cdbc4-srjhx"
Jul 31 23:48:48.247946 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:48:48.247916    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\""
Jul 31 23:48:48.248121 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:48:48.248012    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-kmzxz\""
Jul 31 23:48:48.248121 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:48:48.247924    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\""
Jul 31 23:48:48.248297 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:48:48.247924    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\""
Jul 31 23:48:48.248297 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:48:48.247925    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\""
Jul 31 23:48:48.248297 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:48:48.247925    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\""
Jul 31 23:48:48.253793 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:48:48.253762    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-8i12ta5c71j38\""
Jul 31 23:48:48.255959 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:48:48.255929    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-697d9cdbc4-srjhx"]
Jul 31 23:48:48.364656 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:48:48.364609    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/cbbb8e62-6d70-43f7-8851-f29b45d31862-telemeter-client-tls\") pod \"telemeter-client-697d9cdbc4-srjhx\" (UID: \"cbbb8e62-6d70-43f7-8851-f29b45d31862\") " pod="openshift-monitoring/telemeter-client-697d9cdbc4-srjhx"
Jul 31 23:48:48.364912 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:48:48.364890    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/cbbb8e62-6d70-43f7-8851-f29b45d31862-metrics-client-ca\") pod \"telemeter-client-697d9cdbc4-srjhx\" (UID: \"cbbb8e62-6d70-43f7-8851-f29b45d31862\") " pod="openshift-monitoring/telemeter-client-697d9cdbc4-srjhx"
Jul 31 23:48:48.365032 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:48:48.365018    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v4rkg\" (UniqueName: \"kubernetes.io/projected/cbbb8e62-6d70-43f7-8851-f29b45d31862-kube-api-access-v4rkg\") pod \"telemeter-client-697d9cdbc4-srjhx\" (UID: \"cbbb8e62-6d70-43f7-8851-f29b45d31862\") " pod="openshift-monitoring/telemeter-client-697d9cdbc4-srjhx"
Jul 31 23:48:48.365219 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:48:48.365193    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cbbb8e62-6d70-43f7-8851-f29b45d31862-telemeter-trusted-ca-bundle\") pod \"telemeter-client-697d9cdbc4-srjhx\" (UID: \"cbbb8e62-6d70-43f7-8851-f29b45d31862\") " pod="openshift-monitoring/telemeter-client-697d9cdbc4-srjhx"
Jul 31 23:48:48.365329 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:48:48.365235    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/cbbb8e62-6d70-43f7-8851-f29b45d31862-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-697d9cdbc4-srjhx\" (UID: \"cbbb8e62-6d70-43f7-8851-f29b45d31862\") " pod="openshift-monitoring/telemeter-client-697d9cdbc4-srjhx"
Jul 31 23:48:48.365329 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:48:48.365291    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cbbb8e62-6d70-43f7-8851-f29b45d31862-serving-certs-ca-bundle\") pod \"telemeter-client-697d9cdbc4-srjhx\" (UID: \"cbbb8e62-6d70-43f7-8851-f29b45d31862\") " pod="openshift-monitoring/telemeter-client-697d9cdbc4-srjhx"
Jul 31 23:48:48.365329 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:48:48.365321    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/cbbb8e62-6d70-43f7-8851-f29b45d31862-federate-client-tls\") pod \"telemeter-client-697d9cdbc4-srjhx\" (UID: \"cbbb8e62-6d70-43f7-8851-f29b45d31862\") " pod="openshift-monitoring/telemeter-client-697d9cdbc4-srjhx"
Jul 31 23:48:48.365447 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:48:48.365352    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/cbbb8e62-6d70-43f7-8851-f29b45d31862-secret-telemeter-client\") pod \"telemeter-client-697d9cdbc4-srjhx\" (UID: \"cbbb8e62-6d70-43f7-8851-f29b45d31862\") " pod="openshift-monitoring/telemeter-client-697d9cdbc4-srjhx"
Jul 31 23:48:48.466294 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:48:48.466253    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cbbb8e62-6d70-43f7-8851-f29b45d31862-serving-certs-ca-bundle\") pod \"telemeter-client-697d9cdbc4-srjhx\" (UID: \"cbbb8e62-6d70-43f7-8851-f29b45d31862\") " pod="openshift-monitoring/telemeter-client-697d9cdbc4-srjhx"
Jul 31 23:48:48.466600 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:48:48.466306    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/cbbb8e62-6d70-43f7-8851-f29b45d31862-federate-client-tls\") pod \"telemeter-client-697d9cdbc4-srjhx\" (UID: \"cbbb8e62-6d70-43f7-8851-f29b45d31862\") " pod="openshift-monitoring/telemeter-client-697d9cdbc4-srjhx"
Jul 31 23:48:48.466600 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:48:48.466339    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/cbbb8e62-6d70-43f7-8851-f29b45d31862-secret-telemeter-client\") pod \"telemeter-client-697d9cdbc4-srjhx\" (UID: \"cbbb8e62-6d70-43f7-8851-f29b45d31862\") " pod="openshift-monitoring/telemeter-client-697d9cdbc4-srjhx"
Jul 31 23:48:48.466600 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:48:48.466392    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/cbbb8e62-6d70-43f7-8851-f29b45d31862-telemeter-client-tls\") pod \"telemeter-client-697d9cdbc4-srjhx\" (UID: \"cbbb8e62-6d70-43f7-8851-f29b45d31862\") " pod="openshift-monitoring/telemeter-client-697d9cdbc4-srjhx"
Jul 31 23:48:48.466600 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:48:48.466413    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/cbbb8e62-6d70-43f7-8851-f29b45d31862-metrics-client-ca\") pod \"telemeter-client-697d9cdbc4-srjhx\" (UID: \"cbbb8e62-6d70-43f7-8851-f29b45d31862\") " pod="openshift-monitoring/telemeter-client-697d9cdbc4-srjhx"
Jul 31 23:48:48.466600 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:48:48.466435    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v4rkg\" (UniqueName: \"kubernetes.io/projected/cbbb8e62-6d70-43f7-8851-f29b45d31862-kube-api-access-v4rkg\") pod \"telemeter-client-697d9cdbc4-srjhx\" (UID: \"cbbb8e62-6d70-43f7-8851-f29b45d31862\") " pod="openshift-monitoring/telemeter-client-697d9cdbc4-srjhx"
Jul 31 23:48:48.466600 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:48:48.466474    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cbbb8e62-6d70-43f7-8851-f29b45d31862-telemeter-trusted-ca-bundle\") pod \"telemeter-client-697d9cdbc4-srjhx\" (UID: \"cbbb8e62-6d70-43f7-8851-f29b45d31862\") " pod="openshift-monitoring/telemeter-client-697d9cdbc4-srjhx"
Jul 31 23:48:48.466600 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:48:48.466536    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/cbbb8e62-6d70-43f7-8851-f29b45d31862-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-697d9cdbc4-srjhx\" (UID: \"cbbb8e62-6d70-43f7-8851-f29b45d31862\") " pod="openshift-monitoring/telemeter-client-697d9cdbc4-srjhx"
Jul 31 23:48:48.467222 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:48:48.467165    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cbbb8e62-6d70-43f7-8851-f29b45d31862-serving-certs-ca-bundle\") pod \"telemeter-client-697d9cdbc4-srjhx\" (UID: \"cbbb8e62-6d70-43f7-8851-f29b45d31862\") " pod="openshift-monitoring/telemeter-client-697d9cdbc4-srjhx"
Jul 31 23:48:48.467458 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:48:48.467438    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cbbb8e62-6d70-43f7-8851-f29b45d31862-telemeter-trusted-ca-bundle\") pod \"telemeter-client-697d9cdbc4-srjhx\" (UID: \"cbbb8e62-6d70-43f7-8851-f29b45d31862\") " pod="openshift-monitoring/telemeter-client-697d9cdbc4-srjhx"
Jul 31 23:48:48.467519 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:48:48.467451    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/cbbb8e62-6d70-43f7-8851-f29b45d31862-metrics-client-ca\") pod \"telemeter-client-697d9cdbc4-srjhx\" (UID: \"cbbb8e62-6d70-43f7-8851-f29b45d31862\") " pod="openshift-monitoring/telemeter-client-697d9cdbc4-srjhx"
Jul 31 23:48:48.468946 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:48:48.468915    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/cbbb8e62-6d70-43f7-8851-f29b45d31862-telemeter-client-tls\") pod \"telemeter-client-697d9cdbc4-srjhx\" (UID: \"cbbb8e62-6d70-43f7-8851-f29b45d31862\") " pod="openshift-monitoring/telemeter-client-697d9cdbc4-srjhx"
Jul 31 23:48:48.469030 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:48:48.468976    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/cbbb8e62-6d70-43f7-8851-f29b45d31862-secret-telemeter-client\") pod \"telemeter-client-697d9cdbc4-srjhx\" (UID: \"cbbb8e62-6d70-43f7-8851-f29b45d31862\") " pod="openshift-monitoring/telemeter-client-697d9cdbc4-srjhx"
Jul 31 23:48:48.469253 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:48:48.469235    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/cbbb8e62-6d70-43f7-8851-f29b45d31862-federate-client-tls\") pod \"telemeter-client-697d9cdbc4-srjhx\" (UID: \"cbbb8e62-6d70-43f7-8851-f29b45d31862\") " pod="openshift-monitoring/telemeter-client-697d9cdbc4-srjhx"
Jul 31 23:48:48.469349 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:48:48.469326    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/cbbb8e62-6d70-43f7-8851-f29b45d31862-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-697d9cdbc4-srjhx\" (UID: \"cbbb8e62-6d70-43f7-8851-f29b45d31862\") " pod="openshift-monitoring/telemeter-client-697d9cdbc4-srjhx"
Jul 31 23:48:48.473661 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:48:48.473637    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v4rkg\" (UniqueName: \"kubernetes.io/projected/cbbb8e62-6d70-43f7-8851-f29b45d31862-kube-api-access-v4rkg\") pod \"telemeter-client-697d9cdbc4-srjhx\" (UID: \"cbbb8e62-6d70-43f7-8851-f29b45d31862\") " pod="openshift-monitoring/telemeter-client-697d9cdbc4-srjhx"
Jul 31 23:48:48.558520 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:48:48.558431    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-697d9cdbc4-srjhx"
Jul 31 23:48:48.693619 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:48:48.693584    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-697d9cdbc4-srjhx"]
Jul 31 23:48:48.696499 ip-10-0-136-108 kubenswrapper[2578]: W0731 23:48:48.696468    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcbbb8e62_6d70_43f7_8851_f29b45d31862.slice/crio-0f3acf97463d74e5e9979f83410de403484d6f9f29f980b510cff721a3645a59 WatchSource:0}: Error finding container 0f3acf97463d74e5e9979f83410de403484d6f9f29f980b510cff721a3645a59: Status 404 returned error can't find the container with id 0f3acf97463d74e5e9979f83410de403484d6f9f29f980b510cff721a3645a59
Jul 31 23:48:49.257831 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:48:49.257795    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-697d9cdbc4-srjhx" event={"ID":"cbbb8e62-6d70-43f7-8851-f29b45d31862","Type":"ContainerStarted","Data":"0f3acf97463d74e5e9979f83410de403484d6f9f29f980b510cff721a3645a59"}
Jul 31 23:48:51.268952 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:48:51.268918    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-697d9cdbc4-srjhx" event={"ID":"cbbb8e62-6d70-43f7-8851-f29b45d31862","Type":"ContainerStarted","Data":"c3a48f16ba2cbf70fc7e0c4f60ee8fea78b2890caf381008cd072d887e43a216"}
Jul 31 23:48:51.269366 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:48:51.268975    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-697d9cdbc4-srjhx" event={"ID":"cbbb8e62-6d70-43f7-8851-f29b45d31862","Type":"ContainerStarted","Data":"8d310515f2f25651c1f512f79d5f2cb5b59ed4edc9b397acb80a2b429b9bc876"}
Jul 31 23:48:51.269366 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:48:51.268995    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-697d9cdbc4-srjhx" event={"ID":"cbbb8e62-6d70-43f7-8851-f29b45d31862","Type":"ContainerStarted","Data":"1cf44103f6842c909351e44baafba44dae5395eacce5a96e12143214edf1b234"}
Jul 31 23:48:51.288331 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:48:51.288284    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-697d9cdbc4-srjhx" podStartSLOduration=1.623354969 podStartE2EDuration="3.288267976s" totalImagesPullingTime="1.664913007s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 23:48:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 23:48:51.288206282 +0000 UTC m=+155.255957049" watchObservedRunningTime="2026-07-31 23:48:51.288267976 +0000 UTC m=+155.256018733"
Jul 31 23:48:51.372859 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:48:51.372824    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-5cd747c9c6-gsrrn"]
Jul 31 23:48:51.376090 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:48:51.376065    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5cd747c9c6-gsrrn"
Jul 31 23:48:51.402962 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:48:51.402930    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5cd747c9c6-gsrrn"]
Jul 31 23:48:51.493274 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:48:51.493238    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gttdm_b32e6d42-e4e0-4113-b8ce-f5d5c5dc13f7/konnectivity-agent/0.log" limit=1024
Jul 31 23:48:51.494634 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:48:51.494614    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/9011d6f9-f60e-456a-8da5-c4b5d3225638-service-ca\") pod \"console-5cd747c9c6-gsrrn\" (UID: \"9011d6f9-f60e-456a-8da5-c4b5d3225638\") " pod="openshift-console/console-5cd747c9c6-gsrrn"
Jul 31 23:48:51.494731 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:48:51.494641    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mtfmp\" (UniqueName: \"kubernetes.io/projected/9011d6f9-f60e-456a-8da5-c4b5d3225638-kube-api-access-mtfmp\") pod \"console-5cd747c9c6-gsrrn\" (UID: \"9011d6f9-f60e-456a-8da5-c4b5d3225638\") " pod="openshift-console/console-5cd747c9c6-gsrrn"
Jul 31 23:48:51.494731 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:48:51.494666    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9011d6f9-f60e-456a-8da5-c4b5d3225638-trusted-ca-bundle\") pod \"console-5cd747c9c6-gsrrn\" (UID: \"9011d6f9-f60e-456a-8da5-c4b5d3225638\") " pod="openshift-console/console-5cd747c9c6-gsrrn"
Jul 31 23:48:51.494820 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:48:51.494775    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/9011d6f9-f60e-456a-8da5-c4b5d3225638-console-config\") pod \"console-5cd747c9c6-gsrrn\" (UID: \"9011d6f9-f60e-456a-8da5-c4b5d3225638\") " pod="openshift-console/console-5cd747c9c6-gsrrn"
Jul 31 23:48:51.494856 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:48:51.494819    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/9011d6f9-f60e-456a-8da5-c4b5d3225638-console-oauth-config\") pod \"console-5cd747c9c6-gsrrn\" (UID: \"9011d6f9-f60e-456a-8da5-c4b5d3225638\") " pod="openshift-console/console-5cd747c9c6-gsrrn"
Jul 31 23:48:51.494895 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:48:51.494858    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/9011d6f9-f60e-456a-8da5-c4b5d3225638-oauth-serving-cert\") pod \"console-5cd747c9c6-gsrrn\" (UID: \"9011d6f9-f60e-456a-8da5-c4b5d3225638\") " pod="openshift-console/console-5cd747c9c6-gsrrn"
Jul 31 23:48:51.494895 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:48:51.494882    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/9011d6f9-f60e-456a-8da5-c4b5d3225638-console-serving-cert\") pod \"console-5cd747c9c6-gsrrn\" (UID: \"9011d6f9-f60e-456a-8da5-c4b5d3225638\") " pod="openshift-console/console-5cd747c9c6-gsrrn"
Jul 31 23:48:51.545807 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:48:51.545725    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gttdm_b32e6d42-e4e0-4113-b8ce-f5d5c5dc13f7/konnectivity-agent/0.log" limit=1024
Jul 31 23:48:51.595879 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:48:51.595826    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/9011d6f9-f60e-456a-8da5-c4b5d3225638-service-ca\") pod \"console-5cd747c9c6-gsrrn\" (UID: \"9011d6f9-f60e-456a-8da5-c4b5d3225638\") " pod="openshift-console/console-5cd747c9c6-gsrrn"
Jul 31 23:48:51.595879 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:48:51.595882    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mtfmp\" (UniqueName: \"kubernetes.io/projected/9011d6f9-f60e-456a-8da5-c4b5d3225638-kube-api-access-mtfmp\") pod \"console-5cd747c9c6-gsrrn\" (UID: \"9011d6f9-f60e-456a-8da5-c4b5d3225638\") " pod="openshift-console/console-5cd747c9c6-gsrrn"
Jul 31 23:48:51.596139 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:48:51.595918    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9011d6f9-f60e-456a-8da5-c4b5d3225638-trusted-ca-bundle\") pod \"console-5cd747c9c6-gsrrn\" (UID: \"9011d6f9-f60e-456a-8da5-c4b5d3225638\") " pod="openshift-console/console-5cd747c9c6-gsrrn"
Jul 31 23:48:51.596139 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:48:51.595969    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/9011d6f9-f60e-456a-8da5-c4b5d3225638-console-config\") pod \"console-5cd747c9c6-gsrrn\" (UID: \"9011d6f9-f60e-456a-8da5-c4b5d3225638\") " pod="openshift-console/console-5cd747c9c6-gsrrn"
Jul 31 23:48:51.596139 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:48:51.595986    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/9011d6f9-f60e-456a-8da5-c4b5d3225638-console-oauth-config\") pod \"console-5cd747c9c6-gsrrn\" (UID: \"9011d6f9-f60e-456a-8da5-c4b5d3225638\") " pod="openshift-console/console-5cd747c9c6-gsrrn"
Jul 31 23:48:51.596356 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:48:51.596139    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/9011d6f9-f60e-456a-8da5-c4b5d3225638-oauth-serving-cert\") pod \"console-5cd747c9c6-gsrrn\" (UID: \"9011d6f9-f60e-456a-8da5-c4b5d3225638\") " pod="openshift-console/console-5cd747c9c6-gsrrn"
Jul 31 23:48:51.596356 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:48:51.596278    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/9011d6f9-f60e-456a-8da5-c4b5d3225638-console-serving-cert\") pod \"console-5cd747c9c6-gsrrn\" (UID: \"9011d6f9-f60e-456a-8da5-c4b5d3225638\") " pod="openshift-console/console-5cd747c9c6-gsrrn"
Jul 31 23:48:51.596853 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:48:51.596808    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/9011d6f9-f60e-456a-8da5-c4b5d3225638-console-config\") pod \"console-5cd747c9c6-gsrrn\" (UID: \"9011d6f9-f60e-456a-8da5-c4b5d3225638\") " pod="openshift-console/console-5cd747c9c6-gsrrn"
Jul 31 23:48:51.596970 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:48:51.596879    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/9011d6f9-f60e-456a-8da5-c4b5d3225638-service-ca\") pod \"console-5cd747c9c6-gsrrn\" (UID: \"9011d6f9-f60e-456a-8da5-c4b5d3225638\") " pod="openshift-console/console-5cd747c9c6-gsrrn"
Jul 31 23:48:51.596970 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:48:51.596888    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/9011d6f9-f60e-456a-8da5-c4b5d3225638-oauth-serving-cert\") pod \"console-5cd747c9c6-gsrrn\" (UID: \"9011d6f9-f60e-456a-8da5-c4b5d3225638\") " pod="openshift-console/console-5cd747c9c6-gsrrn"
Jul 31 23:48:51.597127 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:48:51.597108    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9011d6f9-f60e-456a-8da5-c4b5d3225638-trusted-ca-bundle\") pod \"console-5cd747c9c6-gsrrn\" (UID: \"9011d6f9-f60e-456a-8da5-c4b5d3225638\") " pod="openshift-console/console-5cd747c9c6-gsrrn"
Jul 31 23:48:51.598545 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:48:51.598515    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/9011d6f9-f60e-456a-8da5-c4b5d3225638-console-oauth-config\") pod \"console-5cd747c9c6-gsrrn\" (UID: \"9011d6f9-f60e-456a-8da5-c4b5d3225638\") " pod="openshift-console/console-5cd747c9c6-gsrrn"
Jul 31 23:48:51.598813 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:48:51.598794    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/9011d6f9-f60e-456a-8da5-c4b5d3225638-console-serving-cert\") pod \"console-5cd747c9c6-gsrrn\" (UID: \"9011d6f9-f60e-456a-8da5-c4b5d3225638\") " pod="openshift-console/console-5cd747c9c6-gsrrn"
Jul 31 23:48:51.604644 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:48:51.604620    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mtfmp\" (UniqueName: \"kubernetes.io/projected/9011d6f9-f60e-456a-8da5-c4b5d3225638-kube-api-access-mtfmp\") pod \"console-5cd747c9c6-gsrrn\" (UID: \"9011d6f9-f60e-456a-8da5-c4b5d3225638\") " pod="openshift-console/console-5cd747c9c6-gsrrn"
Jul 31 23:48:51.684992 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:48:51.684956    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5cd747c9c6-gsrrn"
Jul 31 23:48:51.814495 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:48:51.814423    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5cd747c9c6-gsrrn"]
Jul 31 23:48:51.817539 ip-10-0-136-108 kubenswrapper[2578]: W0731 23:48:51.817510    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9011d6f9_f60e_456a_8da5_c4b5d3225638.slice/crio-4faeb2d5f17807c79bbfd5ae420aa564e099e10d98c60fbfe41a723fac96fdce WatchSource:0}: Error finding container 4faeb2d5f17807c79bbfd5ae420aa564e099e10d98c60fbfe41a723fac96fdce: Status 404 returned error can't find the container with id 4faeb2d5f17807c79bbfd5ae420aa564e099e10d98c60fbfe41a723fac96fdce
Jul 31 23:48:52.281035 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:48:52.280993    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5cd747c9c6-gsrrn" event={"ID":"9011d6f9-f60e-456a-8da5-c4b5d3225638","Type":"ContainerStarted","Data":"b07e333522c58c610656f5a271aae252c5ce01290f082579d2c0d02e682d4a1c"}
Jul 31 23:48:52.281562 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:48:52.281093    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5cd747c9c6-gsrrn" event={"ID":"9011d6f9-f60e-456a-8da5-c4b5d3225638","Type":"ContainerStarted","Data":"4faeb2d5f17807c79bbfd5ae420aa564e099e10d98c60fbfe41a723fac96fdce"}
Jul 31 23:48:52.297600 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:48:52.297535    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-5cd747c9c6-gsrrn" podStartSLOduration=1.297515035 podStartE2EDuration="1.297515035s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 23:48:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 23:48:52.296309899 +0000 UTC m=+156.264060659" watchObservedRunningTime="2026-07-31 23:48:52.297515035 +0000 UTC m=+156.265265794"
Jul 31 23:48:58.379958 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:48:58.379925    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gttdm_b32e6d42-e4e0-4113-b8ce-f5d5c5dc13f7/konnectivity-agent/0.log" limit=1024
Jul 31 23:48:58.432611 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:48:58.432576    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gttdm_b32e6d42-e4e0-4113-b8ce-f5d5c5dc13f7/konnectivity-agent/0.log" limit=1024
Jul 31 23:49:01.685799 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:49:01.685757    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-5cd747c9c6-gsrrn"
Jul 31 23:49:01.686217 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:49:01.685818    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-5cd747c9c6-gsrrn"
Jul 31 23:49:01.690798 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:49:01.690777    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-5cd747c9c6-gsrrn"
Jul 31 23:49:02.321487 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:49:02.321457    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-5cd747c9c6-gsrrn"
Jul 31 23:49:02.369563 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:49:02.369523    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-56655dc5bf-zbmx4"]
Jul 31 23:49:02.483163 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:49:02.483133    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-5fc6b5dfb9-j4bdc"]
Jul 31 23:49:02.486641 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:49:02.486611    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5fc6b5dfb9-j4bdc"
Jul 31 23:49:02.495112 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:49:02.494836    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gttdm_b32e6d42-e4e0-4113-b8ce-f5d5c5dc13f7/konnectivity-agent/0.log" limit=1024
Jul 31 23:49:02.499945 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:49:02.499922    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5fc6b5dfb9-j4bdc"]
Jul 31 23:49:02.563743 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:49:02.563706    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gttdm_b32e6d42-e4e0-4113-b8ce-f5d5c5dc13f7/konnectivity-agent/0.log" limit=1024
Jul 31 23:49:02.587231 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:49:02.587144    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/14a1f47f-df70-45e8-b5a5-5903ffb385e4-console-oauth-config\") pod \"console-5fc6b5dfb9-j4bdc\" (UID: \"14a1f47f-df70-45e8-b5a5-5903ffb385e4\") " pod="openshift-console/console-5fc6b5dfb9-j4bdc"
Jul 31 23:49:02.587231 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:49:02.587214    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/14a1f47f-df70-45e8-b5a5-5903ffb385e4-oauth-serving-cert\") pod \"console-5fc6b5dfb9-j4bdc\" (UID: \"14a1f47f-df70-45e8-b5a5-5903ffb385e4\") " pod="openshift-console/console-5fc6b5dfb9-j4bdc"
Jul 31 23:49:02.587400 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:49:02.587253    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/14a1f47f-df70-45e8-b5a5-5903ffb385e4-console-serving-cert\") pod \"console-5fc6b5dfb9-j4bdc\" (UID: \"14a1f47f-df70-45e8-b5a5-5903ffb385e4\") " pod="openshift-console/console-5fc6b5dfb9-j4bdc"
Jul 31 23:49:02.587400 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:49:02.587270    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/14a1f47f-df70-45e8-b5a5-5903ffb385e4-trusted-ca-bundle\") pod \"console-5fc6b5dfb9-j4bdc\" (UID: \"14a1f47f-df70-45e8-b5a5-5903ffb385e4\") " pod="openshift-console/console-5fc6b5dfb9-j4bdc"
Jul 31 23:49:02.587400 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:49:02.587344    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/14a1f47f-df70-45e8-b5a5-5903ffb385e4-console-config\") pod \"console-5fc6b5dfb9-j4bdc\" (UID: \"14a1f47f-df70-45e8-b5a5-5903ffb385e4\") " pod="openshift-console/console-5fc6b5dfb9-j4bdc"
Jul 31 23:49:02.587400 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:49:02.587379    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/14a1f47f-df70-45e8-b5a5-5903ffb385e4-service-ca\") pod \"console-5fc6b5dfb9-j4bdc\" (UID: \"14a1f47f-df70-45e8-b5a5-5903ffb385e4\") " pod="openshift-console/console-5fc6b5dfb9-j4bdc"
Jul 31 23:49:02.587519 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:49:02.587413    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dff27\" (UniqueName: \"kubernetes.io/projected/14a1f47f-df70-45e8-b5a5-5903ffb385e4-kube-api-access-dff27\") pod \"console-5fc6b5dfb9-j4bdc\" (UID: \"14a1f47f-df70-45e8-b5a5-5903ffb385e4\") " pod="openshift-console/console-5fc6b5dfb9-j4bdc"
Jul 31 23:49:02.688004 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:49:02.687966    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/14a1f47f-df70-45e8-b5a5-5903ffb385e4-console-config\") pod \"console-5fc6b5dfb9-j4bdc\" (UID: \"14a1f47f-df70-45e8-b5a5-5903ffb385e4\") " pod="openshift-console/console-5fc6b5dfb9-j4bdc"
Jul 31 23:49:02.688478 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:49:02.688028    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/14a1f47f-df70-45e8-b5a5-5903ffb385e4-service-ca\") pod \"console-5fc6b5dfb9-j4bdc\" (UID: \"14a1f47f-df70-45e8-b5a5-5903ffb385e4\") " pod="openshift-console/console-5fc6b5dfb9-j4bdc"
Jul 31 23:49:02.688478 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:49:02.688071    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dff27\" (UniqueName: \"kubernetes.io/projected/14a1f47f-df70-45e8-b5a5-5903ffb385e4-kube-api-access-dff27\") pod \"console-5fc6b5dfb9-j4bdc\" (UID: \"14a1f47f-df70-45e8-b5a5-5903ffb385e4\") " pod="openshift-console/console-5fc6b5dfb9-j4bdc"
Jul 31 23:49:02.688478 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:49:02.688105    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/14a1f47f-df70-45e8-b5a5-5903ffb385e4-console-oauth-config\") pod \"console-5fc6b5dfb9-j4bdc\" (UID: \"14a1f47f-df70-45e8-b5a5-5903ffb385e4\") " pod="openshift-console/console-5fc6b5dfb9-j4bdc"
Jul 31 23:49:02.688478 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:49:02.688128    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/14a1f47f-df70-45e8-b5a5-5903ffb385e4-oauth-serving-cert\") pod \"console-5fc6b5dfb9-j4bdc\" (UID: \"14a1f47f-df70-45e8-b5a5-5903ffb385e4\") " pod="openshift-console/console-5fc6b5dfb9-j4bdc"
Jul 31 23:49:02.688478 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:49:02.688340    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/14a1f47f-df70-45e8-b5a5-5903ffb385e4-console-serving-cert\") pod \"console-5fc6b5dfb9-j4bdc\" (UID: \"14a1f47f-df70-45e8-b5a5-5903ffb385e4\") " pod="openshift-console/console-5fc6b5dfb9-j4bdc"
Jul 31 23:49:02.688478 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:49:02.688389    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/14a1f47f-df70-45e8-b5a5-5903ffb385e4-trusted-ca-bundle\") pod \"console-5fc6b5dfb9-j4bdc\" (UID: \"14a1f47f-df70-45e8-b5a5-5903ffb385e4\") " pod="openshift-console/console-5fc6b5dfb9-j4bdc"
Jul 31 23:49:02.688827 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:49:02.688784    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/14a1f47f-df70-45e8-b5a5-5903ffb385e4-console-config\") pod \"console-5fc6b5dfb9-j4bdc\" (UID: \"14a1f47f-df70-45e8-b5a5-5903ffb385e4\") " pod="openshift-console/console-5fc6b5dfb9-j4bdc"
Jul 31 23:49:02.688895 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:49:02.688876    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/14a1f47f-df70-45e8-b5a5-5903ffb385e4-service-ca\") pod \"console-5fc6b5dfb9-j4bdc\" (UID: \"14a1f47f-df70-45e8-b5a5-5903ffb385e4\") " pod="openshift-console/console-5fc6b5dfb9-j4bdc"
Jul 31 23:49:02.688952 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:49:02.688936    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/14a1f47f-df70-45e8-b5a5-5903ffb385e4-oauth-serving-cert\") pod \"console-5fc6b5dfb9-j4bdc\" (UID: \"14a1f47f-df70-45e8-b5a5-5903ffb385e4\") " pod="openshift-console/console-5fc6b5dfb9-j4bdc"
Jul 31 23:49:02.689299 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:49:02.689277    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/14a1f47f-df70-45e8-b5a5-5903ffb385e4-trusted-ca-bundle\") pod \"console-5fc6b5dfb9-j4bdc\" (UID: \"14a1f47f-df70-45e8-b5a5-5903ffb385e4\") " pod="openshift-console/console-5fc6b5dfb9-j4bdc"
Jul 31 23:49:02.690617 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:49:02.690589    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/14a1f47f-df70-45e8-b5a5-5903ffb385e4-console-oauth-config\") pod \"console-5fc6b5dfb9-j4bdc\" (UID: \"14a1f47f-df70-45e8-b5a5-5903ffb385e4\") " pod="openshift-console/console-5fc6b5dfb9-j4bdc"
Jul 31 23:49:02.690924 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:49:02.690901    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/14a1f47f-df70-45e8-b5a5-5903ffb385e4-console-serving-cert\") pod \"console-5fc6b5dfb9-j4bdc\" (UID: \"14a1f47f-df70-45e8-b5a5-5903ffb385e4\") " pod="openshift-console/console-5fc6b5dfb9-j4bdc"
Jul 31 23:49:02.695553 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:49:02.695529    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dff27\" (UniqueName: \"kubernetes.io/projected/14a1f47f-df70-45e8-b5a5-5903ffb385e4-kube-api-access-dff27\") pod \"console-5fc6b5dfb9-j4bdc\" (UID: \"14a1f47f-df70-45e8-b5a5-5903ffb385e4\") " pod="openshift-console/console-5fc6b5dfb9-j4bdc"
Jul 31 23:49:02.803488 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:49:02.803453    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5fc6b5dfb9-j4bdc"
Jul 31 23:49:02.832522 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:49:02.832467    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gttdm_b32e6d42-e4e0-4113-b8ce-f5d5c5dc13f7/konnectivity-agent/0.log" limit=1024
Jul 31 23:49:02.890283 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:49:02.890235    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gttdm_b32e6d42-e4e0-4113-b8ce-f5d5c5dc13f7/konnectivity-agent/0.log" limit=1024
Jul 31 23:49:02.932466 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:49:02.932419    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5fc6b5dfb9-j4bdc"]
Jul 31 23:49:02.935101 ip-10-0-136-108 kubenswrapper[2578]: W0731 23:49:02.935074    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod14a1f47f_df70_45e8_b5a5_5903ffb385e4.slice/crio-4d0d37532722ae61e769afe34deeda5f4617f7e2dd14535e17a10445933fccb0 WatchSource:0}: Error finding container 4d0d37532722ae61e769afe34deeda5f4617f7e2dd14535e17a10445933fccb0: Status 404 returned error can't find the container with id 4d0d37532722ae61e769afe34deeda5f4617f7e2dd14535e17a10445933fccb0
Jul 31 23:49:03.325289 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:49:03.325248    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5fc6b5dfb9-j4bdc" event={"ID":"14a1f47f-df70-45e8-b5a5-5903ffb385e4","Type":"ContainerStarted","Data":"907d1aa5a01779884816fd388cf3a724001640a5b2e28986f1f04fbab741f3f6"}
Jul 31 23:49:03.325436 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:49:03.325311    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5fc6b5dfb9-j4bdc" event={"ID":"14a1f47f-df70-45e8-b5a5-5903ffb385e4","Type":"ContainerStarted","Data":"4d0d37532722ae61e769afe34deeda5f4617f7e2dd14535e17a10445933fccb0"}
Jul 31 23:49:03.342954 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:49:03.342902    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-5fc6b5dfb9-j4bdc" podStartSLOduration=1.342884182 podStartE2EDuration="1.342884182s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 23:49:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 23:49:03.341637734 +0000 UTC m=+167.309388491" watchObservedRunningTime="2026-07-31 23:49:03.342884182 +0000 UTC m=+167.310634940"
Jul 31 23:49:10.349885 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:49:10.349797    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gttdm_b32e6d42-e4e0-4113-b8ce-f5d5c5dc13f7/konnectivity-agent/0.log" limit=1024
Jul 31 23:49:10.404072 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:49:10.404036    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gttdm_b32e6d42-e4e0-4113-b8ce-f5d5c5dc13f7/konnectivity-agent/0.log" limit=1024
Jul 31 23:49:12.804073 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:49:12.804032    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-5fc6b5dfb9-j4bdc"
Jul 31 23:49:12.804537 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:49:12.804136    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-5fc6b5dfb9-j4bdc"
Jul 31 23:49:12.808890 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:49:12.808865    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-5fc6b5dfb9-j4bdc"
Jul 31 23:49:13.366701 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:49:13.366672    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-5fc6b5dfb9-j4bdc"
Jul 31 23:49:13.404620 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:49:13.404580    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5cd747c9c6-gsrrn"]
Jul 31 23:49:13.525273 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:49:13.525242    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gttdm_b32e6d42-e4e0-4113-b8ce-f5d5c5dc13f7/konnectivity-agent/0.log" limit=1024
Jul 31 23:49:13.578527 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:49:13.578496    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gttdm_b32e6d42-e4e0-4113-b8ce-f5d5c5dc13f7/konnectivity-agent/0.log" limit=1024
Jul 31 23:49:13.631550 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:49:13.631463    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gttdm_b32e6d42-e4e0-4113-b8ce-f5d5c5dc13f7/konnectivity-agent/0.log" limit=1024
Jul 31 23:49:13.953063 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:49:13.952966    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gttdm_b32e6d42-e4e0-4113-b8ce-f5d5c5dc13f7/konnectivity-agent/0.log" limit=1024
Jul 31 23:49:14.055328 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:49:14.055294    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gttdm_b32e6d42-e4e0-4113-b8ce-f5d5c5dc13f7/konnectivity-agent/0.log" limit=1024
Jul 31 23:49:27.389253 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:49:27.389188    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-56655dc5bf-zbmx4" podUID="a6aa6fe2-3453-4af0-a7e4-fe4c6930abb9" containerName="console" containerID="cri-o://a633810c87c86cdc80207564d0ee5aa6a8d25c86cad50aa1c5d55b327417d7d3" gracePeriod=15
Jul 31 23:49:27.480507 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:49:27.480462    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gttdm_b32e6d42-e4e0-4113-b8ce-f5d5c5dc13f7/konnectivity-agent/0.log" limit=1024
Jul 31 23:49:27.541860 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:49:27.541828    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gttdm_b32e6d42-e4e0-4113-b8ce-f5d5c5dc13f7/konnectivity-agent/0.log" limit=1024
Jul 31 23:49:27.628318 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:49:27.628293    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-56655dc5bf-zbmx4"
Jul 31 23:49:27.711210 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:49:27.711083    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-r6n29\" (UniqueName: \"kubernetes.io/projected/a6aa6fe2-3453-4af0-a7e4-fe4c6930abb9-kube-api-access-r6n29\") pod \"a6aa6fe2-3453-4af0-a7e4-fe4c6930abb9\" (UID: \"a6aa6fe2-3453-4af0-a7e4-fe4c6930abb9\") "
Jul 31 23:49:27.711210 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:49:27.711126    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a6aa6fe2-3453-4af0-a7e4-fe4c6930abb9-oauth-serving-cert\") pod \"a6aa6fe2-3453-4af0-a7e4-fe4c6930abb9\" (UID: \"a6aa6fe2-3453-4af0-a7e4-fe4c6930abb9\") "
Jul 31 23:49:27.711210 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:49:27.711160    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a6aa6fe2-3453-4af0-a7e4-fe4c6930abb9-console-serving-cert\") pod \"a6aa6fe2-3453-4af0-a7e4-fe4c6930abb9\" (UID: \"a6aa6fe2-3453-4af0-a7e4-fe4c6930abb9\") "
Jul 31 23:49:27.711210 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:49:27.711213    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/a6aa6fe2-3453-4af0-a7e4-fe4c6930abb9-console-config\") pod \"a6aa6fe2-3453-4af0-a7e4-fe4c6930abb9\" (UID: \"a6aa6fe2-3453-4af0-a7e4-fe4c6930abb9\") "
Jul 31 23:49:27.711544 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:49:27.711283    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a6aa6fe2-3453-4af0-a7e4-fe4c6930abb9-service-ca\") pod \"a6aa6fe2-3453-4af0-a7e4-fe4c6930abb9\" (UID: \"a6aa6fe2-3453-4af0-a7e4-fe4c6930abb9\") "
Jul 31 23:49:27.711544 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:49:27.711311    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a6aa6fe2-3453-4af0-a7e4-fe4c6930abb9-trusted-ca-bundle\") pod \"a6aa6fe2-3453-4af0-a7e4-fe4c6930abb9\" (UID: \"a6aa6fe2-3453-4af0-a7e4-fe4c6930abb9\") "
Jul 31 23:49:27.711544 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:49:27.711343    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a6aa6fe2-3453-4af0-a7e4-fe4c6930abb9-console-oauth-config\") pod \"a6aa6fe2-3453-4af0-a7e4-fe4c6930abb9\" (UID: \"a6aa6fe2-3453-4af0-a7e4-fe4c6930abb9\") "
Jul 31 23:49:27.711704 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:49:27.711632    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a6aa6fe2-3453-4af0-a7e4-fe4c6930abb9-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "a6aa6fe2-3453-4af0-a7e4-fe4c6930abb9" (UID: "a6aa6fe2-3453-4af0-a7e4-fe4c6930abb9"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 31 23:49:27.711755 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:49:27.711708    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a6aa6fe2-3453-4af0-a7e4-fe4c6930abb9-console-config" (OuterVolumeSpecName: "console-config") pod "a6aa6fe2-3453-4af0-a7e4-fe4c6930abb9" (UID: "a6aa6fe2-3453-4af0-a7e4-fe4c6930abb9"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 31 23:49:27.711755 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:49:27.711703    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a6aa6fe2-3453-4af0-a7e4-fe4c6930abb9-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "a6aa6fe2-3453-4af0-a7e4-fe4c6930abb9" (UID: "a6aa6fe2-3453-4af0-a7e4-fe4c6930abb9"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 31 23:49:27.712026 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:49:27.712005    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a6aa6fe2-3453-4af0-a7e4-fe4c6930abb9-service-ca" (OuterVolumeSpecName: "service-ca") pod "a6aa6fe2-3453-4af0-a7e4-fe4c6930abb9" (UID: "a6aa6fe2-3453-4af0-a7e4-fe4c6930abb9"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 31 23:49:27.713441 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:49:27.713417    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a6aa6fe2-3453-4af0-a7e4-fe4c6930abb9-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "a6aa6fe2-3453-4af0-a7e4-fe4c6930abb9" (UID: "a6aa6fe2-3453-4af0-a7e4-fe4c6930abb9"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 31 23:49:27.713531 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:49:27.713449    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a6aa6fe2-3453-4af0-a7e4-fe4c6930abb9-kube-api-access-r6n29" (OuterVolumeSpecName: "kube-api-access-r6n29") pod "a6aa6fe2-3453-4af0-a7e4-fe4c6930abb9" (UID: "a6aa6fe2-3453-4af0-a7e4-fe4c6930abb9"). InnerVolumeSpecName "kube-api-access-r6n29". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 31 23:49:27.713531 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:49:27.713466    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a6aa6fe2-3453-4af0-a7e4-fe4c6930abb9-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "a6aa6fe2-3453-4af0-a7e4-fe4c6930abb9" (UID: "a6aa6fe2-3453-4af0-a7e4-fe4c6930abb9"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 31 23:49:27.812450 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:49:27.812405    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r6n29\" (UniqueName: \"kubernetes.io/projected/a6aa6fe2-3453-4af0-a7e4-fe4c6930abb9-kube-api-access-r6n29\") on node \"ip-10-0-136-108.ec2.internal\" DevicePath \"\""
Jul 31 23:49:27.812450 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:49:27.812441    2578 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a6aa6fe2-3453-4af0-a7e4-fe4c6930abb9-oauth-serving-cert\") on node \"ip-10-0-136-108.ec2.internal\" DevicePath \"\""
Jul 31 23:49:27.812450 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:49:27.812451    2578 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a6aa6fe2-3453-4af0-a7e4-fe4c6930abb9-console-serving-cert\") on node \"ip-10-0-136-108.ec2.internal\" DevicePath \"\""
Jul 31 23:49:27.812450 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:49:27.812460    2578 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/a6aa6fe2-3453-4af0-a7e4-fe4c6930abb9-console-config\") on node \"ip-10-0-136-108.ec2.internal\" DevicePath \"\""
Jul 31 23:49:27.812717 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:49:27.812470    2578 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a6aa6fe2-3453-4af0-a7e4-fe4c6930abb9-service-ca\") on node \"ip-10-0-136-108.ec2.internal\" DevicePath \"\""
Jul 31 23:49:27.812717 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:49:27.812478    2578 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a6aa6fe2-3453-4af0-a7e4-fe4c6930abb9-trusted-ca-bundle\") on node \"ip-10-0-136-108.ec2.internal\" DevicePath \"\""
Jul 31 23:49:27.812717 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:49:27.812488    2578 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a6aa6fe2-3453-4af0-a7e4-fe4c6930abb9-console-oauth-config\") on node \"ip-10-0-136-108.ec2.internal\" DevicePath \"\""
Jul 31 23:49:28.419755 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:49:28.419719    2578 generic.go:356] "Generic (PLEG): container finished" podID="a6aa6fe2-3453-4af0-a7e4-fe4c6930abb9" containerID="a633810c87c86cdc80207564d0ee5aa6a8d25c86cad50aa1c5d55b327417d7d3" exitCode=0
Jul 31 23:49:28.420140 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:49:28.419765    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-56655dc5bf-zbmx4"
Jul 31 23:49:28.420140 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:49:28.419803    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-56655dc5bf-zbmx4" event={"ID":"a6aa6fe2-3453-4af0-a7e4-fe4c6930abb9","Type":"ContainerDied","Data":"a633810c87c86cdc80207564d0ee5aa6a8d25c86cad50aa1c5d55b327417d7d3"}
Jul 31 23:49:28.420140 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:49:28.419859    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-56655dc5bf-zbmx4" event={"ID":"a6aa6fe2-3453-4af0-a7e4-fe4c6930abb9","Type":"ContainerDied","Data":"9ff2d27d2f62d7e99ddf83574f5270e5e5008603c906a80dd19f477b061935dc"}
Jul 31 23:49:28.420140 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:49:28.419881    2578 scope.go:117] "RemoveContainer" containerID="a633810c87c86cdc80207564d0ee5aa6a8d25c86cad50aa1c5d55b327417d7d3"
Jul 31 23:49:28.429101 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:49:28.429080    2578 scope.go:117] "RemoveContainer" containerID="a633810c87c86cdc80207564d0ee5aa6a8d25c86cad50aa1c5d55b327417d7d3"
Jul 31 23:49:28.429514 ip-10-0-136-108 kubenswrapper[2578]: E0731 23:49:28.429493    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a633810c87c86cdc80207564d0ee5aa6a8d25c86cad50aa1c5d55b327417d7d3\": container with ID starting with a633810c87c86cdc80207564d0ee5aa6a8d25c86cad50aa1c5d55b327417d7d3 not found: ID does not exist" containerID="a633810c87c86cdc80207564d0ee5aa6a8d25c86cad50aa1c5d55b327417d7d3"
Jul 31 23:49:28.429571 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:49:28.429526    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a633810c87c86cdc80207564d0ee5aa6a8d25c86cad50aa1c5d55b327417d7d3"} err="failed to get container status \"a633810c87c86cdc80207564d0ee5aa6a8d25c86cad50aa1c5d55b327417d7d3\": rpc error: code = NotFound desc = could not find container \"a633810c87c86cdc80207564d0ee5aa6a8d25c86cad50aa1c5d55b327417d7d3\": container with ID starting with a633810c87c86cdc80207564d0ee5aa6a8d25c86cad50aa1c5d55b327417d7d3 not found: ID does not exist"
Jul 31 23:49:28.441507 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:49:28.441478    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-56655dc5bf-zbmx4"]
Jul 31 23:49:28.444444 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:49:28.444412    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-56655dc5bf-zbmx4"]
Jul 31 23:49:28.488961 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:49:28.488931    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a6aa6fe2-3453-4af0-a7e4-fe4c6930abb9" path="/var/lib/kubelet/pods/a6aa6fe2-3453-4af0-a7e4-fe4c6930abb9/volumes"
Jul 31 23:49:38.428163 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:49:38.428114    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-5cd747c9c6-gsrrn" podUID="9011d6f9-f60e-456a-8da5-c4b5d3225638" containerName="console" containerID="cri-o://b07e333522c58c610656f5a271aae252c5ce01290f082579d2c0d02e682d4a1c" gracePeriod=15
Jul 31 23:49:38.672340 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:49:38.672310    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5cd747c9c6-gsrrn"
Jul 31 23:49:38.811486 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:49:38.811437    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mtfmp\" (UniqueName: \"kubernetes.io/projected/9011d6f9-f60e-456a-8da5-c4b5d3225638-kube-api-access-mtfmp\") pod \"9011d6f9-f60e-456a-8da5-c4b5d3225638\" (UID: \"9011d6f9-f60e-456a-8da5-c4b5d3225638\") "
Jul 31 23:49:38.811486 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:49:38.811491    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/9011d6f9-f60e-456a-8da5-c4b5d3225638-oauth-serving-cert\") pod \"9011d6f9-f60e-456a-8da5-c4b5d3225638\" (UID: \"9011d6f9-f60e-456a-8da5-c4b5d3225638\") "
Jul 31 23:49:38.811794 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:49:38.811515    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/9011d6f9-f60e-456a-8da5-c4b5d3225638-console-serving-cert\") pod \"9011d6f9-f60e-456a-8da5-c4b5d3225638\" (UID: \"9011d6f9-f60e-456a-8da5-c4b5d3225638\") "
Jul 31 23:49:38.811794 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:49:38.811579    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/9011d6f9-f60e-456a-8da5-c4b5d3225638-console-oauth-config\") pod \"9011d6f9-f60e-456a-8da5-c4b5d3225638\" (UID: \"9011d6f9-f60e-456a-8da5-c4b5d3225638\") "
Jul 31 23:49:38.811794 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:49:38.811622    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9011d6f9-f60e-456a-8da5-c4b5d3225638-trusted-ca-bundle\") pod \"9011d6f9-f60e-456a-8da5-c4b5d3225638\" (UID: \"9011d6f9-f60e-456a-8da5-c4b5d3225638\") "
Jul 31 23:49:38.811794 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:49:38.811645    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/9011d6f9-f60e-456a-8da5-c4b5d3225638-console-config\") pod \"9011d6f9-f60e-456a-8da5-c4b5d3225638\" (UID: \"9011d6f9-f60e-456a-8da5-c4b5d3225638\") "
Jul 31 23:49:38.811794 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:49:38.811681    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/9011d6f9-f60e-456a-8da5-c4b5d3225638-service-ca\") pod \"9011d6f9-f60e-456a-8da5-c4b5d3225638\" (UID: \"9011d6f9-f60e-456a-8da5-c4b5d3225638\") "
Jul 31 23:49:38.812049 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:49:38.812009    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9011d6f9-f60e-456a-8da5-c4b5d3225638-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "9011d6f9-f60e-456a-8da5-c4b5d3225638" (UID: "9011d6f9-f60e-456a-8da5-c4b5d3225638"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 31 23:49:38.812105 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:49:38.812052    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9011d6f9-f60e-456a-8da5-c4b5d3225638-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "9011d6f9-f60e-456a-8da5-c4b5d3225638" (UID: "9011d6f9-f60e-456a-8da5-c4b5d3225638"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 31 23:49:38.812212 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:49:38.812159    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9011d6f9-f60e-456a-8da5-c4b5d3225638-console-config" (OuterVolumeSpecName: "console-config") pod "9011d6f9-f60e-456a-8da5-c4b5d3225638" (UID: "9011d6f9-f60e-456a-8da5-c4b5d3225638"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 31 23:49:38.812278 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:49:38.812235    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9011d6f9-f60e-456a-8da5-c4b5d3225638-service-ca" (OuterVolumeSpecName: "service-ca") pod "9011d6f9-f60e-456a-8da5-c4b5d3225638" (UID: "9011d6f9-f60e-456a-8da5-c4b5d3225638"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 31 23:49:38.814005 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:49:38.813962    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9011d6f9-f60e-456a-8da5-c4b5d3225638-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "9011d6f9-f60e-456a-8da5-c4b5d3225638" (UID: "9011d6f9-f60e-456a-8da5-c4b5d3225638"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 31 23:49:38.814005 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:49:38.813979    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9011d6f9-f60e-456a-8da5-c4b5d3225638-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "9011d6f9-f60e-456a-8da5-c4b5d3225638" (UID: "9011d6f9-f60e-456a-8da5-c4b5d3225638"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 31 23:49:38.814187 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:49:38.814030    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9011d6f9-f60e-456a-8da5-c4b5d3225638-kube-api-access-mtfmp" (OuterVolumeSpecName: "kube-api-access-mtfmp") pod "9011d6f9-f60e-456a-8da5-c4b5d3225638" (UID: "9011d6f9-f60e-456a-8da5-c4b5d3225638"). InnerVolumeSpecName "kube-api-access-mtfmp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 31 23:49:38.912398 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:49:38.912364    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mtfmp\" (UniqueName: \"kubernetes.io/projected/9011d6f9-f60e-456a-8da5-c4b5d3225638-kube-api-access-mtfmp\") on node \"ip-10-0-136-108.ec2.internal\" DevicePath \"\""
Jul 31 23:49:38.912398 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:49:38.912393    2578 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/9011d6f9-f60e-456a-8da5-c4b5d3225638-oauth-serving-cert\") on node \"ip-10-0-136-108.ec2.internal\" DevicePath \"\""
Jul 31 23:49:38.912398 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:49:38.912403    2578 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/9011d6f9-f60e-456a-8da5-c4b5d3225638-console-serving-cert\") on node \"ip-10-0-136-108.ec2.internal\" DevicePath \"\""
Jul 31 23:49:38.912620 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:49:38.912412    2578 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/9011d6f9-f60e-456a-8da5-c4b5d3225638-console-oauth-config\") on node \"ip-10-0-136-108.ec2.internal\" DevicePath \"\""
Jul 31 23:49:38.912620 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:49:38.912421    2578 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9011d6f9-f60e-456a-8da5-c4b5d3225638-trusted-ca-bundle\") on node \"ip-10-0-136-108.ec2.internal\" DevicePath \"\""
Jul 31 23:49:38.912620 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:49:38.912430    2578 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/9011d6f9-f60e-456a-8da5-c4b5d3225638-console-config\") on node \"ip-10-0-136-108.ec2.internal\" DevicePath \"\""
Jul 31 23:49:38.912620 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:49:38.912438    2578 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/9011d6f9-f60e-456a-8da5-c4b5d3225638-service-ca\") on node \"ip-10-0-136-108.ec2.internal\" DevicePath \"\""
Jul 31 23:49:39.463389 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:49:39.463350    2578 generic.go:356] "Generic (PLEG): container finished" podID="9011d6f9-f60e-456a-8da5-c4b5d3225638" containerID="b07e333522c58c610656f5a271aae252c5ce01290f082579d2c0d02e682d4a1c" exitCode=0
Jul 31 23:49:39.463782 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:49:39.463399    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5cd747c9c6-gsrrn"
Jul 31 23:49:39.463782 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:49:39.463437    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5cd747c9c6-gsrrn" event={"ID":"9011d6f9-f60e-456a-8da5-c4b5d3225638","Type":"ContainerDied","Data":"b07e333522c58c610656f5a271aae252c5ce01290f082579d2c0d02e682d4a1c"}
Jul 31 23:49:39.463782 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:49:39.463494    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5cd747c9c6-gsrrn" event={"ID":"9011d6f9-f60e-456a-8da5-c4b5d3225638","Type":"ContainerDied","Data":"4faeb2d5f17807c79bbfd5ae420aa564e099e10d98c60fbfe41a723fac96fdce"}
Jul 31 23:49:39.463782 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:49:39.463518    2578 scope.go:117] "RemoveContainer" containerID="b07e333522c58c610656f5a271aae252c5ce01290f082579d2c0d02e682d4a1c"
Jul 31 23:49:39.471728 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:49:39.471711    2578 scope.go:117] "RemoveContainer" containerID="b07e333522c58c610656f5a271aae252c5ce01290f082579d2c0d02e682d4a1c"
Jul 31 23:49:39.472003 ip-10-0-136-108 kubenswrapper[2578]: E0731 23:49:39.471983    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b07e333522c58c610656f5a271aae252c5ce01290f082579d2c0d02e682d4a1c\": container with ID starting with b07e333522c58c610656f5a271aae252c5ce01290f082579d2c0d02e682d4a1c not found: ID does not exist" containerID="b07e333522c58c610656f5a271aae252c5ce01290f082579d2c0d02e682d4a1c"
Jul 31 23:49:39.472053 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:49:39.472011    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b07e333522c58c610656f5a271aae252c5ce01290f082579d2c0d02e682d4a1c"} err="failed to get container status \"b07e333522c58c610656f5a271aae252c5ce01290f082579d2c0d02e682d4a1c\": rpc error: code = NotFound desc = could not find container \"b07e333522c58c610656f5a271aae252c5ce01290f082579d2c0d02e682d4a1c\": container with ID starting with b07e333522c58c610656f5a271aae252c5ce01290f082579d2c0d02e682d4a1c not found: ID does not exist"
Jul 31 23:49:39.482998 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:49:39.482970    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5cd747c9c6-gsrrn"]
Jul 31 23:49:39.486408 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:49:39.486388    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-5cd747c9c6-gsrrn"]
Jul 31 23:49:40.487822 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:49:40.487783    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9011d6f9-f60e-456a-8da5-c4b5d3225638" path="/var/lib/kubelet/pods/9011d6f9-f60e-456a-8da5-c4b5d3225638/volumes"
Jul 31 23:50:23.449780 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:50:23.449740    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gttdm_b32e6d42-e4e0-4113-b8ce-f5d5c5dc13f7/konnectivity-agent/0.log" limit=1024
Jul 31 23:50:23.550621 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:50:23.550584    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gttdm_b32e6d42-e4e0-4113-b8ce-f5d5c5dc13f7/konnectivity-agent/0.log" limit=1024
Jul 31 23:50:23.650656 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:50:23.650620    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gttdm_b32e6d42-e4e0-4113-b8ce-f5d5c5dc13f7/konnectivity-agent/0.log" limit=1024
Jul 31 23:50:24.424467 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:50:24.424429    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5lhp79"]
Jul 31 23:50:24.424780 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:50:24.424767    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9011d6f9-f60e-456a-8da5-c4b5d3225638" containerName="console"
Jul 31 23:50:24.424823 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:50:24.424782    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="9011d6f9-f60e-456a-8da5-c4b5d3225638" containerName="console"
Jul 31 23:50:24.424823 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:50:24.424799    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a6aa6fe2-3453-4af0-a7e4-fe4c6930abb9" containerName="console"
Jul 31 23:50:24.424823 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:50:24.424805    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="a6aa6fe2-3453-4af0-a7e4-fe4c6930abb9" containerName="console"
Jul 31 23:50:24.424919 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:50:24.424861    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="9011d6f9-f60e-456a-8da5-c4b5d3225638" containerName="console"
Jul 31 23:50:24.424919 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:50:24.424872    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="a6aa6fe2-3453-4af0-a7e4-fe4c6930abb9" containerName="console"
Jul 31 23:50:24.429209 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:50:24.429193    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5lhp79"
Jul 31 23:50:24.431377 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:50:24.431354    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Jul 31 23:50:24.431508 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:50:24.431411    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-dhpcw\""
Jul 31 23:50:24.431508 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:50:24.431424    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Jul 31 23:50:24.435490 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:50:24.435465    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5lhp79"]
Jul 31 23:50:24.579651 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:50:24.579617    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h82kz\" (UniqueName: \"kubernetes.io/projected/929d0b38-efa2-4ff4-84a1-d9409b373f79-kube-api-access-h82kz\") pod \"320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5lhp79\" (UID: \"929d0b38-efa2-4ff4-84a1-d9409b373f79\") " pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5lhp79"
Jul 31 23:50:24.580023 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:50:24.579675    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/929d0b38-efa2-4ff4-84a1-d9409b373f79-util\") pod \"320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5lhp79\" (UID: \"929d0b38-efa2-4ff4-84a1-d9409b373f79\") " pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5lhp79"
Jul 31 23:50:24.580023 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:50:24.579703    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/929d0b38-efa2-4ff4-84a1-d9409b373f79-bundle\") pod \"320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5lhp79\" (UID: \"929d0b38-efa2-4ff4-84a1-d9409b373f79\") " pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5lhp79"
Jul 31 23:50:24.680481 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:50:24.680386    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h82kz\" (UniqueName: \"kubernetes.io/projected/929d0b38-efa2-4ff4-84a1-d9409b373f79-kube-api-access-h82kz\") pod \"320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5lhp79\" (UID: \"929d0b38-efa2-4ff4-84a1-d9409b373f79\") " pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5lhp79"
Jul 31 23:50:24.680481 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:50:24.680444    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/929d0b38-efa2-4ff4-84a1-d9409b373f79-util\") pod \"320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5lhp79\" (UID: \"929d0b38-efa2-4ff4-84a1-d9409b373f79\") " pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5lhp79"
Jul 31 23:50:24.680481 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:50:24.680468    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/929d0b38-efa2-4ff4-84a1-d9409b373f79-bundle\") pod \"320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5lhp79\" (UID: \"929d0b38-efa2-4ff4-84a1-d9409b373f79\") " pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5lhp79"
Jul 31 23:50:24.680814 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:50:24.680796    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/929d0b38-efa2-4ff4-84a1-d9409b373f79-bundle\") pod \"320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5lhp79\" (UID: \"929d0b38-efa2-4ff4-84a1-d9409b373f79\") " pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5lhp79"
Jul 31 23:50:24.680875 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:50:24.680857    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/929d0b38-efa2-4ff4-84a1-d9409b373f79-util\") pod \"320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5lhp79\" (UID: \"929d0b38-efa2-4ff4-84a1-d9409b373f79\") " pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5lhp79"
Jul 31 23:50:24.688462 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:50:24.688430    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h82kz\" (UniqueName: \"kubernetes.io/projected/929d0b38-efa2-4ff4-84a1-d9409b373f79-kube-api-access-h82kz\") pod \"320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5lhp79\" (UID: \"929d0b38-efa2-4ff4-84a1-d9409b373f79\") " pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5lhp79"
Jul 31 23:50:24.739480 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:50:24.739439    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5lhp79"
Jul 31 23:50:24.870039 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:50:24.870006    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5lhp79"]
Jul 31 23:50:24.873284 ip-10-0-136-108 kubenswrapper[2578]: W0731 23:50:24.873258    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod929d0b38_efa2_4ff4_84a1_d9409b373f79.slice/crio-9cc8b3e5ae03c445c5c7ba516117e3b9512934b17a2855ab6275448a8a26a4b6 WatchSource:0}: Error finding container 9cc8b3e5ae03c445c5c7ba516117e3b9512934b17a2855ab6275448a8a26a4b6: Status 404 returned error can't find the container with id 9cc8b3e5ae03c445c5c7ba516117e3b9512934b17a2855ab6275448a8a26a4b6
Jul 31 23:50:25.628213 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:50:25.628167    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5lhp79" event={"ID":"929d0b38-efa2-4ff4-84a1-d9409b373f79","Type":"ContainerStarted","Data":"9cc8b3e5ae03c445c5c7ba516117e3b9512934b17a2855ab6275448a8a26a4b6"}
Jul 31 23:50:30.650390 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:50:30.650354    2578 generic.go:356] "Generic (PLEG): container finished" podID="929d0b38-efa2-4ff4-84a1-d9409b373f79" containerID="b13b442c752bb2d7cfd37acccc04acc7637497982129b4be60125bebd990c18e" exitCode=0
Jul 31 23:50:30.650779 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:50:30.650422    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5lhp79" event={"ID":"929d0b38-efa2-4ff4-84a1-d9409b373f79","Type":"ContainerDied","Data":"b13b442c752bb2d7cfd37acccc04acc7637497982129b4be60125bebd990c18e"}
Jul 31 23:50:32.823484 ip-10-0-136-108 kubenswrapper[2578]: E0731 23:50:32.823450    2578 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod929d0b38_efa2_4ff4_84a1_d9409b373f79.slice/crio-89f99d68ecbdd173fab188511dd5e201024e4869afc202095de2619f16cf6080.scope\": RecentStats: unable to find data in memory cache]"
Jul 31 23:50:33.665882 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:50:33.665840    2578 generic.go:356] "Generic (PLEG): container finished" podID="929d0b38-efa2-4ff4-84a1-d9409b373f79" containerID="89f99d68ecbdd173fab188511dd5e201024e4869afc202095de2619f16cf6080" exitCode=0
Jul 31 23:50:33.666056 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:50:33.665922    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5lhp79" event={"ID":"929d0b38-efa2-4ff4-84a1-d9409b373f79","Type":"ContainerDied","Data":"89f99d68ecbdd173fab188511dd5e201024e4869afc202095de2619f16cf6080"}
Jul 31 23:50:36.960972 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:50:36.960900    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gttdm_b32e6d42-e4e0-4113-b8ce-f5d5c5dc13f7/konnectivity-agent/0.log" limit=1024
Jul 31 23:50:40.701305 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:50:40.701270    2578 generic.go:356] "Generic (PLEG): container finished" podID="929d0b38-efa2-4ff4-84a1-d9409b373f79" containerID="3a25170a04908f7997abde39b44b3c07dd83fb36c50e4e94537b0293c729c63a" exitCode=0
Jul 31 23:50:40.701305 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:50:40.701309    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5lhp79" event={"ID":"929d0b38-efa2-4ff4-84a1-d9409b373f79","Type":"ContainerDied","Data":"3a25170a04908f7997abde39b44b3c07dd83fb36c50e4e94537b0293c729c63a"}
Jul 31 23:50:41.830372 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:50:41.830323    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5lhp79"
Jul 31 23:50:41.927646 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:50:41.927606    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-h82kz\" (UniqueName: \"kubernetes.io/projected/929d0b38-efa2-4ff4-84a1-d9409b373f79-kube-api-access-h82kz\") pod \"929d0b38-efa2-4ff4-84a1-d9409b373f79\" (UID: \"929d0b38-efa2-4ff4-84a1-d9409b373f79\") "
Jul 31 23:50:41.927793 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:50:41.927718    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/929d0b38-efa2-4ff4-84a1-d9409b373f79-util\") pod \"929d0b38-efa2-4ff4-84a1-d9409b373f79\" (UID: \"929d0b38-efa2-4ff4-84a1-d9409b373f79\") "
Jul 31 23:50:41.927793 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:50:41.927736    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/929d0b38-efa2-4ff4-84a1-d9409b373f79-bundle\") pod \"929d0b38-efa2-4ff4-84a1-d9409b373f79\" (UID: \"929d0b38-efa2-4ff4-84a1-d9409b373f79\") "
Jul 31 23:50:41.928584 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:50:41.928555    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/929d0b38-efa2-4ff4-84a1-d9409b373f79-bundle" (OuterVolumeSpecName: "bundle") pod "929d0b38-efa2-4ff4-84a1-d9409b373f79" (UID: "929d0b38-efa2-4ff4-84a1-d9409b373f79"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 31 23:50:41.929780 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:50:41.929755    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/929d0b38-efa2-4ff4-84a1-d9409b373f79-kube-api-access-h82kz" (OuterVolumeSpecName: "kube-api-access-h82kz") pod "929d0b38-efa2-4ff4-84a1-d9409b373f79" (UID: "929d0b38-efa2-4ff4-84a1-d9409b373f79"). InnerVolumeSpecName "kube-api-access-h82kz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 31 23:50:41.932440 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:50:41.932409    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/929d0b38-efa2-4ff4-84a1-d9409b373f79-util" (OuterVolumeSpecName: "util") pod "929d0b38-efa2-4ff4-84a1-d9409b373f79" (UID: "929d0b38-efa2-4ff4-84a1-d9409b373f79"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 31 23:50:42.028234 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:50:42.028196    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h82kz\" (UniqueName: \"kubernetes.io/projected/929d0b38-efa2-4ff4-84a1-d9409b373f79-kube-api-access-h82kz\") on node \"ip-10-0-136-108.ec2.internal\" DevicePath \"\""
Jul 31 23:50:42.028234 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:50:42.028231    2578 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/929d0b38-efa2-4ff4-84a1-d9409b373f79-util\") on node \"ip-10-0-136-108.ec2.internal\" DevicePath \"\""
Jul 31 23:50:42.028234 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:50:42.028241    2578 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/929d0b38-efa2-4ff4-84a1-d9409b373f79-bundle\") on node \"ip-10-0-136-108.ec2.internal\" DevicePath \"\""
Jul 31 23:50:42.710803 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:50:42.710769    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5lhp79" event={"ID":"929d0b38-efa2-4ff4-84a1-d9409b373f79","Type":"ContainerDied","Data":"9cc8b3e5ae03c445c5c7ba516117e3b9512934b17a2855ab6275448a8a26a4b6"}
Jul 31 23:50:42.710959 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:50:42.710818    2578 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9cc8b3e5ae03c445c5c7ba516117e3b9512934b17a2855ab6275448a8a26a4b6"
Jul 31 23:50:42.710959 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:50:42.710820    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5lhp79"
Jul 31 23:50:45.548097 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:50:45.548061    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gttdm_b32e6d42-e4e0-4113-b8ce-f5d5c5dc13f7/konnectivity-agent/0.log" limit=1024
Jul 31 23:50:45.641821 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:50:45.641782    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gttdm_b32e6d42-e4e0-4113-b8ce-f5d5c5dc13f7/konnectivity-agent/0.log" limit=1024
Jul 31 23:50:47.339363 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:50:47.339319    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-keda/custom-metrics-autoscaler-operator-6969cb7dfb-f8sgn"]
Jul 31 23:50:47.339757 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:50:47.339664    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="929d0b38-efa2-4ff4-84a1-d9409b373f79" containerName="util"
Jul 31 23:50:47.339757 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:50:47.339677    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="929d0b38-efa2-4ff4-84a1-d9409b373f79" containerName="util"
Jul 31 23:50:47.339757 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:50:47.339691    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="929d0b38-efa2-4ff4-84a1-d9409b373f79" containerName="extract"
Jul 31 23:50:47.339757 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:50:47.339698    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="929d0b38-efa2-4ff4-84a1-d9409b373f79" containerName="extract"
Jul 31 23:50:47.339757 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:50:47.339709    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="929d0b38-efa2-4ff4-84a1-d9409b373f79" containerName="pull"
Jul 31 23:50:47.339757 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:50:47.339717    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="929d0b38-efa2-4ff4-84a1-d9409b373f79" containerName="pull"
Jul 31 23:50:47.339939 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:50:47.339781    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="929d0b38-efa2-4ff4-84a1-d9409b373f79" containerName="extract"
Jul 31 23:50:47.343225 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:50:47.343205    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/custom-metrics-autoscaler-operator-6969cb7dfb-f8sgn"
Jul 31 23:50:47.345403 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:50:47.345376    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"kedaorg-certs\""
Jul 31 23:50:47.345403 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:50:47.345395    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"openshift-service-ca.crt\""
Jul 31 23:50:47.345566 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:50:47.345380    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"kube-root-ca.crt\""
Jul 31 23:50:47.345566 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:50:47.345440    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"custom-metrics-autoscaler-operator-dockercfg-5bgtr\""
Jul 31 23:50:47.356511 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:50:47.356487    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/custom-metrics-autoscaler-operator-6969cb7dfb-f8sgn"]
Jul 31 23:50:47.370788 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:50:47.370761    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/secret/92e82be2-00fd-459b-a5a4-42e6f11fd2c4-certificates\") pod \"custom-metrics-autoscaler-operator-6969cb7dfb-f8sgn\" (UID: \"92e82be2-00fd-459b-a5a4-42e6f11fd2c4\") " pod="openshift-keda/custom-metrics-autoscaler-operator-6969cb7dfb-f8sgn"
Jul 31 23:50:47.370932 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:50:47.370819    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-44xr8\" (UniqueName: \"kubernetes.io/projected/92e82be2-00fd-459b-a5a4-42e6f11fd2c4-kube-api-access-44xr8\") pod \"custom-metrics-autoscaler-operator-6969cb7dfb-f8sgn\" (UID: \"92e82be2-00fd-459b-a5a4-42e6f11fd2c4\") " pod="openshift-keda/custom-metrics-autoscaler-operator-6969cb7dfb-f8sgn"
Jul 31 23:50:47.471449 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:50:47.471409    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/secret/92e82be2-00fd-459b-a5a4-42e6f11fd2c4-certificates\") pod \"custom-metrics-autoscaler-operator-6969cb7dfb-f8sgn\" (UID: \"92e82be2-00fd-459b-a5a4-42e6f11fd2c4\") " pod="openshift-keda/custom-metrics-autoscaler-operator-6969cb7dfb-f8sgn"
Jul 31 23:50:47.471621 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:50:47.471467    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-44xr8\" (UniqueName: \"kubernetes.io/projected/92e82be2-00fd-459b-a5a4-42e6f11fd2c4-kube-api-access-44xr8\") pod \"custom-metrics-autoscaler-operator-6969cb7dfb-f8sgn\" (UID: \"92e82be2-00fd-459b-a5a4-42e6f11fd2c4\") " pod="openshift-keda/custom-metrics-autoscaler-operator-6969cb7dfb-f8sgn"
Jul 31 23:50:47.473768 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:50:47.473738    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"certificates\" (UniqueName: \"kubernetes.io/secret/92e82be2-00fd-459b-a5a4-42e6f11fd2c4-certificates\") pod \"custom-metrics-autoscaler-operator-6969cb7dfb-f8sgn\" (UID: \"92e82be2-00fd-459b-a5a4-42e6f11fd2c4\") " pod="openshift-keda/custom-metrics-autoscaler-operator-6969cb7dfb-f8sgn"
Jul 31 23:50:47.483298 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:50:47.483275    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-44xr8\" (UniqueName: \"kubernetes.io/projected/92e82be2-00fd-459b-a5a4-42e6f11fd2c4-kube-api-access-44xr8\") pod \"custom-metrics-autoscaler-operator-6969cb7dfb-f8sgn\" (UID: \"92e82be2-00fd-459b-a5a4-42e6f11fd2c4\") " pod="openshift-keda/custom-metrics-autoscaler-operator-6969cb7dfb-f8sgn"
Jul 31 23:50:47.652890 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:50:47.652798    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/custom-metrics-autoscaler-operator-6969cb7dfb-f8sgn"
Jul 31 23:50:47.785936 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:50:47.785905    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/custom-metrics-autoscaler-operator-6969cb7dfb-f8sgn"]
Jul 31 23:50:47.789253 ip-10-0-136-108 kubenswrapper[2578]: W0731 23:50:47.789228    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod92e82be2_00fd_459b_a5a4_42e6f11fd2c4.slice/crio-54d00abefafbca74a8d88c38af2353869076287a1bf691466c2c4a6a0d70e652 WatchSource:0}: Error finding container 54d00abefafbca74a8d88c38af2353869076287a1bf691466c2c4a6a0d70e652: Status 404 returned error can't find the container with id 54d00abefafbca74a8d88c38af2353869076287a1bf691466c2c4a6a0d70e652
Jul 31 23:50:48.736939 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:50:48.736906    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/custom-metrics-autoscaler-operator-6969cb7dfb-f8sgn" event={"ID":"92e82be2-00fd-459b-a5a4-42e6f11fd2c4","Type":"ContainerStarted","Data":"54d00abefafbca74a8d88c38af2353869076287a1bf691466c2c4a6a0d70e652"}
Jul 31 23:50:50.539153 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:50:50.539124    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gttdm_b32e6d42-e4e0-4113-b8ce-f5d5c5dc13f7/konnectivity-agent/0.log" limit=1024
Jul 31 23:50:50.746288 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:50:50.746250    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/custom-metrics-autoscaler-operator-6969cb7dfb-f8sgn" event={"ID":"92e82be2-00fd-459b-a5a4-42e6f11fd2c4","Type":"ContainerStarted","Data":"789867aa15097a9f6af6952150209c35e611b52996fea39fd3471253cdd6b4f7"}
Jul 31 23:50:50.746478 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:50:50.746354    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-keda/custom-metrics-autoscaler-operator-6969cb7dfb-f8sgn"
Jul 31 23:50:50.769112 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:50:50.769060    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-keda/custom-metrics-autoscaler-operator-6969cb7dfb-f8sgn" podStartSLOduration=1.432220489 podStartE2EDuration="3.769044167s" totalImagesPullingTime="2.336823678s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 23:50:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 23:50:50.766925736 +0000 UTC m=+274.734676494" watchObservedRunningTime="2026-07-31 23:50:50.769044167 +0000 UTC m=+274.736794925"
Jul 31 23:50:50.842506 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:50:50.842392    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gttdm_b32e6d42-e4e0-4113-b8ce-f5d5c5dc13f7/konnectivity-agent/0.log" limit=1024
Jul 31 23:50:50.845901 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:50:50.845874    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-keda/keda-operator-77f648d694-hx674"]
Jul 31 23:50:50.851539 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:50:50.850129    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-operator-77f648d694-hx674"
Jul 31 23:50:50.852709 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:50:50.852687    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-operator-certs\""
Jul 31 23:50:50.852829 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:50:50.852712    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"keda-ocp-cabundle\""
Jul 31 23:50:50.852829 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:50:50.852744    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-operator-dockercfg-2lkzb\""
Jul 31 23:50:50.859370 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:50:50.859338    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-operator-77f648d694-hx674"]
Jul 31 23:50:50.902090 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:50:50.902057    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4z4n8\" (UniqueName: \"kubernetes.io/projected/8b7a412c-4d6a-4619-8cb6-5bada3791b5c-kube-api-access-4z4n8\") pod \"keda-operator-77f648d694-hx674\" (UID: \"8b7a412c-4d6a-4619-8cb6-5bada3791b5c\") " pod="openshift-keda/keda-operator-77f648d694-hx674"
Jul 31 23:50:50.902275 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:50:50.902110    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/8b7a412c-4d6a-4619-8cb6-5bada3791b5c-certificates\") pod \"keda-operator-77f648d694-hx674\" (UID: \"8b7a412c-4d6a-4619-8cb6-5bada3791b5c\") " pod="openshift-keda/keda-operator-77f648d694-hx674"
Jul 31 23:50:50.902275 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:50:50.902199    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle0\" (UniqueName: \"kubernetes.io/configmap/8b7a412c-4d6a-4619-8cb6-5bada3791b5c-cabundle0\") pod \"keda-operator-77f648d694-hx674\" (UID: \"8b7a412c-4d6a-4619-8cb6-5bada3791b5c\") " pod="openshift-keda/keda-operator-77f648d694-hx674"
Jul 31 23:50:50.936277 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:50:50.936241    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gttdm_b32e6d42-e4e0-4113-b8ce-f5d5c5dc13f7/konnectivity-agent/0.log" limit=1024
Jul 31 23:50:51.002641 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:50:51.002603    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4z4n8\" (UniqueName: \"kubernetes.io/projected/8b7a412c-4d6a-4619-8cb6-5bada3791b5c-kube-api-access-4z4n8\") pod \"keda-operator-77f648d694-hx674\" (UID: \"8b7a412c-4d6a-4619-8cb6-5bada3791b5c\") " pod="openshift-keda/keda-operator-77f648d694-hx674"
Jul 31 23:50:51.002840 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:50:51.002686    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/8b7a412c-4d6a-4619-8cb6-5bada3791b5c-certificates\") pod \"keda-operator-77f648d694-hx674\" (UID: \"8b7a412c-4d6a-4619-8cb6-5bada3791b5c\") " pod="openshift-keda/keda-operator-77f648d694-hx674"
Jul 31 23:50:51.002840 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:50:51.002727    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle0\" (UniqueName: \"kubernetes.io/configmap/8b7a412c-4d6a-4619-8cb6-5bada3791b5c-cabundle0\") pod \"keda-operator-77f648d694-hx674\" (UID: \"8b7a412c-4d6a-4619-8cb6-5bada3791b5c\") " pod="openshift-keda/keda-operator-77f648d694-hx674"
Jul 31 23:50:51.002840 ip-10-0-136-108 kubenswrapper[2578]: E0731 23:50:51.002823    2578 secret.go:281] references non-existent secret key: ca.crt
Jul 31 23:50:51.002840 ip-10-0-136-108 kubenswrapper[2578]: E0731 23:50:51.002843    2578 projected.go:279] Couldn't get secret payload openshift-keda/kedaorg-certs: references non-existent secret key: ca.crt
Jul 31 23:50:51.003085 ip-10-0-136-108 kubenswrapper[2578]: E0731 23:50:51.002852    2578 projected.go:196] Error preparing data for projected volume certificates for pod openshift-keda/keda-operator-77f648d694-hx674: references non-existent secret key: ca.crt
Jul 31 23:50:51.003085 ip-10-0-136-108 kubenswrapper[2578]: E0731 23:50:51.002921    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8b7a412c-4d6a-4619-8cb6-5bada3791b5c-certificates podName:8b7a412c-4d6a-4619-8cb6-5bada3791b5c nodeName:}" failed. No retries permitted until 2026-07-31 23:50:51.502903598 +0000 UTC m=+275.470654333 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "certificates" (UniqueName: "kubernetes.io/projected/8b7a412c-4d6a-4619-8cb6-5bada3791b5c-certificates") pod "keda-operator-77f648d694-hx674" (UID: "8b7a412c-4d6a-4619-8cb6-5bada3791b5c") : references non-existent secret key: ca.crt
Jul 31 23:50:51.003475 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:50:51.003453    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle0\" (UniqueName: \"kubernetes.io/configmap/8b7a412c-4d6a-4619-8cb6-5bada3791b5c-cabundle0\") pod \"keda-operator-77f648d694-hx674\" (UID: \"8b7a412c-4d6a-4619-8cb6-5bada3791b5c\") " pod="openshift-keda/keda-operator-77f648d694-hx674"
Jul 31 23:50:51.011791 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:50:51.011765    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4z4n8\" (UniqueName: \"kubernetes.io/projected/8b7a412c-4d6a-4619-8cb6-5bada3791b5c-kube-api-access-4z4n8\") pod \"keda-operator-77f648d694-hx674\" (UID: \"8b7a412c-4d6a-4619-8cb6-5bada3791b5c\") " pod="openshift-keda/keda-operator-77f648d694-hx674"
Jul 31 23:50:51.506852 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:50:51.506814    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/8b7a412c-4d6a-4619-8cb6-5bada3791b5c-certificates\") pod \"keda-operator-77f648d694-hx674\" (UID: \"8b7a412c-4d6a-4619-8cb6-5bada3791b5c\") " pod="openshift-keda/keda-operator-77f648d694-hx674"
Jul 31 23:50:51.507036 ip-10-0-136-108 kubenswrapper[2578]: E0731 23:50:51.507014    2578 secret.go:281] references non-existent secret key: ca.crt
Jul 31 23:50:51.507036 ip-10-0-136-108 kubenswrapper[2578]: E0731 23:50:51.507034    2578 projected.go:279] Couldn't get secret payload openshift-keda/kedaorg-certs: references non-existent secret key: ca.crt
Jul 31 23:50:51.507108 ip-10-0-136-108 kubenswrapper[2578]: E0731 23:50:51.507042    2578 projected.go:196] Error preparing data for projected volume certificates for pod openshift-keda/keda-operator-77f648d694-hx674: references non-existent secret key: ca.crt
Jul 31 23:50:51.507141 ip-10-0-136-108 kubenswrapper[2578]: E0731 23:50:51.507118    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8b7a412c-4d6a-4619-8cb6-5bada3791b5c-certificates podName:8b7a412c-4d6a-4619-8cb6-5bada3791b5c nodeName:}" failed. No retries permitted until 2026-07-31 23:50:52.507093938 +0000 UTC m=+276.474844688 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "certificates" (UniqueName: "kubernetes.io/projected/8b7a412c-4d6a-4619-8cb6-5bada3791b5c-certificates") pod "keda-operator-77f648d694-hx674" (UID: "8b7a412c-4d6a-4619-8cb6-5bada3791b5c") : references non-existent secret key: ca.crt
Jul 31 23:50:51.712782 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:50:51.712738    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-keda/keda-metrics-apiserver-55557df8c-nbtbz"]
Jul 31 23:50:51.716198 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:50:51.716164    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-metrics-apiserver-55557df8c-nbtbz"
Jul 31 23:50:51.718092 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:50:51.718068    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-metrics-apiserver-certs\""
Jul 31 23:50:51.719261 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:50:51.719234    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gttdm_b32e6d42-e4e0-4113-b8ce-f5d5c5dc13f7/konnectivity-agent/0.log" limit=1024
Jul 31 23:50:51.725390 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:50:51.725367    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-metrics-apiserver-55557df8c-nbtbz"]
Jul 31 23:50:51.811003 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:50:51.810918    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/3ff6f792-9f1b-465a-a8f1-52f1bc7e35d9-certificates\") pod \"keda-metrics-apiserver-55557df8c-nbtbz\" (UID: \"3ff6f792-9f1b-465a-a8f1-52f1bc7e35d9\") " pod="openshift-keda/keda-metrics-apiserver-55557df8c-nbtbz"
Jul 31 23:50:51.811003 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:50:51.810978    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"temp-vol\" (UniqueName: \"kubernetes.io/empty-dir/3ff6f792-9f1b-465a-a8f1-52f1bc7e35d9-temp-vol\") pod \"keda-metrics-apiserver-55557df8c-nbtbz\" (UID: \"3ff6f792-9f1b-465a-a8f1-52f1bc7e35d9\") " pod="openshift-keda/keda-metrics-apiserver-55557df8c-nbtbz"
Jul 31 23:50:51.811221 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:50:51.811007    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kqk8w\" (UniqueName: \"kubernetes.io/projected/3ff6f792-9f1b-465a-a8f1-52f1bc7e35d9-kube-api-access-kqk8w\") pod \"keda-metrics-apiserver-55557df8c-nbtbz\" (UID: \"3ff6f792-9f1b-465a-a8f1-52f1bc7e35d9\") " pod="openshift-keda/keda-metrics-apiserver-55557df8c-nbtbz"
Jul 31 23:50:51.912259 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:50:51.912224    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/3ff6f792-9f1b-465a-a8f1-52f1bc7e35d9-certificates\") pod \"keda-metrics-apiserver-55557df8c-nbtbz\" (UID: \"3ff6f792-9f1b-465a-a8f1-52f1bc7e35d9\") " pod="openshift-keda/keda-metrics-apiserver-55557df8c-nbtbz"
Jul 31 23:50:51.912259 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:50:51.912264    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"temp-vol\" (UniqueName: \"kubernetes.io/empty-dir/3ff6f792-9f1b-465a-a8f1-52f1bc7e35d9-temp-vol\") pod \"keda-metrics-apiserver-55557df8c-nbtbz\" (UID: \"3ff6f792-9f1b-465a-a8f1-52f1bc7e35d9\") " pod="openshift-keda/keda-metrics-apiserver-55557df8c-nbtbz"
Jul 31 23:50:51.912473 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:50:51.912285    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kqk8w\" (UniqueName: \"kubernetes.io/projected/3ff6f792-9f1b-465a-a8f1-52f1bc7e35d9-kube-api-access-kqk8w\") pod \"keda-metrics-apiserver-55557df8c-nbtbz\" (UID: \"3ff6f792-9f1b-465a-a8f1-52f1bc7e35d9\") " pod="openshift-keda/keda-metrics-apiserver-55557df8c-nbtbz"
Jul 31 23:50:51.912473 ip-10-0-136-108 kubenswrapper[2578]: E0731 23:50:51.912392    2578 projected.go:266] Couldn't get secret openshift-keda/keda-metrics-apiserver-certs: secret "keda-metrics-apiserver-certs" not found
Jul 31 23:50:51.912473 ip-10-0-136-108 kubenswrapper[2578]: E0731 23:50:51.912421    2578 projected.go:196] Error preparing data for projected volume certificates for pod openshift-keda/keda-metrics-apiserver-55557df8c-nbtbz: secret "keda-metrics-apiserver-certs" not found
Jul 31 23:50:51.912596 ip-10-0-136-108 kubenswrapper[2578]: E0731 23:50:51.912476    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3ff6f792-9f1b-465a-a8f1-52f1bc7e35d9-certificates podName:3ff6f792-9f1b-465a-a8f1-52f1bc7e35d9 nodeName:}" failed. No retries permitted until 2026-07-31 23:50:52.412461227 +0000 UTC m=+276.380211964 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "certificates" (UniqueName: "kubernetes.io/projected/3ff6f792-9f1b-465a-a8f1-52f1bc7e35d9-certificates") pod "keda-metrics-apiserver-55557df8c-nbtbz" (UID: "3ff6f792-9f1b-465a-a8f1-52f1bc7e35d9") : secret "keda-metrics-apiserver-certs" not found
Jul 31 23:50:51.912647 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:50:51.912624    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"temp-vol\" (UniqueName: \"kubernetes.io/empty-dir/3ff6f792-9f1b-465a-a8f1-52f1bc7e35d9-temp-vol\") pod \"keda-metrics-apiserver-55557df8c-nbtbz\" (UID: \"3ff6f792-9f1b-465a-a8f1-52f1bc7e35d9\") " pod="openshift-keda/keda-metrics-apiserver-55557df8c-nbtbz"
Jul 31 23:50:51.915804 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:50:51.915778    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-keda/keda-admission-56f9798bb4-x48hd"]
Jul 31 23:50:51.917850 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:50:51.917824    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gttdm_b32e6d42-e4e0-4113-b8ce-f5d5c5dc13f7/konnectivity-agent/0.log" limit=1024
Jul 31 23:50:51.919333 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:50:51.919315    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-admission-56f9798bb4-x48hd"
Jul 31 23:50:51.921123 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:50:51.921105    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-admission-webhooks-certs\""
Jul 31 23:50:51.924341 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:50:51.924321    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kqk8w\" (UniqueName: \"kubernetes.io/projected/3ff6f792-9f1b-465a-a8f1-52f1bc7e35d9-kube-api-access-kqk8w\") pod \"keda-metrics-apiserver-55557df8c-nbtbz\" (UID: \"3ff6f792-9f1b-465a-a8f1-52f1bc7e35d9\") " pod="openshift-keda/keda-metrics-apiserver-55557df8c-nbtbz"
Jul 31 23:50:51.927099 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:50:51.927068    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-admission-56f9798bb4-x48hd"]
Jul 31 23:50:52.012925 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:50:52.012897    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8txbs\" (UniqueName: \"kubernetes.io/projected/42035771-3715-4917-985b-797339e44864-kube-api-access-8txbs\") pod \"keda-admission-56f9798bb4-x48hd\" (UID: \"42035771-3715-4917-985b-797339e44864\") " pod="openshift-keda/keda-admission-56f9798bb4-x48hd"
Jul 31 23:50:52.013316 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:50:52.013293    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/42035771-3715-4917-985b-797339e44864-certificates\") pod \"keda-admission-56f9798bb4-x48hd\" (UID: \"42035771-3715-4917-985b-797339e44864\") " pod="openshift-keda/keda-admission-56f9798bb4-x48hd"
Jul 31 23:50:52.013683 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:50:52.013653    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gttdm_b32e6d42-e4e0-4113-b8ce-f5d5c5dc13f7/konnectivity-agent/0.log" limit=1024
Jul 31 23:50:52.109311 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:50:52.109219    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gttdm_b32e6d42-e4e0-4113-b8ce-f5d5c5dc13f7/konnectivity-agent/0.log" limit=1024
Jul 31 23:50:52.114401 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:50:52.114374    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/42035771-3715-4917-985b-797339e44864-certificates\") pod \"keda-admission-56f9798bb4-x48hd\" (UID: \"42035771-3715-4917-985b-797339e44864\") " pod="openshift-keda/keda-admission-56f9798bb4-x48hd"
Jul 31 23:50:52.114557 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:50:52.114502    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8txbs\" (UniqueName: \"kubernetes.io/projected/42035771-3715-4917-985b-797339e44864-kube-api-access-8txbs\") pod \"keda-admission-56f9798bb4-x48hd\" (UID: \"42035771-3715-4917-985b-797339e44864\") " pod="openshift-keda/keda-admission-56f9798bb4-x48hd"
Jul 31 23:50:52.117074 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:50:52.117047    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/42035771-3715-4917-985b-797339e44864-certificates\") pod \"keda-admission-56f9798bb4-x48hd\" (UID: \"42035771-3715-4917-985b-797339e44864\") " pod="openshift-keda/keda-admission-56f9798bb4-x48hd"
Jul 31 23:50:52.124476 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:50:52.124448    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8txbs\" (UniqueName: \"kubernetes.io/projected/42035771-3715-4917-985b-797339e44864-kube-api-access-8txbs\") pod \"keda-admission-56f9798bb4-x48hd\" (UID: \"42035771-3715-4917-985b-797339e44864\") " pod="openshift-keda/keda-admission-56f9798bb4-x48hd"
Jul 31 23:50:52.211883 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:50:52.211848    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gttdm_b32e6d42-e4e0-4113-b8ce-f5d5c5dc13f7/konnectivity-agent/0.log" limit=1024
Jul 31 23:50:52.238985 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:50:52.238947    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-admission-56f9798bb4-x48hd"
Jul 31 23:50:52.312955 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:50:52.312918    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gttdm_b32e6d42-e4e0-4113-b8ce-f5d5c5dc13f7/konnectivity-agent/0.log" limit=1024
Jul 31 23:50:52.391817 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:50:52.391689    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-admission-56f9798bb4-x48hd"]
Jul 31 23:50:52.395427 ip-10-0-136-108 kubenswrapper[2578]: W0731 23:50:52.395399    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod42035771_3715_4917_985b_797339e44864.slice/crio-60b7987bcf3115f3107217c7328060d1ddc5e463f3d26695191910f5b0bbc219 WatchSource:0}: Error finding container 60b7987bcf3115f3107217c7328060d1ddc5e463f3d26695191910f5b0bbc219: Status 404 returned error can't find the container with id 60b7987bcf3115f3107217c7328060d1ddc5e463f3d26695191910f5b0bbc219
Jul 31 23:50:52.415731 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:50:52.415648    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gttdm_b32e6d42-e4e0-4113-b8ce-f5d5c5dc13f7/konnectivity-agent/0.log" limit=1024
Jul 31 23:50:52.417413 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:50:52.417390    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/3ff6f792-9f1b-465a-a8f1-52f1bc7e35d9-certificates\") pod \"keda-metrics-apiserver-55557df8c-nbtbz\" (UID: \"3ff6f792-9f1b-465a-a8f1-52f1bc7e35d9\") " pod="openshift-keda/keda-metrics-apiserver-55557df8c-nbtbz"
Jul 31 23:50:52.419979 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:50:52.419959    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/3ff6f792-9f1b-465a-a8f1-52f1bc7e35d9-certificates\") pod \"keda-metrics-apiserver-55557df8c-nbtbz\" (UID: \"3ff6f792-9f1b-465a-a8f1-52f1bc7e35d9\") " pod="openshift-keda/keda-metrics-apiserver-55557df8c-nbtbz"
Jul 31 23:50:52.518845 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:50:52.518809    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/8b7a412c-4d6a-4619-8cb6-5bada3791b5c-certificates\") pod \"keda-operator-77f648d694-hx674\" (UID: \"8b7a412c-4d6a-4619-8cb6-5bada3791b5c\") " pod="openshift-keda/keda-operator-77f648d694-hx674"
Jul 31 23:50:52.521465 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:50:52.521433    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/8b7a412c-4d6a-4619-8cb6-5bada3791b5c-certificates\") pod \"keda-operator-77f648d694-hx674\" (UID: \"8b7a412c-4d6a-4619-8cb6-5bada3791b5c\") " pod="openshift-keda/keda-operator-77f648d694-hx674"
Jul 31 23:50:52.620230 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:50:52.620190    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gttdm_b32e6d42-e4e0-4113-b8ce-f5d5c5dc13f7/konnectivity-agent/0.log" limit=1024
Jul 31 23:50:52.628116 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:50:52.628090    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-metrics-apiserver-55557df8c-nbtbz"
Jul 31 23:50:52.663569 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:50:52.663486    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-operator-77f648d694-hx674"
Jul 31 23:50:52.720313 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:50:52.720274    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gttdm_b32e6d42-e4e0-4113-b8ce-f5d5c5dc13f7/konnectivity-agent/0.log" limit=1024
Jul 31 23:50:52.757368 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:50:52.757322    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-admission-56f9798bb4-x48hd" event={"ID":"42035771-3715-4917-985b-797339e44864","Type":"ContainerStarted","Data":"60b7987bcf3115f3107217c7328060d1ddc5e463f3d26695191910f5b0bbc219"}
Jul 31 23:50:52.770823 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:50:52.770797    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-metrics-apiserver-55557df8c-nbtbz"]
Jul 31 23:50:52.773805 ip-10-0-136-108 kubenswrapper[2578]: W0731 23:50:52.773771    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3ff6f792_9f1b_465a_a8f1_52f1bc7e35d9.slice/crio-85014678628e71796baad88778ffddaed8deb7cde3631fdb5ffca152e3879e6e WatchSource:0}: Error finding container 85014678628e71796baad88778ffddaed8deb7cde3631fdb5ffca152e3879e6e: Status 404 returned error can't find the container with id 85014678628e71796baad88778ffddaed8deb7cde3631fdb5ffca152e3879e6e
Jul 31 23:50:52.808128 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:50:52.807955    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-operator-77f648d694-hx674"]
Jul 31 23:50:52.810675 ip-10-0-136-108 kubenswrapper[2578]: W0731 23:50:52.810647    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8b7a412c_4d6a_4619_8cb6_5bada3791b5c.slice/crio-ddb10544e5aea98f14070e27288f1e55df630710e3de50a149deec8abde27e17 WatchSource:0}: Error finding container ddb10544e5aea98f14070e27288f1e55df630710e3de50a149deec8abde27e17: Status 404 returned error can't find the container with id ddb10544e5aea98f14070e27288f1e55df630710e3de50a149deec8abde27e17
Jul 31 23:50:53.766425 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:50:53.766388    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-metrics-apiserver-55557df8c-nbtbz" event={"ID":"3ff6f792-9f1b-465a-a8f1-52f1bc7e35d9","Type":"ContainerStarted","Data":"85014678628e71796baad88778ffddaed8deb7cde3631fdb5ffca152e3879e6e"}
Jul 31 23:50:53.768129 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:50:53.768089    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-operator-77f648d694-hx674" event={"ID":"8b7a412c-4d6a-4619-8cb6-5bada3791b5c","Type":"ContainerStarted","Data":"ddb10544e5aea98f14070e27288f1e55df630710e3de50a149deec8abde27e17"}
Jul 31 23:50:54.775298 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:50:54.775257    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-admission-56f9798bb4-x48hd" event={"ID":"42035771-3715-4917-985b-797339e44864","Type":"ContainerStarted","Data":"8fefa6bc1bfd16a69c547cbfbba2a0d76eddf67ca2605660b220218cb2237605"}
Jul 31 23:50:54.775757 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:50:54.775385    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-keda/keda-admission-56f9798bb4-x48hd"
Jul 31 23:50:54.795447 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:50:54.795375    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-keda/keda-admission-56f9798bb4-x48hd" podStartSLOduration=2.163595798 podStartE2EDuration="3.795351853s" totalImagesPullingTime="1.631756055s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 23:50:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 23:50:54.792449978 +0000 UTC m=+278.760200740" watchObservedRunningTime="2026-07-31 23:50:54.795351853 +0000 UTC m=+278.763102613"
Jul 31 23:50:56.786188 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:50:56.786133    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-operator-77f648d694-hx674" event={"ID":"8b7a412c-4d6a-4619-8cb6-5bada3791b5c","Type":"ContainerStarted","Data":"03ba863b5f286b88fe84da4085beafb4c95b8c7e89b466eb1696692854ee7830"}
Jul 31 23:50:56.786640 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:50:56.786236    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-keda/keda-operator-77f648d694-hx674"
Jul 31 23:50:56.787639 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:50:56.787619    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-metrics-apiserver-55557df8c-nbtbz" event={"ID":"3ff6f792-9f1b-465a-a8f1-52f1bc7e35d9","Type":"ContainerStarted","Data":"12ceb153061c870dd77a78a9f7390feb7e44c924ea6fdaf0a082ef4a15713a31"}
Jul 31 23:50:56.787709 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:50:56.787663    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-keda/keda-metrics-apiserver-55557df8c-nbtbz"
Jul 31 23:50:56.801743 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:50:56.801691    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-keda/keda-operator-77f648d694-hx674" podStartSLOduration=3.264616798 podStartE2EDuration="6.80167813s" totalImagesPullingTime="3.537061332s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 23:50:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 23:50:56.801122556 +0000 UTC m=+280.768873314" watchObservedRunningTime="2026-07-31 23:50:56.80167813 +0000 UTC m=+280.769428888"
Jul 31 23:50:56.815925 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:50:56.815870    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-keda/keda-metrics-apiserver-55557df8c-nbtbz" podStartSLOduration=2.24732728 podStartE2EDuration="5.815852779s" totalImagesPullingTime="3.568525499s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 23:50:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 23:50:56.8148532 +0000 UTC m=+280.782603958" watchObservedRunningTime="2026-07-31 23:50:56.815852779 +0000 UTC m=+280.783603544"
Jul 31 23:51:07.796806 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:51:07.796779    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-keda/keda-metrics-apiserver-55557df8c-nbtbz"
Jul 31 23:51:11.753073 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:51:11.753040    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-keda/custom-metrics-autoscaler-operator-6969cb7dfb-f8sgn"
Jul 31 23:51:15.781455 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:51:15.781426    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-keda/keda-admission-56f9798bb4-x48hd"
Jul 31 23:51:16.362728 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:51:16.362699    2578 kubelet.go:1648] "Image garbage collection succeeded"
Jul 31 23:51:17.794614 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:51:17.794585    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-keda/keda-operator-77f648d694-hx674"
Jul 31 23:51:44.865112 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:51:44.865070    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gttdm_b32e6d42-e4e0-4113-b8ce-f5d5c5dc13f7/konnectivity-agent/0.log" limit=1024
Jul 31 23:51:46.128863 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:51:46.128820    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gttdm_b32e6d42-e4e0-4113-b8ce-f5d5c5dc13f7/konnectivity-agent/0.log" limit=1024
Jul 31 23:52:13.026570 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:52:13.026475    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gttdm_b32e6d42-e4e0-4113-b8ce-f5d5c5dc13f7/konnectivity-agent/0.log" limit=1024
Jul 31 23:52:13.115887 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:52:13.115849    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/kserve-controller-manager-674d959fb8-rsmsd"]
Jul 31 23:52:13.119265 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:52:13.119243    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-674d959fb8-rsmsd"
Jul 31 23:52:13.122954 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:52:13.122920    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"kserve-webhook-server-cert\""
Jul 31 23:52:13.123079 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:52:13.123016    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"kube-root-ca.crt\""
Jul 31 23:52:13.123360 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:52:13.123341    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"openshift-service-ca.crt\""
Jul 31 23:52:13.123476 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:52:13.123373    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"kserve-controller-manager-dockercfg-wtcd5\""
Jul 31 23:52:13.133453 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:52:13.133430    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/llmisvc-controller-manager-654c557856-8d8w7"]
Jul 31 23:52:13.135767 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:52:13.135742    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gttdm_b32e6d42-e4e0-4113-b8ce-f5d5c5dc13f7/konnectivity-agent/0.log" limit=1024
Jul 31 23:52:13.137122 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:52:13.137101    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-674d959fb8-rsmsd"]
Jul 31 23:52:13.137327 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:52:13.137203    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/llmisvc-controller-manager-654c557856-8d8w7"
Jul 31 23:52:13.139328 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:52:13.139307    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"llmisvc-webhook-server-cert\""
Jul 31 23:52:13.139450 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:52:13.139436    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"llmisvc-controller-manager-dockercfg-h5wjb\""
Jul 31 23:52:13.152755 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:52:13.152730    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/llmisvc-controller-manager-654c557856-8d8w7"]
Jul 31 23:52:13.228780 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:52:13.228740    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gttdm_b32e6d42-e4e0-4113-b8ce-f5d5c5dc13f7/konnectivity-agent/0.log" limit=1024
Jul 31 23:52:13.281028 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:52:13.280955    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/74f227fb-7bdc-4402-9140-a7704e0afa63-cert\") pod \"llmisvc-controller-manager-654c557856-8d8w7\" (UID: \"74f227fb-7bdc-4402-9140-a7704e0afa63\") " pod="kserve/llmisvc-controller-manager-654c557856-8d8w7"
Jul 31 23:52:13.281028 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:52:13.280998    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7kcxr\" (UniqueName: \"kubernetes.io/projected/f27faa5d-7caa-4b79-a3ec-6bed34a3fc4a-kube-api-access-7kcxr\") pod \"kserve-controller-manager-674d959fb8-rsmsd\" (UID: \"f27faa5d-7caa-4b79-a3ec-6bed34a3fc4a\") " pod="kserve/kserve-controller-manager-674d959fb8-rsmsd"
Jul 31 23:52:13.281197 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:52:13.281033    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f27faa5d-7caa-4b79-a3ec-6bed34a3fc4a-cert\") pod \"kserve-controller-manager-674d959fb8-rsmsd\" (UID: \"f27faa5d-7caa-4b79-a3ec-6bed34a3fc4a\") " pod="kserve/kserve-controller-manager-674d959fb8-rsmsd"
Jul 31 23:52:13.281197 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:52:13.281050    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m5rvz\" (UniqueName: \"kubernetes.io/projected/74f227fb-7bdc-4402-9140-a7704e0afa63-kube-api-access-m5rvz\") pod \"llmisvc-controller-manager-654c557856-8d8w7\" (UID: \"74f227fb-7bdc-4402-9140-a7704e0afa63\") " pod="kserve/llmisvc-controller-manager-654c557856-8d8w7"
Jul 31 23:52:13.325543 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:52:13.325503    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gttdm_b32e6d42-e4e0-4113-b8ce-f5d5c5dc13f7/konnectivity-agent/0.log" limit=1024
Jul 31 23:52:13.382062 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:52:13.382034    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7kcxr\" (UniqueName: \"kubernetes.io/projected/f27faa5d-7caa-4b79-a3ec-6bed34a3fc4a-kube-api-access-7kcxr\") pod \"kserve-controller-manager-674d959fb8-rsmsd\" (UID: \"f27faa5d-7caa-4b79-a3ec-6bed34a3fc4a\") " pod="kserve/kserve-controller-manager-674d959fb8-rsmsd"
Jul 31 23:52:13.382250 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:52:13.382087    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f27faa5d-7caa-4b79-a3ec-6bed34a3fc4a-cert\") pod \"kserve-controller-manager-674d959fb8-rsmsd\" (UID: \"f27faa5d-7caa-4b79-a3ec-6bed34a3fc4a\") " pod="kserve/kserve-controller-manager-674d959fb8-rsmsd"
Jul 31 23:52:13.382250 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:52:13.382208    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m5rvz\" (UniqueName: \"kubernetes.io/projected/74f227fb-7bdc-4402-9140-a7704e0afa63-kube-api-access-m5rvz\") pod \"llmisvc-controller-manager-654c557856-8d8w7\" (UID: \"74f227fb-7bdc-4402-9140-a7704e0afa63\") " pod="kserve/llmisvc-controller-manager-654c557856-8d8w7"
Jul 31 23:52:13.382351 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:52:13.382331    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/74f227fb-7bdc-4402-9140-a7704e0afa63-cert\") pod \"llmisvc-controller-manager-654c557856-8d8w7\" (UID: \"74f227fb-7bdc-4402-9140-a7704e0afa63\") " pod="kserve/llmisvc-controller-manager-654c557856-8d8w7"
Jul 31 23:52:13.384692 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:52:13.384660    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f27faa5d-7caa-4b79-a3ec-6bed34a3fc4a-cert\") pod \"kserve-controller-manager-674d959fb8-rsmsd\" (UID: \"f27faa5d-7caa-4b79-a3ec-6bed34a3fc4a\") " pod="kserve/kserve-controller-manager-674d959fb8-rsmsd"
Jul 31 23:52:13.384800 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:52:13.384666    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/74f227fb-7bdc-4402-9140-a7704e0afa63-cert\") pod \"llmisvc-controller-manager-654c557856-8d8w7\" (UID: \"74f227fb-7bdc-4402-9140-a7704e0afa63\") " pod="kserve/llmisvc-controller-manager-654c557856-8d8w7"
Jul 31 23:52:13.391984 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:52:13.391956    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7kcxr\" (UniqueName: \"kubernetes.io/projected/f27faa5d-7caa-4b79-a3ec-6bed34a3fc4a-kube-api-access-7kcxr\") pod \"kserve-controller-manager-674d959fb8-rsmsd\" (UID: \"f27faa5d-7caa-4b79-a3ec-6bed34a3fc4a\") " pod="kserve/kserve-controller-manager-674d959fb8-rsmsd"
Jul 31 23:52:13.392302 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:52:13.392284    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m5rvz\" (UniqueName: \"kubernetes.io/projected/74f227fb-7bdc-4402-9140-a7704e0afa63-kube-api-access-m5rvz\") pod \"llmisvc-controller-manager-654c557856-8d8w7\" (UID: \"74f227fb-7bdc-4402-9140-a7704e0afa63\") " pod="kserve/llmisvc-controller-manager-654c557856-8d8w7"
Jul 31 23:52:13.429092 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:52:13.429066    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-674d959fb8-rsmsd"
Jul 31 23:52:13.430158 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:52:13.430128    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gttdm_b32e6d42-e4e0-4113-b8ce-f5d5c5dc13f7/konnectivity-agent/0.log" limit=1024
Jul 31 23:52:13.447691 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:52:13.447664    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/llmisvc-controller-manager-654c557856-8d8w7"
Jul 31 23:52:13.598102 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:52:13.597937    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-674d959fb8-rsmsd"]
Jul 31 23:52:13.600884 ip-10-0-136-108 kubenswrapper[2578]: W0731 23:52:13.600860    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf27faa5d_7caa_4b79_a3ec_6bed34a3fc4a.slice/crio-b483b4a2539a21eb046c9a894cb83da6a4570086da83652063f37a7cce6b5aa8 WatchSource:0}: Error finding container b483b4a2539a21eb046c9a894cb83da6a4570086da83652063f37a7cce6b5aa8: Status 404 returned error can't find the container with id b483b4a2539a21eb046c9a894cb83da6a4570086da83652063f37a7cce6b5aa8
Jul 31 23:52:13.602202 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:52:13.602184    2578 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 31 23:52:13.613521 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:52:13.613496    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/llmisvc-controller-manager-654c557856-8d8w7"]
Jul 31 23:52:13.616664 ip-10-0-136-108 kubenswrapper[2578]: W0731 23:52:13.616641    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod74f227fb_7bdc_4402_9140_a7704e0afa63.slice/crio-42ccc0ea42f7cebaa1c0cc5915550edea8cf44cce15d40b893cf6741b1219bce WatchSource:0}: Error finding container 42ccc0ea42f7cebaa1c0cc5915550edea8cf44cce15d40b893cf6741b1219bce: Status 404 returned error can't find the container with id 42ccc0ea42f7cebaa1c0cc5915550edea8cf44cce15d40b893cf6741b1219bce
Jul 31 23:52:13.781455 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:52:13.781422    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gttdm_b32e6d42-e4e0-4113-b8ce-f5d5c5dc13f7/konnectivity-agent/0.log" limit=1024
Jul 31 23:52:13.882723 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:52:13.882628    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gttdm_b32e6d42-e4e0-4113-b8ce-f5d5c5dc13f7/konnectivity-agent/0.log" limit=1024
Jul 31 23:52:13.942706 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:52:13.942677    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gttdm_b32e6d42-e4e0-4113-b8ce-f5d5c5dc13f7/konnectivity-agent/0.log" limit=1024
Jul 31 23:52:14.132560 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:52:14.132517    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-674d959fb8-rsmsd" event={"ID":"f27faa5d-7caa-4b79-a3ec-6bed34a3fc4a","Type":"ContainerStarted","Data":"b483b4a2539a21eb046c9a894cb83da6a4570086da83652063f37a7cce6b5aa8"}
Jul 31 23:52:14.135391 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:52:14.135304    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/llmisvc-controller-manager-654c557856-8d8w7" event={"ID":"74f227fb-7bdc-4402-9140-a7704e0afa63","Type":"ContainerStarted","Data":"42ccc0ea42f7cebaa1c0cc5915550edea8cf44cce15d40b893cf6741b1219bce"}
Jul 31 23:52:17.150085 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:52:17.149998    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-674d959fb8-rsmsd" event={"ID":"f27faa5d-7caa-4b79-a3ec-6bed34a3fc4a","Type":"ContainerStarted","Data":"b039bb5fe42ded3ffb86ae955326dc3c2e6b425e8a874d5e01c58083d1e1373e"}
Jul 31 23:52:17.150507 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:52:17.150199    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/kserve-controller-manager-674d959fb8-rsmsd"
Jul 31 23:52:17.151546 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:52:17.151521    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/llmisvc-controller-manager-654c557856-8d8w7" event={"ID":"74f227fb-7bdc-4402-9140-a7704e0afa63","Type":"ContainerStarted","Data":"1953abbcb4969720ef2d9fcf97cc44732da5181183fbfdd034052be2964429e7"}
Jul 31 23:52:17.151683 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:52:17.151665    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/llmisvc-controller-manager-654c557856-8d8w7"
Jul 31 23:52:17.177064 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:52:17.177012    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/kserve-controller-manager-674d959fb8-rsmsd" podStartSLOduration=0.97022765 podStartE2EDuration="4.176994105s" totalImagesPullingTime="3.206766455s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 23:52:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 23:52:17.174625673 +0000 UTC m=+361.142376431" watchObservedRunningTime="2026-07-31 23:52:17.176994105 +0000 UTC m=+361.144744863"
Jul 31 23:52:25.827483 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:52:25.827419    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/llmisvc-controller-manager-654c557856-8d8w7" podStartSLOduration=9.636357466 podStartE2EDuration="12.827403066s" totalImagesPullingTime="3.1910456s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 23:52:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 23:52:17.20020813 +0000 UTC m=+361.167958890" watchObservedRunningTime="2026-07-31 23:52:25.827403066 +0000 UTC m=+369.795153824"
Jul 31 23:52:25.828000 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:52:25.827985    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-66d866d495-nhllr"]
Jul 31 23:52:25.831460 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:52:25.831439    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-66d866d495-nhllr"
Jul 31 23:52:25.846136 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:52:25.846108    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-66d866d495-nhllr"]
Jul 31 23:52:25.891545 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:52:25.891505    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gttdm_b32e6d42-e4e0-4113-b8ce-f5d5c5dc13f7/konnectivity-agent/0.log" limit=1024
Jul 31 23:52:25.896754 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:52:25.896728    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/2e86968e-4498-462f-baa8-da63b80d8ae7-oauth-serving-cert\") pod \"console-66d866d495-nhllr\" (UID: \"2e86968e-4498-462f-baa8-da63b80d8ae7\") " pod="openshift-console/console-66d866d495-nhllr"
Jul 31 23:52:25.896903 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:52:25.896761    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/2e86968e-4498-462f-baa8-da63b80d8ae7-console-oauth-config\") pod \"console-66d866d495-nhllr\" (UID: \"2e86968e-4498-462f-baa8-da63b80d8ae7\") " pod="openshift-console/console-66d866d495-nhllr"
Jul 31 23:52:25.896903 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:52:25.896822    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gmwfd\" (UniqueName: \"kubernetes.io/projected/2e86968e-4498-462f-baa8-da63b80d8ae7-kube-api-access-gmwfd\") pod \"console-66d866d495-nhllr\" (UID: \"2e86968e-4498-462f-baa8-da63b80d8ae7\") " pod="openshift-console/console-66d866d495-nhllr"
Jul 31 23:52:25.896903 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:52:25.896856    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/2e86968e-4498-462f-baa8-da63b80d8ae7-console-config\") pod \"console-66d866d495-nhllr\" (UID: \"2e86968e-4498-462f-baa8-da63b80d8ae7\") " pod="openshift-console/console-66d866d495-nhllr"
Jul 31 23:52:25.896903 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:52:25.896874    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/2e86968e-4498-462f-baa8-da63b80d8ae7-service-ca\") pod \"console-66d866d495-nhllr\" (UID: \"2e86968e-4498-462f-baa8-da63b80d8ae7\") " pod="openshift-console/console-66d866d495-nhllr"
Jul 31 23:52:25.896903 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:52:25.896895    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/2e86968e-4498-462f-baa8-da63b80d8ae7-console-serving-cert\") pod \"console-66d866d495-nhllr\" (UID: \"2e86968e-4498-462f-baa8-da63b80d8ae7\") " pod="openshift-console/console-66d866d495-nhllr"
Jul 31 23:52:25.897116 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:52:25.896914    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2e86968e-4498-462f-baa8-da63b80d8ae7-trusted-ca-bundle\") pod \"console-66d866d495-nhllr\" (UID: \"2e86968e-4498-462f-baa8-da63b80d8ae7\") " pod="openshift-console/console-66d866d495-nhllr"
Jul 31 23:52:25.986117 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:52:25.986077    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gttdm_b32e6d42-e4e0-4113-b8ce-f5d5c5dc13f7/konnectivity-agent/0.log" limit=1024
Jul 31 23:52:25.998228 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:52:25.998200    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gmwfd\" (UniqueName: \"kubernetes.io/projected/2e86968e-4498-462f-baa8-da63b80d8ae7-kube-api-access-gmwfd\") pod \"console-66d866d495-nhllr\" (UID: \"2e86968e-4498-462f-baa8-da63b80d8ae7\") " pod="openshift-console/console-66d866d495-nhllr"
Jul 31 23:52:25.998334 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:52:25.998246    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/2e86968e-4498-462f-baa8-da63b80d8ae7-console-config\") pod \"console-66d866d495-nhllr\" (UID: \"2e86968e-4498-462f-baa8-da63b80d8ae7\") " pod="openshift-console/console-66d866d495-nhllr"
Jul 31 23:52:25.998334 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:52:25.998269    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/2e86968e-4498-462f-baa8-da63b80d8ae7-service-ca\") pod \"console-66d866d495-nhllr\" (UID: \"2e86968e-4498-462f-baa8-da63b80d8ae7\") " pod="openshift-console/console-66d866d495-nhllr"
Jul 31 23:52:25.998334 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:52:25.998314    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/2e86968e-4498-462f-baa8-da63b80d8ae7-console-serving-cert\") pod \"console-66d866d495-nhllr\" (UID: \"2e86968e-4498-462f-baa8-da63b80d8ae7\") " pod="openshift-console/console-66d866d495-nhllr"
Jul 31 23:52:25.998482 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:52:25.998338    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2e86968e-4498-462f-baa8-da63b80d8ae7-trusted-ca-bundle\") pod \"console-66d866d495-nhllr\" (UID: \"2e86968e-4498-462f-baa8-da63b80d8ae7\") " pod="openshift-console/console-66d866d495-nhllr"
Jul 31 23:52:25.998482 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:52:25.998404    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/2e86968e-4498-462f-baa8-da63b80d8ae7-oauth-serving-cert\") pod \"console-66d866d495-nhllr\" (UID: \"2e86968e-4498-462f-baa8-da63b80d8ae7\") " pod="openshift-console/console-66d866d495-nhllr"
Jul 31 23:52:25.998482 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:52:25.998428    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/2e86968e-4498-462f-baa8-da63b80d8ae7-console-oauth-config\") pod \"console-66d866d495-nhllr\" (UID: \"2e86968e-4498-462f-baa8-da63b80d8ae7\") " pod="openshift-console/console-66d866d495-nhllr"
Jul 31 23:52:25.999380 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:52:25.999341    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/2e86968e-4498-462f-baa8-da63b80d8ae7-console-config\") pod \"console-66d866d495-nhllr\" (UID: \"2e86968e-4498-462f-baa8-da63b80d8ae7\") " pod="openshift-console/console-66d866d495-nhllr"
Jul 31 23:52:26.004917 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:52:25.999370    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/2e86968e-4498-462f-baa8-da63b80d8ae7-service-ca\") pod \"console-66d866d495-nhllr\" (UID: \"2e86968e-4498-462f-baa8-da63b80d8ae7\") " pod="openshift-console/console-66d866d495-nhllr"
Jul 31 23:52:26.004917 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:52:25.999601    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/2e86968e-4498-462f-baa8-da63b80d8ae7-oauth-serving-cert\") pod \"console-66d866d495-nhllr\" (UID: \"2e86968e-4498-462f-baa8-da63b80d8ae7\") " pod="openshift-console/console-66d866d495-nhllr"
Jul 31 23:52:26.004917 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:52:25.999688    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2e86968e-4498-462f-baa8-da63b80d8ae7-trusted-ca-bundle\") pod \"console-66d866d495-nhllr\" (UID: \"2e86968e-4498-462f-baa8-da63b80d8ae7\") " pod="openshift-console/console-66d866d495-nhllr"
Jul 31 23:52:26.005350 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:52:26.005332    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/2e86968e-4498-462f-baa8-da63b80d8ae7-console-serving-cert\") pod \"console-66d866d495-nhllr\" (UID: \"2e86968e-4498-462f-baa8-da63b80d8ae7\") " pod="openshift-console/console-66d866d495-nhllr"
Jul 31 23:52:26.006934 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:52:26.006906    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/2e86968e-4498-462f-baa8-da63b80d8ae7-console-oauth-config\") pod \"console-66d866d495-nhllr\" (UID: \"2e86968e-4498-462f-baa8-da63b80d8ae7\") " pod="openshift-console/console-66d866d495-nhllr"
Jul 31 23:52:26.007387 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:52:26.007367    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gmwfd\" (UniqueName: \"kubernetes.io/projected/2e86968e-4498-462f-baa8-da63b80d8ae7-kube-api-access-gmwfd\") pod \"console-66d866d495-nhllr\" (UID: \"2e86968e-4498-462f-baa8-da63b80d8ae7\") " pod="openshift-console/console-66d866d495-nhllr"
Jul 31 23:52:26.086117 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:52:26.086030    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gttdm_b32e6d42-e4e0-4113-b8ce-f5d5c5dc13f7/konnectivity-agent/0.log" limit=1024
Jul 31 23:52:26.141238 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:52:26.141133    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-66d866d495-nhllr"
Jul 31 23:52:26.284782 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:52:26.284639    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-66d866d495-nhllr"]
Jul 31 23:52:26.287692 ip-10-0-136-108 kubenswrapper[2578]: W0731 23:52:26.287666    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2e86968e_4498_462f_baa8_da63b80d8ae7.slice/crio-03407a9a375f54225c95a86701c2967ec0cf6982ca50d40220b0b4d83eb69e1d WatchSource:0}: Error finding container 03407a9a375f54225c95a86701c2967ec0cf6982ca50d40220b0b4d83eb69e1d: Status 404 returned error can't find the container with id 03407a9a375f54225c95a86701c2967ec0cf6982ca50d40220b0b4d83eb69e1d
Jul 31 23:52:26.637758 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:52:26.637670    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gttdm_b32e6d42-e4e0-4113-b8ce-f5d5c5dc13f7/konnectivity-agent/0.log" limit=1024
Jul 31 23:52:26.737627 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:52:26.737593    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gttdm_b32e6d42-e4e0-4113-b8ce-f5d5c5dc13f7/konnectivity-agent/0.log" limit=1024
Jul 31 23:52:27.196329 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:52:27.196295    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-66d866d495-nhllr" event={"ID":"2e86968e-4498-462f-baa8-da63b80d8ae7","Type":"ContainerStarted","Data":"85bc1b61b7965401d0a6e1301a6a213ac87434470957c3482652a88b8d9eb608"}
Jul 31 23:52:27.196711 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:52:27.196343    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-66d866d495-nhllr" event={"ID":"2e86968e-4498-462f-baa8-da63b80d8ae7","Type":"ContainerStarted","Data":"03407a9a375f54225c95a86701c2967ec0cf6982ca50d40220b0b4d83eb69e1d"}
Jul 31 23:52:27.215570 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:52:27.215514    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-66d866d495-nhllr" podStartSLOduration=2.215495009 podStartE2EDuration="2.215495009s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 23:52:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 23:52:27.214639402 +0000 UTC m=+371.182390160" watchObservedRunningTime="2026-07-31 23:52:27.215495009 +0000 UTC m=+371.183245766"
Jul 31 23:52:36.141919 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:52:36.141876    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-66d866d495-nhllr"
Jul 31 23:52:36.142334 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:52:36.141944    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-66d866d495-nhllr"
Jul 31 23:52:36.146670 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:52:36.146648    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-66d866d495-nhllr"
Jul 31 23:52:36.238655 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:52:36.238629    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-66d866d495-nhllr"
Jul 31 23:52:36.326720 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:52:36.326689    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5fc6b5dfb9-j4bdc"]
Jul 31 23:52:36.443608 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:52:36.443518    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gttdm_b32e6d42-e4e0-4113-b8ce-f5d5c5dc13f7/konnectivity-agent/0.log" limit=1024
Jul 31 23:52:36.543212 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:52:36.543150    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gttdm_b32e6d42-e4e0-4113-b8ce-f5d5c5dc13f7/konnectivity-agent/0.log" limit=1024
Jul 31 23:52:36.602375 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:52:36.602329    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gttdm_b32e6d42-e4e0-4113-b8ce-f5d5c5dc13f7/konnectivity-agent/0.log" limit=1024
Jul 31 23:52:48.158810 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:52:48.158768    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/llmisvc-controller-manager-654c557856-8d8w7"
Jul 31 23:52:48.161685 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:52:48.161658    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/kserve-controller-manager-674d959fb8-rsmsd"
Jul 31 23:52:56.311847 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:52:56.311808    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve/kserve-controller-manager-674d959fb8-rsmsd"]
Jul 31 23:52:56.312475 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:52:56.312017    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve/kserve-controller-manager-674d959fb8-rsmsd" podUID="f27faa5d-7caa-4b79-a3ec-6bed34a3fc4a" containerName="manager" containerID="cri-o://b039bb5fe42ded3ffb86ae955326dc3c2e6b425e8a874d5e01c58083d1e1373e" gracePeriod=10
Jul 31 23:52:56.333975 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:52:56.333950    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/kserve-controller-manager-674d959fb8-w5sb4"]
Jul 31 23:52:56.337293 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:52:56.337278    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-674d959fb8-w5sb4"
Jul 31 23:52:56.344133 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:52:56.344106    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-674d959fb8-w5sb4"]
Jul 31 23:52:56.460708 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:52:56.460673    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/685ff127-eee9-4acf-bb9b-9a5607dc5d20-cert\") pod \"kserve-controller-manager-674d959fb8-w5sb4\" (UID: \"685ff127-eee9-4acf-bb9b-9a5607dc5d20\") " pod="kserve/kserve-controller-manager-674d959fb8-w5sb4"
Jul 31 23:52:56.460708 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:52:56.460711    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mb5kc\" (UniqueName: \"kubernetes.io/projected/685ff127-eee9-4acf-bb9b-9a5607dc5d20-kube-api-access-mb5kc\") pod \"kserve-controller-manager-674d959fb8-w5sb4\" (UID: \"685ff127-eee9-4acf-bb9b-9a5607dc5d20\") " pod="kserve/kserve-controller-manager-674d959fb8-w5sb4"
Jul 31 23:52:56.557453 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:52:56.557427    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-674d959fb8-rsmsd"
Jul 31 23:52:56.561098 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:52:56.561072    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/685ff127-eee9-4acf-bb9b-9a5607dc5d20-cert\") pod \"kserve-controller-manager-674d959fb8-w5sb4\" (UID: \"685ff127-eee9-4acf-bb9b-9a5607dc5d20\") " pod="kserve/kserve-controller-manager-674d959fb8-w5sb4"
Jul 31 23:52:56.561268 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:52:56.561102    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mb5kc\" (UniqueName: \"kubernetes.io/projected/685ff127-eee9-4acf-bb9b-9a5607dc5d20-kube-api-access-mb5kc\") pod \"kserve-controller-manager-674d959fb8-w5sb4\" (UID: \"685ff127-eee9-4acf-bb9b-9a5607dc5d20\") " pod="kserve/kserve-controller-manager-674d959fb8-w5sb4"
Jul 31 23:52:56.563394 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:52:56.563341    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/685ff127-eee9-4acf-bb9b-9a5607dc5d20-cert\") pod \"kserve-controller-manager-674d959fb8-w5sb4\" (UID: \"685ff127-eee9-4acf-bb9b-9a5607dc5d20\") " pod="kserve/kserve-controller-manager-674d959fb8-w5sb4"
Jul 31 23:52:56.567848 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:52:56.567822    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mb5kc\" (UniqueName: \"kubernetes.io/projected/685ff127-eee9-4acf-bb9b-9a5607dc5d20-kube-api-access-mb5kc\") pod \"kserve-controller-manager-674d959fb8-w5sb4\" (UID: \"685ff127-eee9-4acf-bb9b-9a5607dc5d20\") " pod="kserve/kserve-controller-manager-674d959fb8-w5sb4"
Jul 31 23:52:56.661758 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:52:56.661719    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f27faa5d-7caa-4b79-a3ec-6bed34a3fc4a-cert\") pod \"f27faa5d-7caa-4b79-a3ec-6bed34a3fc4a\" (UID: \"f27faa5d-7caa-4b79-a3ec-6bed34a3fc4a\") "
Jul 31 23:52:56.661963 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:52:56.661771    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7kcxr\" (UniqueName: \"kubernetes.io/projected/f27faa5d-7caa-4b79-a3ec-6bed34a3fc4a-kube-api-access-7kcxr\") pod \"f27faa5d-7caa-4b79-a3ec-6bed34a3fc4a\" (UID: \"f27faa5d-7caa-4b79-a3ec-6bed34a3fc4a\") "
Jul 31 23:52:56.664058 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:52:56.664032    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f27faa5d-7caa-4b79-a3ec-6bed34a3fc4a-cert" (OuterVolumeSpecName: "cert") pod "f27faa5d-7caa-4b79-a3ec-6bed34a3fc4a" (UID: "f27faa5d-7caa-4b79-a3ec-6bed34a3fc4a"). InnerVolumeSpecName "cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 31 23:52:56.664187 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:52:56.664034    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f27faa5d-7caa-4b79-a3ec-6bed34a3fc4a-kube-api-access-7kcxr" (OuterVolumeSpecName: "kube-api-access-7kcxr") pod "f27faa5d-7caa-4b79-a3ec-6bed34a3fc4a" (UID: "f27faa5d-7caa-4b79-a3ec-6bed34a3fc4a"). InnerVolumeSpecName "kube-api-access-7kcxr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 31 23:52:56.688786 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:52:56.688754    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-674d959fb8-w5sb4"
Jul 31 23:52:56.762537 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:52:56.762500    2578 reconciler_common.go:299] "Volume detached for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f27faa5d-7caa-4b79-a3ec-6bed34a3fc4a-cert\") on node \"ip-10-0-136-108.ec2.internal\" DevicePath \"\""
Jul 31 23:52:56.767813 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:52:56.762989    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7kcxr\" (UniqueName: \"kubernetes.io/projected/f27faa5d-7caa-4b79-a3ec-6bed34a3fc4a-kube-api-access-7kcxr\") on node \"ip-10-0-136-108.ec2.internal\" DevicePath \"\""
Jul 31 23:52:56.823939 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:52:56.823846    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-674d959fb8-w5sb4"]
Jul 31 23:52:56.826747 ip-10-0-136-108 kubenswrapper[2578]: W0731 23:52:56.826716    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod685ff127_eee9_4acf_bb9b_9a5607dc5d20.slice/crio-f8d7e52fe04597bb18ec1b5b76a917c5953eaa55c595e7365bcc2f889cc90032 WatchSource:0}: Error finding container f8d7e52fe04597bb18ec1b5b76a917c5953eaa55c595e7365bcc2f889cc90032: Status 404 returned error can't find the container with id f8d7e52fe04597bb18ec1b5b76a917c5953eaa55c595e7365bcc2f889cc90032
Jul 31 23:52:57.321416 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:52:57.321379    2578 generic.go:356] "Generic (PLEG): container finished" podID="f27faa5d-7caa-4b79-a3ec-6bed34a3fc4a" containerID="b039bb5fe42ded3ffb86ae955326dc3c2e6b425e8a874d5e01c58083d1e1373e" exitCode=0
Jul 31 23:52:57.321858 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:52:57.321444    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-674d959fb8-rsmsd"
Jul 31 23:52:57.321858 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:52:57.321467    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-674d959fb8-rsmsd" event={"ID":"f27faa5d-7caa-4b79-a3ec-6bed34a3fc4a","Type":"ContainerDied","Data":"b039bb5fe42ded3ffb86ae955326dc3c2e6b425e8a874d5e01c58083d1e1373e"}
Jul 31 23:52:57.321858 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:52:57.321535    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-674d959fb8-rsmsd" event={"ID":"f27faa5d-7caa-4b79-a3ec-6bed34a3fc4a","Type":"ContainerDied","Data":"b483b4a2539a21eb046c9a894cb83da6a4570086da83652063f37a7cce6b5aa8"}
Jul 31 23:52:57.321858 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:52:57.321572    2578 scope.go:117] "RemoveContainer" containerID="b039bb5fe42ded3ffb86ae955326dc3c2e6b425e8a874d5e01c58083d1e1373e"
Jul 31 23:52:57.323201 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:52:57.323159    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-674d959fb8-w5sb4" event={"ID":"685ff127-eee9-4acf-bb9b-9a5607dc5d20","Type":"ContainerStarted","Data":"5e56b2317b1da2d15ce42e4e60bb9a20be5b11ec27b6f4c541d132ed4b6dc547"}
Jul 31 23:52:57.323296 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:52:57.323224    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-674d959fb8-w5sb4" event={"ID":"685ff127-eee9-4acf-bb9b-9a5607dc5d20","Type":"ContainerStarted","Data":"f8d7e52fe04597bb18ec1b5b76a917c5953eaa55c595e7365bcc2f889cc90032"}
Jul 31 23:52:57.323520 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:52:57.323323    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/kserve-controller-manager-674d959fb8-w5sb4"
Jul 31 23:52:57.340605 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:52:57.340551    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/kserve-controller-manager-674d959fb8-w5sb4" podStartSLOduration=0.952150788 podStartE2EDuration="1.340530163s" totalImagesPullingTime="388.379375ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 23:52:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 23:52:57.338516265 +0000 UTC m=+401.306267021" watchObservedRunningTime="2026-07-31 23:52:57.340530163 +0000 UTC m=+401.308280922"
Jul 31 23:52:57.343117 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:52:57.343091    2578 scope.go:117] "RemoveContainer" containerID="b039bb5fe42ded3ffb86ae955326dc3c2e6b425e8a874d5e01c58083d1e1373e"
Jul 31 23:52:57.343525 ip-10-0-136-108 kubenswrapper[2578]: E0731 23:52:57.343492    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b039bb5fe42ded3ffb86ae955326dc3c2e6b425e8a874d5e01c58083d1e1373e\": container with ID starting with b039bb5fe42ded3ffb86ae955326dc3c2e6b425e8a874d5e01c58083d1e1373e not found: ID does not exist" containerID="b039bb5fe42ded3ffb86ae955326dc3c2e6b425e8a874d5e01c58083d1e1373e"
Jul 31 23:52:57.343639 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:52:57.343555    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b039bb5fe42ded3ffb86ae955326dc3c2e6b425e8a874d5e01c58083d1e1373e"} err="failed to get container status \"b039bb5fe42ded3ffb86ae955326dc3c2e6b425e8a874d5e01c58083d1e1373e\": rpc error: code = NotFound desc = could not find container \"b039bb5fe42ded3ffb86ae955326dc3c2e6b425e8a874d5e01c58083d1e1373e\": container with ID starting with b039bb5fe42ded3ffb86ae955326dc3c2e6b425e8a874d5e01c58083d1e1373e not found: ID does not exist"
Jul 31 23:52:57.352695 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:52:57.352673    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve/kserve-controller-manager-674d959fb8-rsmsd"]
Jul 31 23:52:57.356259 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:52:57.356238    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve/kserve-controller-manager-674d959fb8-rsmsd"]
Jul 31 23:52:58.488649 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:52:58.488617    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f27faa5d-7caa-4b79-a3ec-6bed34a3fc4a" path="/var/lib/kubelet/pods/f27faa5d-7caa-4b79-a3ec-6bed34a3fc4a/volumes"
Jul 31 23:53:00.229089 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:53:00.229058    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gttdm_b32e6d42-e4e0-4113-b8ce-f5d5c5dc13f7/konnectivity-agent/0.log" limit=1024
Jul 31 23:53:01.346579 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:53:01.346538    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-5fc6b5dfb9-j4bdc" podUID="14a1f47f-df70-45e8-b5a5-5903ffb385e4" containerName="console" containerID="cri-o://907d1aa5a01779884816fd388cf3a724001640a5b2e28986f1f04fbab741f3f6" gracePeriod=15
Jul 31 23:53:01.589793 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:53:01.589767    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5fc6b5dfb9-j4bdc"
Jul 31 23:53:01.706862 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:53:01.706769    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/14a1f47f-df70-45e8-b5a5-5903ffb385e4-trusted-ca-bundle\") pod \"14a1f47f-df70-45e8-b5a5-5903ffb385e4\" (UID: \"14a1f47f-df70-45e8-b5a5-5903ffb385e4\") "
Jul 31 23:53:01.706862 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:53:01.706812    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/14a1f47f-df70-45e8-b5a5-5903ffb385e4-oauth-serving-cert\") pod \"14a1f47f-df70-45e8-b5a5-5903ffb385e4\" (UID: \"14a1f47f-df70-45e8-b5a5-5903ffb385e4\") "
Jul 31 23:53:01.706862 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:53:01.706837    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/14a1f47f-df70-45e8-b5a5-5903ffb385e4-console-serving-cert\") pod \"14a1f47f-df70-45e8-b5a5-5903ffb385e4\" (UID: \"14a1f47f-df70-45e8-b5a5-5903ffb385e4\") "
Jul 31 23:53:01.707137 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:53:01.706874    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/14a1f47f-df70-45e8-b5a5-5903ffb385e4-service-ca\") pod \"14a1f47f-df70-45e8-b5a5-5903ffb385e4\" (UID: \"14a1f47f-df70-45e8-b5a5-5903ffb385e4\") "
Jul 31 23:53:01.707137 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:53:01.706911    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/14a1f47f-df70-45e8-b5a5-5903ffb385e4-console-config\") pod \"14a1f47f-df70-45e8-b5a5-5903ffb385e4\" (UID: \"14a1f47f-df70-45e8-b5a5-5903ffb385e4\") "
Jul 31 23:53:01.707137 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:53:01.706969    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dff27\" (UniqueName: \"kubernetes.io/projected/14a1f47f-df70-45e8-b5a5-5903ffb385e4-kube-api-access-dff27\") pod \"14a1f47f-df70-45e8-b5a5-5903ffb385e4\" (UID: \"14a1f47f-df70-45e8-b5a5-5903ffb385e4\") "
Jul 31 23:53:01.707137 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:53:01.707092    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/14a1f47f-df70-45e8-b5a5-5903ffb385e4-console-oauth-config\") pod \"14a1f47f-df70-45e8-b5a5-5903ffb385e4\" (UID: \"14a1f47f-df70-45e8-b5a5-5903ffb385e4\") "
Jul 31 23:53:01.708026 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:53:01.707502    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/14a1f47f-df70-45e8-b5a5-5903ffb385e4-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "14a1f47f-df70-45e8-b5a5-5903ffb385e4" (UID: "14a1f47f-df70-45e8-b5a5-5903ffb385e4"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 31 23:53:01.708026 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:53:01.707699    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/14a1f47f-df70-45e8-b5a5-5903ffb385e4-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "14a1f47f-df70-45e8-b5a5-5903ffb385e4" (UID: "14a1f47f-df70-45e8-b5a5-5903ffb385e4"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 31 23:53:01.708026 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:53:01.707752    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/14a1f47f-df70-45e8-b5a5-5903ffb385e4-console-config" (OuterVolumeSpecName: "console-config") pod "14a1f47f-df70-45e8-b5a5-5903ffb385e4" (UID: "14a1f47f-df70-45e8-b5a5-5903ffb385e4"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 31 23:53:01.708026 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:53:01.707776    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/14a1f47f-df70-45e8-b5a5-5903ffb385e4-service-ca" (OuterVolumeSpecName: "service-ca") pod "14a1f47f-df70-45e8-b5a5-5903ffb385e4" (UID: "14a1f47f-df70-45e8-b5a5-5903ffb385e4"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 31 23:53:01.708026 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:53:01.707826    2578 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/14a1f47f-df70-45e8-b5a5-5903ffb385e4-trusted-ca-bundle\") on node \"ip-10-0-136-108.ec2.internal\" DevicePath \"\""
Jul 31 23:53:01.710424 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:53:01.710393    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/14a1f47f-df70-45e8-b5a5-5903ffb385e4-kube-api-access-dff27" (OuterVolumeSpecName: "kube-api-access-dff27") pod "14a1f47f-df70-45e8-b5a5-5903ffb385e4" (UID: "14a1f47f-df70-45e8-b5a5-5903ffb385e4"). InnerVolumeSpecName "kube-api-access-dff27". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 31 23:53:01.714832 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:53:01.714806    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/14a1f47f-df70-45e8-b5a5-5903ffb385e4-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "14a1f47f-df70-45e8-b5a5-5903ffb385e4" (UID: "14a1f47f-df70-45e8-b5a5-5903ffb385e4"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 31 23:53:01.715096 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:53:01.715077    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/14a1f47f-df70-45e8-b5a5-5903ffb385e4-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "14a1f47f-df70-45e8-b5a5-5903ffb385e4" (UID: "14a1f47f-df70-45e8-b5a5-5903ffb385e4"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 31 23:53:01.809265 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:53:01.809228    2578 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/14a1f47f-df70-45e8-b5a5-5903ffb385e4-oauth-serving-cert\") on node \"ip-10-0-136-108.ec2.internal\" DevicePath \"\""
Jul 31 23:53:01.809265 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:53:01.809266    2578 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/14a1f47f-df70-45e8-b5a5-5903ffb385e4-console-serving-cert\") on node \"ip-10-0-136-108.ec2.internal\" DevicePath \"\""
Jul 31 23:53:01.809481 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:53:01.809282    2578 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/14a1f47f-df70-45e8-b5a5-5903ffb385e4-service-ca\") on node \"ip-10-0-136-108.ec2.internal\" DevicePath \"\""
Jul 31 23:53:01.809481 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:53:01.809295    2578 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/14a1f47f-df70-45e8-b5a5-5903ffb385e4-console-config\") on node \"ip-10-0-136-108.ec2.internal\" DevicePath \"\""
Jul 31 23:53:01.809481 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:53:01.809308    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dff27\" (UniqueName: \"kubernetes.io/projected/14a1f47f-df70-45e8-b5a5-5903ffb385e4-kube-api-access-dff27\") on node \"ip-10-0-136-108.ec2.internal\" DevicePath \"\""
Jul 31 23:53:01.809481 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:53:01.809321    2578 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/14a1f47f-df70-45e8-b5a5-5903ffb385e4-console-oauth-config\") on node \"ip-10-0-136-108.ec2.internal\" DevicePath \"\""
Jul 31 23:53:02.348435 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:53:02.348399    2578 generic.go:356] "Generic (PLEG): container finished" podID="14a1f47f-df70-45e8-b5a5-5903ffb385e4" containerID="907d1aa5a01779884816fd388cf3a724001640a5b2e28986f1f04fbab741f3f6" exitCode=0
Jul 31 23:53:02.348886 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:53:02.348466    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5fc6b5dfb9-j4bdc"
Jul 31 23:53:02.348886 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:53:02.348493    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5fc6b5dfb9-j4bdc" event={"ID":"14a1f47f-df70-45e8-b5a5-5903ffb385e4","Type":"ContainerDied","Data":"907d1aa5a01779884816fd388cf3a724001640a5b2e28986f1f04fbab741f3f6"}
Jul 31 23:53:02.348886 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:53:02.348573    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5fc6b5dfb9-j4bdc" event={"ID":"14a1f47f-df70-45e8-b5a5-5903ffb385e4","Type":"ContainerDied","Data":"4d0d37532722ae61e769afe34deeda5f4617f7e2dd14535e17a10445933fccb0"}
Jul 31 23:53:02.348886 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:53:02.348595    2578 scope.go:117] "RemoveContainer" containerID="907d1aa5a01779884816fd388cf3a724001640a5b2e28986f1f04fbab741f3f6"
Jul 31 23:53:02.357734 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:53:02.357719    2578 scope.go:117] "RemoveContainer" containerID="907d1aa5a01779884816fd388cf3a724001640a5b2e28986f1f04fbab741f3f6"
Jul 31 23:53:02.357976 ip-10-0-136-108 kubenswrapper[2578]: E0731 23:53:02.357953    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"907d1aa5a01779884816fd388cf3a724001640a5b2e28986f1f04fbab741f3f6\": container with ID starting with 907d1aa5a01779884816fd388cf3a724001640a5b2e28986f1f04fbab741f3f6 not found: ID does not exist" containerID="907d1aa5a01779884816fd388cf3a724001640a5b2e28986f1f04fbab741f3f6"
Jul 31 23:53:02.358026 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:53:02.357987    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"907d1aa5a01779884816fd388cf3a724001640a5b2e28986f1f04fbab741f3f6"} err="failed to get container status \"907d1aa5a01779884816fd388cf3a724001640a5b2e28986f1f04fbab741f3f6\": rpc error: code = NotFound desc = could not find container \"907d1aa5a01779884816fd388cf3a724001640a5b2e28986f1f04fbab741f3f6\": container with ID starting with 907d1aa5a01779884816fd388cf3a724001640a5b2e28986f1f04fbab741f3f6 not found: ID does not exist"
Jul 31 23:53:02.369285 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:53:02.369264    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5fc6b5dfb9-j4bdc"]
Jul 31 23:53:02.373615 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:53:02.373593    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-5fc6b5dfb9-j4bdc"]
Jul 31 23:53:02.488647 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:53:02.488612    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="14a1f47f-df70-45e8-b5a5-5903ffb385e4" path="/var/lib/kubelet/pods/14a1f47f-df70-45e8-b5a5-5903ffb385e4/volumes"
Jul 31 23:53:06.316321 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:53:06.316279    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gttdm_b32e6d42-e4e0-4113-b8ce-f5d5c5dc13f7/konnectivity-agent/0.log" limit=1024
Jul 31 23:53:28.335225 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:53:28.335189    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/kserve-controller-manager-674d959fb8-w5sb4"
Jul 31 23:53:29.351601 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:53:29.351557    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gttdm_b32e6d42-e4e0-4113-b8ce-f5d5c5dc13f7/konnectivity-agent/0.log" limit=1024
Jul 31 23:53:29.395718 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:53:29.395683    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/model-serving-api-7b954ffc95-sgfgx"]
Jul 31 23:53:29.396039 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:53:29.396025    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="14a1f47f-df70-45e8-b5a5-5903ffb385e4" containerName="console"
Jul 31 23:53:29.396039 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:53:29.396039    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="14a1f47f-df70-45e8-b5a5-5903ffb385e4" containerName="console"
Jul 31 23:53:29.396153 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:53:29.396051    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f27faa5d-7caa-4b79-a3ec-6bed34a3fc4a" containerName="manager"
Jul 31 23:53:29.396153 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:53:29.396057    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="f27faa5d-7caa-4b79-a3ec-6bed34a3fc4a" containerName="manager"
Jul 31 23:53:29.396153 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:53:29.396113    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="f27faa5d-7caa-4b79-a3ec-6bed34a3fc4a" containerName="manager"
Jul 31 23:53:29.396153 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:53:29.396122    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="14a1f47f-df70-45e8-b5a5-5903ffb385e4" containerName="console"
Jul 31 23:53:29.400423 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:53:29.400404    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/model-serving-api-7b954ffc95-sgfgx"
Jul 31 23:53:29.403236 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:53:29.403212    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"model-serving-api-dockercfg-c99j6\""
Jul 31 23:53:29.403349 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:53:29.403214    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"model-serving-api-tls\""
Jul 31 23:53:29.409516 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:53:29.409491    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/model-serving-api-7b954ffc95-sgfgx"]
Jul 31 23:53:29.412449 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:53:29.412428    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/odh-model-controller-8588677cfb-l6cv4"]
Jul 31 23:53:29.415862 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:53:29.415839    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/odh-model-controller-8588677cfb-l6cv4"
Jul 31 23:53:29.417732 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:53:29.417716    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"odh-model-controller-webhook-cert\""
Jul 31 23:53:29.418264 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:53:29.418246    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"odh-model-controller-metrics-tls\""
Jul 31 23:53:29.418537 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:53:29.418518    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"odh-model-controller-dockercfg-lvqdr\""
Jul 31 23:53:29.428407 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:53:29.428384    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/odh-model-controller-8588677cfb-l6cv4"]
Jul 31 23:53:29.436815 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:53:29.436783    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f7748b8b-f773-43ac-9a99-79ab1be97aef-cert\") pod \"odh-model-controller-8588677cfb-l6cv4\" (UID: \"f7748b8b-f773-43ac-9a99-79ab1be97aef\") " pod="kserve/odh-model-controller-8588677cfb-l6cv4"
Jul 31 23:53:29.436965 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:53:29.436851    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tp6ht\" (UniqueName: \"kubernetes.io/projected/72bf9f60-5250-489f-b905-aae303ca3304-kube-api-access-tp6ht\") pod \"model-serving-api-7b954ffc95-sgfgx\" (UID: \"72bf9f60-5250-489f-b905-aae303ca3304\") " pod="kserve/model-serving-api-7b954ffc95-sgfgx"
Jul 31 23:53:29.436965 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:53:29.436884    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f7748b8b-f773-43ac-9a99-79ab1be97aef-metrics-certs\") pod \"odh-model-controller-8588677cfb-l6cv4\" (UID: \"f7748b8b-f773-43ac-9a99-79ab1be97aef\") " pod="kserve/odh-model-controller-8588677cfb-l6cv4"
Jul 31 23:53:29.436965 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:53:29.436916    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bdttp\" (UniqueName: \"kubernetes.io/projected/f7748b8b-f773-43ac-9a99-79ab1be97aef-kube-api-access-bdttp\") pod \"odh-model-controller-8588677cfb-l6cv4\" (UID: \"f7748b8b-f773-43ac-9a99-79ab1be97aef\") " pod="kserve/odh-model-controller-8588677cfb-l6cv4"
Jul 31 23:53:29.437072 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:53:29.437046    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/72bf9f60-5250-489f-b905-aae303ca3304-tls-certs\") pod \"model-serving-api-7b954ffc95-sgfgx\" (UID: \"72bf9f60-5250-489f-b905-aae303ca3304\") " pod="kserve/model-serving-api-7b954ffc95-sgfgx"
Jul 31 23:53:29.445875 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:53:29.445835    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gttdm_b32e6d42-e4e0-4113-b8ce-f5d5c5dc13f7/konnectivity-agent/0.log" limit=1024
Jul 31 23:53:29.506492 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:53:29.506457    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gttdm_b32e6d42-e4e0-4113-b8ce-f5d5c5dc13f7/konnectivity-agent/0.log" limit=1024
Jul 31 23:53:29.538372 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:53:29.538340    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/72bf9f60-5250-489f-b905-aae303ca3304-tls-certs\") pod \"model-serving-api-7b954ffc95-sgfgx\" (UID: \"72bf9f60-5250-489f-b905-aae303ca3304\") " pod="kserve/model-serving-api-7b954ffc95-sgfgx"
Jul 31 23:53:29.538538 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:53:29.538383    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f7748b8b-f773-43ac-9a99-79ab1be97aef-cert\") pod \"odh-model-controller-8588677cfb-l6cv4\" (UID: \"f7748b8b-f773-43ac-9a99-79ab1be97aef\") " pod="kserve/odh-model-controller-8588677cfb-l6cv4"
Jul 31 23:53:29.538538 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:53:29.538417    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tp6ht\" (UniqueName: \"kubernetes.io/projected/72bf9f60-5250-489f-b905-aae303ca3304-kube-api-access-tp6ht\") pod \"model-serving-api-7b954ffc95-sgfgx\" (UID: \"72bf9f60-5250-489f-b905-aae303ca3304\") " pod="kserve/model-serving-api-7b954ffc95-sgfgx"
Jul 31 23:53:29.538538 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:53:29.538442    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f7748b8b-f773-43ac-9a99-79ab1be97aef-metrics-certs\") pod \"odh-model-controller-8588677cfb-l6cv4\" (UID: \"f7748b8b-f773-43ac-9a99-79ab1be97aef\") " pod="kserve/odh-model-controller-8588677cfb-l6cv4"
Jul 31 23:53:29.538538 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:53:29.538458    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bdttp\" (UniqueName: \"kubernetes.io/projected/f7748b8b-f773-43ac-9a99-79ab1be97aef-kube-api-access-bdttp\") pod \"odh-model-controller-8588677cfb-l6cv4\" (UID: \"f7748b8b-f773-43ac-9a99-79ab1be97aef\") " pod="kserve/odh-model-controller-8588677cfb-l6cv4"
Jul 31 23:53:29.538741 ip-10-0-136-108 kubenswrapper[2578]: E0731 23:53:29.538577    2578 secret.go:189] Couldn't get secret kserve/odh-model-controller-metrics-tls: secret "odh-model-controller-metrics-tls" not found
Jul 31 23:53:29.538741 ip-10-0-136-108 kubenswrapper[2578]: E0731 23:53:29.538662    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f7748b8b-f773-43ac-9a99-79ab1be97aef-metrics-certs podName:f7748b8b-f773-43ac-9a99-79ab1be97aef nodeName:}" failed. No retries permitted until 2026-07-31 23:53:30.038638741 +0000 UTC m=+434.006389491 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f7748b8b-f773-43ac-9a99-79ab1be97aef-metrics-certs") pod "odh-model-controller-8588677cfb-l6cv4" (UID: "f7748b8b-f773-43ac-9a99-79ab1be97aef") : secret "odh-model-controller-metrics-tls" not found
Jul 31 23:53:29.541046 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:53:29.541014    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f7748b8b-f773-43ac-9a99-79ab1be97aef-cert\") pod \"odh-model-controller-8588677cfb-l6cv4\" (UID: \"f7748b8b-f773-43ac-9a99-79ab1be97aef\") " pod="kserve/odh-model-controller-8588677cfb-l6cv4"
Jul 31 23:53:29.541205 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:53:29.541074    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/72bf9f60-5250-489f-b905-aae303ca3304-tls-certs\") pod \"model-serving-api-7b954ffc95-sgfgx\" (UID: \"72bf9f60-5250-489f-b905-aae303ca3304\") " pod="kserve/model-serving-api-7b954ffc95-sgfgx"
Jul 31 23:53:29.546647 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:53:29.546619    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tp6ht\" (UniqueName: \"kubernetes.io/projected/72bf9f60-5250-489f-b905-aae303ca3304-kube-api-access-tp6ht\") pod \"model-serving-api-7b954ffc95-sgfgx\" (UID: \"72bf9f60-5250-489f-b905-aae303ca3304\") " pod="kserve/model-serving-api-7b954ffc95-sgfgx"
Jul 31 23:53:29.546780 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:53:29.546673    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bdttp\" (UniqueName: \"kubernetes.io/projected/f7748b8b-f773-43ac-9a99-79ab1be97aef-kube-api-access-bdttp\") pod \"odh-model-controller-8588677cfb-l6cv4\" (UID: \"f7748b8b-f773-43ac-9a99-79ab1be97aef\") " pod="kserve/odh-model-controller-8588677cfb-l6cv4"
Jul 31 23:53:29.608373 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:53:29.608285    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gttdm_b32e6d42-e4e0-4113-b8ce-f5d5c5dc13f7/konnectivity-agent/0.log" limit=1024
Jul 31 23:53:29.711353 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:53:29.711302    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/model-serving-api-7b954ffc95-sgfgx"
Jul 31 23:53:29.845376 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:53:29.845351    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/model-serving-api-7b954ffc95-sgfgx"]
Jul 31 23:53:29.848086 ip-10-0-136-108 kubenswrapper[2578]: W0731 23:53:29.848058    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod72bf9f60_5250_489f_b905_aae303ca3304.slice/crio-aa36ced51506ccb41524013a80ed3362c35061ff9e328d79d91a515fcaea1e15 WatchSource:0}: Error finding container aa36ced51506ccb41524013a80ed3362c35061ff9e328d79d91a515fcaea1e15: Status 404 returned error can't find the container with id aa36ced51506ccb41524013a80ed3362c35061ff9e328d79d91a515fcaea1e15
Jul 31 23:53:30.043115 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:53:30.043082    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f7748b8b-f773-43ac-9a99-79ab1be97aef-metrics-certs\") pod \"odh-model-controller-8588677cfb-l6cv4\" (UID: \"f7748b8b-f773-43ac-9a99-79ab1be97aef\") " pod="kserve/odh-model-controller-8588677cfb-l6cv4"
Jul 31 23:53:30.043316 ip-10-0-136-108 kubenswrapper[2578]: E0731 23:53:30.043273    2578 secret.go:189] Couldn't get secret kserve/odh-model-controller-metrics-tls: secret "odh-model-controller-metrics-tls" not found
Jul 31 23:53:30.043366 ip-10-0-136-108 kubenswrapper[2578]: E0731 23:53:30.043348    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f7748b8b-f773-43ac-9a99-79ab1be97aef-metrics-certs podName:f7748b8b-f773-43ac-9a99-79ab1be97aef nodeName:}" failed. No retries permitted until 2026-07-31 23:53:31.043329696 +0000 UTC m=+435.011080448 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f7748b8b-f773-43ac-9a99-79ab1be97aef-metrics-certs") pod "odh-model-controller-8588677cfb-l6cv4" (UID: "f7748b8b-f773-43ac-9a99-79ab1be97aef") : secret "odh-model-controller-metrics-tls" not found
Jul 31 23:53:30.044186 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:53:30.044151    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gttdm_b32e6d42-e4e0-4113-b8ce-f5d5c5dc13f7/konnectivity-agent/0.log" limit=1024
Jul 31 23:53:30.145413 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:53:30.145380    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gttdm_b32e6d42-e4e0-4113-b8ce-f5d5c5dc13f7/konnectivity-agent/0.log" limit=1024
Jul 31 23:53:30.472028 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:53:30.471938    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/model-serving-api-7b954ffc95-sgfgx" event={"ID":"72bf9f60-5250-489f-b905-aae303ca3304","Type":"ContainerStarted","Data":"aa36ced51506ccb41524013a80ed3362c35061ff9e328d79d91a515fcaea1e15"}
Jul 31 23:53:30.538051 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:53:30.538010    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gttdm_b32e6d42-e4e0-4113-b8ce-f5d5c5dc13f7/konnectivity-agent/0.log" limit=1024
Jul 31 23:53:31.053810 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:53:31.053783    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f7748b8b-f773-43ac-9a99-79ab1be97aef-metrics-certs\") pod \"odh-model-controller-8588677cfb-l6cv4\" (UID: \"f7748b8b-f773-43ac-9a99-79ab1be97aef\") " pod="kserve/odh-model-controller-8588677cfb-l6cv4"
Jul 31 23:53:31.056210 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:53:31.056161    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f7748b8b-f773-43ac-9a99-79ab1be97aef-metrics-certs\") pod \"odh-model-controller-8588677cfb-l6cv4\" (UID: \"f7748b8b-f773-43ac-9a99-79ab1be97aef\") " pod="kserve/odh-model-controller-8588677cfb-l6cv4"
Jul 31 23:53:31.227783 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:53:31.227745    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/odh-model-controller-8588677cfb-l6cv4"
Jul 31 23:53:31.359600 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:53:31.359567    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/odh-model-controller-8588677cfb-l6cv4"]
Jul 31 23:53:31.362655 ip-10-0-136-108 kubenswrapper[2578]: W0731 23:53:31.362622    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf7748b8b_f773_43ac_9a99_79ab1be97aef.slice/crio-fc8a159089122a9f1a47832ad6694e62ee98b6fe129f7ad077115f432129affa WatchSource:0}: Error finding container fc8a159089122a9f1a47832ad6694e62ee98b6fe129f7ad077115f432129affa: Status 404 returned error can't find the container with id fc8a159089122a9f1a47832ad6694e62ee98b6fe129f7ad077115f432129affa
Jul 31 23:53:31.478515 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:53:31.478480    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/model-serving-api-7b954ffc95-sgfgx" event={"ID":"72bf9f60-5250-489f-b905-aae303ca3304","Type":"ContainerStarted","Data":"17aff485fb17e10b07d72f0207e3351528665b41ccdda993010e1c8935ddf9f3"}
Jul 31 23:53:31.478983 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:53:31.478658    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/model-serving-api-7b954ffc95-sgfgx"
Jul 31 23:53:31.479712 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:53:31.479687    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/odh-model-controller-8588677cfb-l6cv4" event={"ID":"f7748b8b-f773-43ac-9a99-79ab1be97aef","Type":"ContainerStarted","Data":"fc8a159089122a9f1a47832ad6694e62ee98b6fe129f7ad077115f432129affa"}
Jul 31 23:53:31.494061 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:53:31.494015    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/model-serving-api-7b954ffc95-sgfgx" podStartSLOduration=1.351901252 podStartE2EDuration="2.494000086s" totalImagesPullingTime="1.142098834s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 23:53:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 23:53:31.492052701 +0000 UTC m=+435.459803460" watchObservedRunningTime="2026-07-31 23:53:31.494000086 +0000 UTC m=+435.461750844"
Jul 31 23:53:34.499905 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:53:34.499869    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/odh-model-controller-8588677cfb-l6cv4" event={"ID":"f7748b8b-f773-43ac-9a99-79ab1be97aef","Type":"ContainerStarted","Data":"a2b588ab6d0544a63acfa3fa5e6b00fd762acea27b86993cd9b934263a40cc87"}
Jul 31 23:53:34.500322 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:53:34.500095    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/odh-model-controller-8588677cfb-l6cv4"
Jul 31 23:53:34.532411 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:53:34.532357    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/odh-model-controller-8588677cfb-l6cv4" podStartSLOduration=3.128920308 podStartE2EDuration="5.532341632s" totalImagesPullingTime="2.403421324s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 23:53:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 23:53:34.531002112 +0000 UTC m=+438.498752871" watchObservedRunningTime="2026-07-31 23:53:34.532341632 +0000 UTC m=+438.500092389"
Jul 31 23:53:42.491824 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:53:42.491747    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/model-serving-api-7b954ffc95-sgfgx"
Jul 31 23:53:45.507198 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:53:45.507160    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/odh-model-controller-8588677cfb-l6cv4"
Jul 31 23:54:37.913659 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:54:37.913621    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gttdm_b32e6d42-e4e0-4113-b8ce-f5d5c5dc13f7/konnectivity-agent/0.log" limit=1024
Jul 31 23:55:04.798362 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:55:04.798323    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gttdm_b32e6d42-e4e0-4113-b8ce-f5d5c5dc13f7/konnectivity-agent/0.log" limit=1024
Jul 31 23:55:04.900469 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:55:04.900429    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gttdm_b32e6d42-e4e0-4113-b8ce-f5d5c5dc13f7/konnectivity-agent/0.log" limit=1024
Jul 31 23:55:16.907932 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:55:16.907838    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gttdm_b32e6d42-e4e0-4113-b8ce-f5d5c5dc13f7/konnectivity-agent/0.log" limit=1024
Jul 31 23:55:16.988541 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:55:16.988505    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gttdm_b32e6d42-e4e0-4113-b8ce-f5d5c5dc13f7/konnectivity-agent/0.log" limit=1024
Jul 31 23:55:17.126782 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:55:17.126740    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gttdm_b32e6d42-e4e0-4113-b8ce-f5d5c5dc13f7/konnectivity-agent/0.log" limit=1024
Jul 31 23:55:17.150964 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:55:17.150930    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-716dc-predictor-6b5b687b9f-dvfl2"]
Jul 31 23:55:17.154312 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:55:17.154293    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-716dc-predictor-6b5b687b9f-dvfl2"
Jul 31 23:55:17.156441 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:55:17.156385    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-716dc-predictor-serving-cert\""
Jul 31 23:55:17.156622 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:55:17.156602    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 31 23:55:17.156810 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:55:17.156790    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 31 23:55:17.157135 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:55:17.157120    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-j55jp\""
Jul 31 23:55:17.157242 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:55:17.157197    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-716dc-kube-rbac-proxy-sar-config\""
Jul 31 23:55:17.163728 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:55:17.163701    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-716dc-predictor-6b5b687b9f-dvfl2"]
Jul 31 23:55:17.229933 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:55:17.229893    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gttdm_b32e6d42-e4e0-4113-b8ce-f5d5c5dc13f7/konnectivity-agent/0.log" limit=1024
Jul 31 23:55:17.241740 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:55:17.241704    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3d9b874f-5a27-498b-93bc-d0771624ffd9-proxy-tls\") pod \"success-200-isvc-716dc-predictor-6b5b687b9f-dvfl2\" (UID: \"3d9b874f-5a27-498b-93bc-d0771624ffd9\") " pod="kserve-ci-e2e-test/success-200-isvc-716dc-predictor-6b5b687b9f-dvfl2"
Jul 31 23:55:17.241905 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:55:17.241768    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6h92x\" (UniqueName: \"kubernetes.io/projected/3d9b874f-5a27-498b-93bc-d0771624ffd9-kube-api-access-6h92x\") pod \"success-200-isvc-716dc-predictor-6b5b687b9f-dvfl2\" (UID: \"3d9b874f-5a27-498b-93bc-d0771624ffd9\") " pod="kserve-ci-e2e-test/success-200-isvc-716dc-predictor-6b5b687b9f-dvfl2"
Jul 31 23:55:17.241905 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:55:17.241840    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3d9b874f-5a27-498b-93bc-d0771624ffd9-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-716dc-predictor-6b5b687b9f-dvfl2\" (UID: \"3d9b874f-5a27-498b-93bc-d0771624ffd9\") " pod="kserve-ci-e2e-test/success-200-isvc-716dc-predictor-6b5b687b9f-dvfl2"
Jul 31 23:55:17.333725 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:55:17.333674    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gttdm_b32e6d42-e4e0-4113-b8ce-f5d5c5dc13f7/konnectivity-agent/0.log" limit=1024
Jul 31 23:55:17.342398 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:55:17.342361    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3d9b874f-5a27-498b-93bc-d0771624ffd9-proxy-tls\") pod \"success-200-isvc-716dc-predictor-6b5b687b9f-dvfl2\" (UID: \"3d9b874f-5a27-498b-93bc-d0771624ffd9\") " pod="kserve-ci-e2e-test/success-200-isvc-716dc-predictor-6b5b687b9f-dvfl2"
Jul 31 23:55:17.342561 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:55:17.342413    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6h92x\" (UniqueName: \"kubernetes.io/projected/3d9b874f-5a27-498b-93bc-d0771624ffd9-kube-api-access-6h92x\") pod \"success-200-isvc-716dc-predictor-6b5b687b9f-dvfl2\" (UID: \"3d9b874f-5a27-498b-93bc-d0771624ffd9\") " pod="kserve-ci-e2e-test/success-200-isvc-716dc-predictor-6b5b687b9f-dvfl2"
Jul 31 23:55:17.342561 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:55:17.342451    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3d9b874f-5a27-498b-93bc-d0771624ffd9-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-716dc-predictor-6b5b687b9f-dvfl2\" (UID: \"3d9b874f-5a27-498b-93bc-d0771624ffd9\") " pod="kserve-ci-e2e-test/success-200-isvc-716dc-predictor-6b5b687b9f-dvfl2"
Jul 31 23:55:17.343145 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:55:17.343117    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3d9b874f-5a27-498b-93bc-d0771624ffd9-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-716dc-predictor-6b5b687b9f-dvfl2\" (UID: \"3d9b874f-5a27-498b-93bc-d0771624ffd9\") " pod="kserve-ci-e2e-test/success-200-isvc-716dc-predictor-6b5b687b9f-dvfl2"
Jul 31 23:55:17.344980 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:55:17.344960    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3d9b874f-5a27-498b-93bc-d0771624ffd9-proxy-tls\") pod \"success-200-isvc-716dc-predictor-6b5b687b9f-dvfl2\" (UID: \"3d9b874f-5a27-498b-93bc-d0771624ffd9\") " pod="kserve-ci-e2e-test/success-200-isvc-716dc-predictor-6b5b687b9f-dvfl2"
Jul 31 23:55:17.352087 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:55:17.352060    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6h92x\" (UniqueName: \"kubernetes.io/projected/3d9b874f-5a27-498b-93bc-d0771624ffd9-kube-api-access-6h92x\") pod \"success-200-isvc-716dc-predictor-6b5b687b9f-dvfl2\" (UID: \"3d9b874f-5a27-498b-93bc-d0771624ffd9\") " pod="kserve-ci-e2e-test/success-200-isvc-716dc-predictor-6b5b687b9f-dvfl2"
Jul 31 23:55:17.420979 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:55:17.420896    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gttdm_b32e6d42-e4e0-4113-b8ce-f5d5c5dc13f7/konnectivity-agent/0.log" limit=1024
Jul 31 23:55:17.468143 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:55:17.468113    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-716dc-predictor-6b5b687b9f-dvfl2"
Jul 31 23:55:17.486288 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:55:17.486253    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gttdm_b32e6d42-e4e0-4113-b8ce-f5d5c5dc13f7/konnectivity-agent/0.log" limit=1024
Jul 31 23:55:17.594601 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:55:17.594569    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gttdm_b32e6d42-e4e0-4113-b8ce-f5d5c5dc13f7/konnectivity-agent/0.log" limit=1024
Jul 31 23:55:17.611759 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:55:17.611726    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-716dc-predictor-6b5b687b9f-dvfl2"]
Jul 31 23:55:17.614881 ip-10-0-136-108 kubenswrapper[2578]: W0731 23:55:17.614855    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3d9b874f_5a27_498b_93bc_d0771624ffd9.slice/crio-717088ae212ab1dc58238781f7856aa47f4de3e4a6684b782c0afa5922c2bfa5 WatchSource:0}: Error finding container 717088ae212ab1dc58238781f7856aa47f4de3e4a6684b782c0afa5922c2bfa5: Status 404 returned error can't find the container with id 717088ae212ab1dc58238781f7856aa47f4de3e4a6684b782c0afa5922c2bfa5
Jul 31 23:55:17.956335 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:55:17.956303    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-716dc-predictor-6b5b687b9f-dvfl2" event={"ID":"3d9b874f-5a27-498b-93bc-d0771624ffd9","Type":"ContainerStarted","Data":"717088ae212ab1dc58238781f7856aa47f4de3e4a6684b782c0afa5922c2bfa5"}
Jul 31 23:55:30.030238 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:55:30.030181    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-716dc-predictor-6b5b687b9f-dvfl2" event={"ID":"3d9b874f-5a27-498b-93bc-d0771624ffd9","Type":"ContainerStarted","Data":"a0406961cf3d9dbce5acd2e7d34c4ebffa6f893f7ef20b6ff14e2e0c3e2bb315"}
Jul 31 23:55:31.757643 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:55:31.757606    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gttdm_b32e6d42-e4e0-4113-b8ce-f5d5c5dc13f7/konnectivity-agent/0.log" limit=1024
Jul 31 23:55:32.044060 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:55:32.043972    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-716dc-predictor-6b5b687b9f-dvfl2" event={"ID":"3d9b874f-5a27-498b-93bc-d0771624ffd9","Type":"ContainerStarted","Data":"e76035c682350428c0ab02e7a7f883467d453f15ca69ea8ec0a8da1a2ad54d2e"}
Jul 31 23:55:32.044232 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:55:32.044194    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-716dc-predictor-6b5b687b9f-dvfl2"
Jul 31 23:55:32.061232 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:55:32.061158    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/success-200-isvc-716dc-predictor-6b5b687b9f-dvfl2" podStartSLOduration=0.911253652 podStartE2EDuration="15.06114084s" totalImagesPullingTime="14.149887188s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 23:55:17 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 23:55:32.059092941 +0000 UTC m=+556.026843699" watchObservedRunningTime="2026-07-31 23:55:32.06114084 +0000 UTC m=+556.028891598"
Jul 31 23:55:33.048303 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:55:33.048265    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-716dc-predictor-6b5b687b9f-dvfl2"
Jul 31 23:55:33.049691 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:55:33.049666    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-716dc-predictor-6b5b687b9f-dvfl2" podUID="3d9b874f-5a27-498b-93bc-d0771624ffd9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.30:8080: connect: connection refused"
Jul 31 23:55:34.052556 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:55:34.052521    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-716dc-predictor-6b5b687b9f-dvfl2" podUID="3d9b874f-5a27-498b-93bc-d0771624ffd9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.30:8080: connect: connection refused"
Jul 31 23:55:39.058494 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:55:39.058465    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-716dc-predictor-6b5b687b9f-dvfl2"
Jul 31 23:55:39.058949 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:55:39.058916    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-716dc-predictor-6b5b687b9f-dvfl2" podUID="3d9b874f-5a27-498b-93bc-d0771624ffd9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.30:8080: connect: connection refused"
Jul 31 23:55:49.059302 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:55:49.059262    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-716dc-predictor-6b5b687b9f-dvfl2" podUID="3d9b874f-5a27-498b-93bc-d0771624ffd9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.30:8080: connect: connection refused"
Jul 31 23:55:59.058933 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:55:59.058891    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-716dc-predictor-6b5b687b9f-dvfl2" podUID="3d9b874f-5a27-498b-93bc-d0771624ffd9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.30:8080: connect: connection refused"
Jul 31 23:56:09.059515 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:56:09.059469    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-716dc-predictor-6b5b687b9f-dvfl2" podUID="3d9b874f-5a27-498b-93bc-d0771624ffd9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.30:8080: connect: connection refused"
Jul 31 23:56:19.059817 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:56:19.059784    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-716dc-predictor-6b5b687b9f-dvfl2"
Jul 31 23:56:36.887438 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:56:36.887405    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/switch-graph-716dc-7d7ffc79d4-2bdn6"]
Jul 31 23:56:36.890857 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:56:36.890836    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/switch-graph-716dc-7d7ffc79d4-2bdn6"
Jul 31 23:56:36.891124 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:56:36.891093    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gttdm_b32e6d42-e4e0-4113-b8ce-f5d5c5dc13f7/konnectivity-agent/0.log" limit=1024
Jul 31 23:56:36.892503 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:56:36.892481    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"switch-graph-716dc-serving-cert\""
Jul 31 23:56:36.892617 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:56:36.892598    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"switch-graph-716dc-kube-rbac-proxy-sar-config\""
Jul 31 23:56:36.900418 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:56:36.900396    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/switch-graph-716dc-7d7ffc79d4-2bdn6"]
Jul 31 23:56:37.055279 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:56:37.055230    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/161566e9-254c-4c2d-aa6c-351f0d666a07-proxy-tls\") pod \"switch-graph-716dc-7d7ffc79d4-2bdn6\" (UID: \"161566e9-254c-4c2d-aa6c-351f0d666a07\") " pod="kserve-ci-e2e-test/switch-graph-716dc-7d7ffc79d4-2bdn6"
Jul 31 23:56:37.055469 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:56:37.055297    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/161566e9-254c-4c2d-aa6c-351f0d666a07-openshift-service-ca-bundle\") pod \"switch-graph-716dc-7d7ffc79d4-2bdn6\" (UID: \"161566e9-254c-4c2d-aa6c-351f0d666a07\") " pod="kserve-ci-e2e-test/switch-graph-716dc-7d7ffc79d4-2bdn6"
Jul 31 23:56:37.122677 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:56:37.122644    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gttdm_b32e6d42-e4e0-4113-b8ce-f5d5c5dc13f7/konnectivity-agent/0.log" limit=1024
Jul 31 23:56:37.156686 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:56:37.156617    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/161566e9-254c-4c2d-aa6c-351f0d666a07-openshift-service-ca-bundle\") pod \"switch-graph-716dc-7d7ffc79d4-2bdn6\" (UID: \"161566e9-254c-4c2d-aa6c-351f0d666a07\") " pod="kserve-ci-e2e-test/switch-graph-716dc-7d7ffc79d4-2bdn6"
Jul 31 23:56:37.156797 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:56:37.156724    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/161566e9-254c-4c2d-aa6c-351f0d666a07-proxy-tls\") pod \"switch-graph-716dc-7d7ffc79d4-2bdn6\" (UID: \"161566e9-254c-4c2d-aa6c-351f0d666a07\") " pod="kserve-ci-e2e-test/switch-graph-716dc-7d7ffc79d4-2bdn6"
Jul 31 23:56:37.157422 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:56:37.157400    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/161566e9-254c-4c2d-aa6c-351f0d666a07-openshift-service-ca-bundle\") pod \"switch-graph-716dc-7d7ffc79d4-2bdn6\" (UID: \"161566e9-254c-4c2d-aa6c-351f0d666a07\") " pod="kserve-ci-e2e-test/switch-graph-716dc-7d7ffc79d4-2bdn6"
Jul 31 23:56:37.159209 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:56:37.159192    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/161566e9-254c-4c2d-aa6c-351f0d666a07-proxy-tls\") pod \"switch-graph-716dc-7d7ffc79d4-2bdn6\" (UID: \"161566e9-254c-4c2d-aa6c-351f0d666a07\") " pod="kserve-ci-e2e-test/switch-graph-716dc-7d7ffc79d4-2bdn6"
Jul 31 23:56:37.201894 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:56:37.201869    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/switch-graph-716dc-7d7ffc79d4-2bdn6"
Jul 31 23:56:37.332231 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:56:37.332203    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/switch-graph-716dc-7d7ffc79d4-2bdn6"]
Jul 31 23:56:37.334474 ip-10-0-136-108 kubenswrapper[2578]: W0731 23:56:37.334447    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod161566e9_254c_4c2d_aa6c_351f0d666a07.slice/crio-7a49c123f04da46f4ab9cb0b72546216eb761e352e21c52c68e09812060eeb89 WatchSource:0}: Error finding container 7a49c123f04da46f4ab9cb0b72546216eb761e352e21c52c68e09812060eeb89: Status 404 returned error can't find the container with id 7a49c123f04da46f4ab9cb0b72546216eb761e352e21c52c68e09812060eeb89
Jul 31 23:56:37.343482 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:56:37.343449    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/switch-graph-716dc-7d7ffc79d4-2bdn6" event={"ID":"161566e9-254c-4c2d-aa6c-351f0d666a07","Type":"ContainerStarted","Data":"7a49c123f04da46f4ab9cb0b72546216eb761e352e21c52c68e09812060eeb89"}
Jul 31 23:56:39.355901 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:56:39.355861    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/switch-graph-716dc-7d7ffc79d4-2bdn6" event={"ID":"161566e9-254c-4c2d-aa6c-351f0d666a07","Type":"ContainerStarted","Data":"2d3c68192a8740af48c4833efc21b8ae557dd7d4825998826521c85cdd3778d9"}
Jul 31 23:56:39.356308 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:56:39.355944    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/switch-graph-716dc-7d7ffc79d4-2bdn6"
Jul 31 23:56:39.371165 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:56:39.371118    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/switch-graph-716dc-7d7ffc79d4-2bdn6" podStartSLOduration=1.460423152 podStartE2EDuration="3.371102194s" totalImagesPullingTime="1.910679042s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 23:56:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 23:56:39.369972995 +0000 UTC m=+623.337723755" watchObservedRunningTime="2026-07-31 23:56:39.371102194 +0000 UTC m=+623.338852959"
Jul 31 23:56:45.367494 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:56:45.367466    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/switch-graph-716dc-7d7ffc79d4-2bdn6"
Jul 31 23:56:47.035165 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:56:47.035135    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/switch-graph-716dc-7d7ffc79d4-2bdn6"]
Jul 31 23:56:47.035611 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:56:47.035345    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/switch-graph-716dc-7d7ffc79d4-2bdn6" podUID="161566e9-254c-4c2d-aa6c-351f0d666a07" containerName="switch-graph-716dc" containerID="cri-o://2d3c68192a8740af48c4833efc21b8ae557dd7d4825998826521c85cdd3778d9" gracePeriod=30
Jul 31 23:56:47.056403 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:56:47.056365    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gttdm_b32e6d42-e4e0-4113-b8ce-f5d5c5dc13f7/konnectivity-agent/0.log" limit=1024
Jul 31 23:56:47.155957 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:56:47.155923    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-716dc-predictor-6b5b687b9f-dvfl2"]
Jul 31 23:56:47.156184 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:56:47.156141    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-716dc-predictor-6b5b687b9f-dvfl2" podUID="3d9b874f-5a27-498b-93bc-d0771624ffd9" containerName="kserve-container" containerID="cri-o://a0406961cf3d9dbce5acd2e7d34c4ebffa6f893f7ef20b6ff14e2e0c3e2bb315" gracePeriod=30
Jul 31 23:56:47.156317 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:56:47.156223    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-716dc-predictor-6b5b687b9f-dvfl2" podUID="3d9b874f-5a27-498b-93bc-d0771624ffd9" containerName="kube-rbac-proxy" containerID="cri-o://e76035c682350428c0ab02e7a7f883467d453f15ca69ea8ec0a8da1a2ad54d2e" gracePeriod=30
Jul 31 23:56:47.231997 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:56:47.231959    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-32541-predictor-7ff4c79d55-9mbgd"]
Jul 31 23:56:47.235648 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:56:47.235624    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-32541-predictor-7ff4c79d55-9mbgd"
Jul 31 23:56:47.237358 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:56:47.237336    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-32541-predictor-serving-cert\""
Jul 31 23:56:47.237465 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:56:47.237361    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-32541-kube-rbac-proxy-sar-config\""
Jul 31 23:56:47.245442 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:56:47.245418    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-32541-predictor-7ff4c79d55-9mbgd"]
Jul 31 23:56:47.245817 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:56:47.245791    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fbac4361-0369-4cea-ab12-03a4e4ea1171-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-32541-predictor-7ff4c79d55-9mbgd\" (UID: \"fbac4361-0369-4cea-ab12-03a4e4ea1171\") " pod="kserve-ci-e2e-test/success-200-isvc-32541-predictor-7ff4c79d55-9mbgd"
Jul 31 23:56:47.245934 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:56:47.245915    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fbac4361-0369-4cea-ab12-03a4e4ea1171-proxy-tls\") pod \"success-200-isvc-32541-predictor-7ff4c79d55-9mbgd\" (UID: \"fbac4361-0369-4cea-ab12-03a4e4ea1171\") " pod="kserve-ci-e2e-test/success-200-isvc-32541-predictor-7ff4c79d55-9mbgd"
Jul 31 23:56:47.246017 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:56:47.245999    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z5fvl\" (UniqueName: \"kubernetes.io/projected/fbac4361-0369-4cea-ab12-03a4e4ea1171-kube-api-access-z5fvl\") pod \"success-200-isvc-32541-predictor-7ff4c79d55-9mbgd\" (UID: \"fbac4361-0369-4cea-ab12-03a4e4ea1171\") " pod="kserve-ci-e2e-test/success-200-isvc-32541-predictor-7ff4c79d55-9mbgd"
Jul 31 23:56:47.346802 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:56:47.346715    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fbac4361-0369-4cea-ab12-03a4e4ea1171-proxy-tls\") pod \"success-200-isvc-32541-predictor-7ff4c79d55-9mbgd\" (UID: \"fbac4361-0369-4cea-ab12-03a4e4ea1171\") " pod="kserve-ci-e2e-test/success-200-isvc-32541-predictor-7ff4c79d55-9mbgd"
Jul 31 23:56:47.346802 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:56:47.346758    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z5fvl\" (UniqueName: \"kubernetes.io/projected/fbac4361-0369-4cea-ab12-03a4e4ea1171-kube-api-access-z5fvl\") pod \"success-200-isvc-32541-predictor-7ff4c79d55-9mbgd\" (UID: \"fbac4361-0369-4cea-ab12-03a4e4ea1171\") " pod="kserve-ci-e2e-test/success-200-isvc-32541-predictor-7ff4c79d55-9mbgd"
Jul 31 23:56:47.347043 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:56:47.346812    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fbac4361-0369-4cea-ab12-03a4e4ea1171-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-32541-predictor-7ff4c79d55-9mbgd\" (UID: \"fbac4361-0369-4cea-ab12-03a4e4ea1171\") " pod="kserve-ci-e2e-test/success-200-isvc-32541-predictor-7ff4c79d55-9mbgd"
Jul 31 23:56:47.347043 ip-10-0-136-108 kubenswrapper[2578]: E0731 23:56:47.346875    2578 secret.go:189] Couldn't get secret kserve-ci-e2e-test/success-200-isvc-32541-predictor-serving-cert: secret "success-200-isvc-32541-predictor-serving-cert" not found
Jul 31 23:56:47.347043 ip-10-0-136-108 kubenswrapper[2578]: E0731 23:56:47.346952    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fbac4361-0369-4cea-ab12-03a4e4ea1171-proxy-tls podName:fbac4361-0369-4cea-ab12-03a4e4ea1171 nodeName:}" failed. No retries permitted until 2026-07-31 23:56:47.846935876 +0000 UTC m=+631.814686613 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/fbac4361-0369-4cea-ab12-03a4e4ea1171-proxy-tls") pod "success-200-isvc-32541-predictor-7ff4c79d55-9mbgd" (UID: "fbac4361-0369-4cea-ab12-03a4e4ea1171") : secret "success-200-isvc-32541-predictor-serving-cert" not found
Jul 31 23:56:47.347626 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:56:47.347604    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fbac4361-0369-4cea-ab12-03a4e4ea1171-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-32541-predictor-7ff4c79d55-9mbgd\" (UID: \"fbac4361-0369-4cea-ab12-03a4e4ea1171\") " pod="kserve-ci-e2e-test/success-200-isvc-32541-predictor-7ff4c79d55-9mbgd"
Jul 31 23:56:47.356520 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:56:47.356491    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z5fvl\" (UniqueName: \"kubernetes.io/projected/fbac4361-0369-4cea-ab12-03a4e4ea1171-kube-api-access-z5fvl\") pod \"success-200-isvc-32541-predictor-7ff4c79d55-9mbgd\" (UID: \"fbac4361-0369-4cea-ab12-03a4e4ea1171\") " pod="kserve-ci-e2e-test/success-200-isvc-32541-predictor-7ff4c79d55-9mbgd"
Jul 31 23:56:47.395226 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:56:47.395189    2578 generic.go:356] "Generic (PLEG): container finished" podID="3d9b874f-5a27-498b-93bc-d0771624ffd9" containerID="e76035c682350428c0ab02e7a7f883467d453f15ca69ea8ec0a8da1a2ad54d2e" exitCode=2
Jul 31 23:56:47.395226 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:56:47.395212    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-716dc-predictor-6b5b687b9f-dvfl2" event={"ID":"3d9b874f-5a27-498b-93bc-d0771624ffd9","Type":"ContainerDied","Data":"e76035c682350428c0ab02e7a7f883467d453f15ca69ea8ec0a8da1a2ad54d2e"}
Jul 31 23:56:47.850323 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:56:47.850279    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fbac4361-0369-4cea-ab12-03a4e4ea1171-proxy-tls\") pod \"success-200-isvc-32541-predictor-7ff4c79d55-9mbgd\" (UID: \"fbac4361-0369-4cea-ab12-03a4e4ea1171\") " pod="kserve-ci-e2e-test/success-200-isvc-32541-predictor-7ff4c79d55-9mbgd"
Jul 31 23:56:47.852747 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:56:47.852726    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fbac4361-0369-4cea-ab12-03a4e4ea1171-proxy-tls\") pod \"success-200-isvc-32541-predictor-7ff4c79d55-9mbgd\" (UID: \"fbac4361-0369-4cea-ab12-03a4e4ea1171\") " pod="kserve-ci-e2e-test/success-200-isvc-32541-predictor-7ff4c79d55-9mbgd"
Jul 31 23:56:48.148233 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:56:48.148115    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-32541-predictor-7ff4c79d55-9mbgd"
Jul 31 23:56:48.279589 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:56:48.279538    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-32541-predictor-7ff4c79d55-9mbgd"]
Jul 31 23:56:48.281959 ip-10-0-136-108 kubenswrapper[2578]: W0731 23:56:48.281932    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfbac4361_0369_4cea_ab12_03a4e4ea1171.slice/crio-08b3b302d4a426b0787e2668071094e755b45096966634284dcc8a357a74010e WatchSource:0}: Error finding container 08b3b302d4a426b0787e2668071094e755b45096966634284dcc8a357a74010e: Status 404 returned error can't find the container with id 08b3b302d4a426b0787e2668071094e755b45096966634284dcc8a357a74010e
Jul 31 23:56:48.401155 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:56:48.401078    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-32541-predictor-7ff4c79d55-9mbgd" event={"ID":"fbac4361-0369-4cea-ab12-03a4e4ea1171","Type":"ContainerStarted","Data":"dd6ffc8a649db82dc9b5264b31424cb8cf0eb3d7b80a7dcab67190bcb7f19ce3"}
Jul 31 23:56:48.401155 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:56:48.401147    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-32541-predictor-7ff4c79d55-9mbgd" event={"ID":"fbac4361-0369-4cea-ab12-03a4e4ea1171","Type":"ContainerStarted","Data":"08b3b302d4a426b0787e2668071094e755b45096966634284dcc8a357a74010e"}
Jul 31 23:56:48.467728 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:56:48.467694    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gttdm_b32e6d42-e4e0-4113-b8ce-f5d5c5dc13f7/konnectivity-agent/0.log" limit=1024
Jul 31 23:56:49.052983 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:56:49.052943    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-716dc-predictor-6b5b687b9f-dvfl2" podUID="3d9b874f-5a27-498b-93bc-d0771624ffd9" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.30:8643/healthz\": dial tcp 10.133.0.30:8643: connect: connection refused"
Jul 31 23:56:49.058924 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:56:49.058893    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-716dc-predictor-6b5b687b9f-dvfl2" podUID="3d9b874f-5a27-498b-93bc-d0771624ffd9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.30:8080: connect: connection refused"
Jul 31 23:56:49.406799 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:56:49.406712    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-32541-predictor-7ff4c79d55-9mbgd" event={"ID":"fbac4361-0369-4cea-ab12-03a4e4ea1171","Type":"ContainerStarted","Data":"1e2705e8be8dc7917c32c232fbb3032eeea23fc7787005fce270947a90260746"}
Jul 31 23:56:49.407257 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:56:49.406856    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-32541-predictor-7ff4c79d55-9mbgd"
Jul 31 23:56:49.424768 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:56:49.424718    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/success-200-isvc-32541-predictor-7ff4c79d55-9mbgd" podStartSLOduration=2.424702427 podStartE2EDuration="2.424702427s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 23:56:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 23:56:49.422725053 +0000 UTC m=+633.390475813" watchObservedRunningTime="2026-07-31 23:56:49.424702427 +0000 UTC m=+633.392453185"
Jul 31 23:56:50.249454 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:56:50.249419    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gttdm_b32e6d42-e4e0-4113-b8ce-f5d5c5dc13f7/konnectivity-agent/0.log" limit=1024
Jul 31 23:56:50.304855 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:56:50.304816    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gttdm_b32e6d42-e4e0-4113-b8ce-f5d5c5dc13f7/konnectivity-agent/0.log" limit=1024
Jul 31 23:56:50.366358 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:56:50.366321    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-716dc-7d7ffc79d4-2bdn6" podUID="161566e9-254c-4c2d-aa6c-351f0d666a07" containerName="switch-graph-716dc" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 31 23:56:50.413940 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:56:50.413907    2578 generic.go:356] "Generic (PLEG): container finished" podID="3d9b874f-5a27-498b-93bc-d0771624ffd9" containerID="a0406961cf3d9dbce5acd2e7d34c4ebffa6f893f7ef20b6ff14e2e0c3e2bb315" exitCode=0
Jul 31 23:56:50.414351 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:56:50.413975    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-716dc-predictor-6b5b687b9f-dvfl2" event={"ID":"3d9b874f-5a27-498b-93bc-d0771624ffd9","Type":"ContainerDied","Data":"a0406961cf3d9dbce5acd2e7d34c4ebffa6f893f7ef20b6ff14e2e0c3e2bb315"}
Jul 31 23:56:50.414351 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:56:50.414290    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-32541-predictor-7ff4c79d55-9mbgd"
Jul 31 23:56:50.415844 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:56:50.415811    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-32541-predictor-7ff4c79d55-9mbgd" podUID="fbac4361-0369-4cea-ab12-03a4e4ea1171" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.32:8080: connect: connection refused"
Jul 31 23:56:50.510001 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:56:50.509942    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-716dc-predictor-6b5b687b9f-dvfl2"
Jul 31 23:56:50.572001 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:56:50.571969    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3d9b874f-5a27-498b-93bc-d0771624ffd9-kube-rbac-proxy-sar-config\") pod \"3d9b874f-5a27-498b-93bc-d0771624ffd9\" (UID: \"3d9b874f-5a27-498b-93bc-d0771624ffd9\") "
Jul 31 23:56:50.572187 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:56:50.572020    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6h92x\" (UniqueName: \"kubernetes.io/projected/3d9b874f-5a27-498b-93bc-d0771624ffd9-kube-api-access-6h92x\") pod \"3d9b874f-5a27-498b-93bc-d0771624ffd9\" (UID: \"3d9b874f-5a27-498b-93bc-d0771624ffd9\") "
Jul 31 23:56:50.572187 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:56:50.572044    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3d9b874f-5a27-498b-93bc-d0771624ffd9-proxy-tls\") pod \"3d9b874f-5a27-498b-93bc-d0771624ffd9\" (UID: \"3d9b874f-5a27-498b-93bc-d0771624ffd9\") "
Jul 31 23:56:50.572467 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:56:50.572437    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3d9b874f-5a27-498b-93bc-d0771624ffd9-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "3d9b874f-5a27-498b-93bc-d0771624ffd9" (UID: "3d9b874f-5a27-498b-93bc-d0771624ffd9"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 31 23:56:50.574312 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:56:50.574280    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3d9b874f-5a27-498b-93bc-d0771624ffd9-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "3d9b874f-5a27-498b-93bc-d0771624ffd9" (UID: "3d9b874f-5a27-498b-93bc-d0771624ffd9"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 31 23:56:50.574422 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:56:50.574394    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3d9b874f-5a27-498b-93bc-d0771624ffd9-kube-api-access-6h92x" (OuterVolumeSpecName: "kube-api-access-6h92x") pod "3d9b874f-5a27-498b-93bc-d0771624ffd9" (UID: "3d9b874f-5a27-498b-93bc-d0771624ffd9"). InnerVolumeSpecName "kube-api-access-6h92x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 31 23:56:50.673542 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:56:50.673507    2578 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3d9b874f-5a27-498b-93bc-d0771624ffd9-kube-rbac-proxy-sar-config\") on node \"ip-10-0-136-108.ec2.internal\" DevicePath \"\""
Jul 31 23:56:50.673542 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:56:50.673537    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6h92x\" (UniqueName: \"kubernetes.io/projected/3d9b874f-5a27-498b-93bc-d0771624ffd9-kube-api-access-6h92x\") on node \"ip-10-0-136-108.ec2.internal\" DevicePath \"\""
Jul 31 23:56:50.673542 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:56:50.673551    2578 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3d9b874f-5a27-498b-93bc-d0771624ffd9-proxy-tls\") on node \"ip-10-0-136-108.ec2.internal\" DevicePath \"\""
Jul 31 23:56:51.419966 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:56:51.419938    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-716dc-predictor-6b5b687b9f-dvfl2"
Jul 31 23:56:51.420460 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:56:51.419979    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-716dc-predictor-6b5b687b9f-dvfl2" event={"ID":"3d9b874f-5a27-498b-93bc-d0771624ffd9","Type":"ContainerDied","Data":"717088ae212ab1dc58238781f7856aa47f4de3e4a6684b782c0afa5922c2bfa5"}
Jul 31 23:56:51.420460 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:56:51.420051    2578 scope.go:117] "RemoveContainer" containerID="e76035c682350428c0ab02e7a7f883467d453f15ca69ea8ec0a8da1a2ad54d2e"
Jul 31 23:56:51.420581 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:56:51.420544    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-32541-predictor-7ff4c79d55-9mbgd" podUID="fbac4361-0369-4cea-ab12-03a4e4ea1171" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.32:8080: connect: connection refused"
Jul 31 23:56:51.428995 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:56:51.428975    2578 scope.go:117] "RemoveContainer" containerID="a0406961cf3d9dbce5acd2e7d34c4ebffa6f893f7ef20b6ff14e2e0c3e2bb315"
Jul 31 23:56:51.440586 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:56:51.440564    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-716dc-predictor-6b5b687b9f-dvfl2"]
Jul 31 23:56:51.445858 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:56:51.445836    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-716dc-predictor-6b5b687b9f-dvfl2"]
Jul 31 23:56:52.489267 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:56:52.489239    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3d9b874f-5a27-498b-93bc-d0771624ffd9" path="/var/lib/kubelet/pods/3d9b874f-5a27-498b-93bc-d0771624ffd9/volumes"
Jul 31 23:56:55.366501 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:56:55.366463    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-716dc-7d7ffc79d4-2bdn6" podUID="161566e9-254c-4c2d-aa6c-351f0d666a07" containerName="switch-graph-716dc" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 31 23:56:56.425351 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:56:56.425319    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-32541-predictor-7ff4c79d55-9mbgd"
Jul 31 23:56:56.425947 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:56:56.425920    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-32541-predictor-7ff4c79d55-9mbgd" podUID="fbac4361-0369-4cea-ab12-03a4e4ea1171" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.32:8080: connect: connection refused"
Jul 31 23:57:00.366311 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:57:00.366265    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-716dc-7d7ffc79d4-2bdn6" podUID="161566e9-254c-4c2d-aa6c-351f0d666a07" containerName="switch-graph-716dc" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 31 23:57:00.366767 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:57:00.366375    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/switch-graph-716dc-7d7ffc79d4-2bdn6"
Jul 31 23:57:05.366564 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:57:05.366517    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-716dc-7d7ffc79d4-2bdn6" podUID="161566e9-254c-4c2d-aa6c-351f0d666a07" containerName="switch-graph-716dc" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 31 23:57:06.426439 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:57:06.426392    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-32541-predictor-7ff4c79d55-9mbgd" podUID="fbac4361-0369-4cea-ab12-03a4e4ea1171" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.32:8080: connect: connection refused"
Jul 31 23:57:10.366215 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:57:10.366153    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-716dc-7d7ffc79d4-2bdn6" podUID="161566e9-254c-4c2d-aa6c-351f0d666a07" containerName="switch-graph-716dc" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 31 23:57:15.366684 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:57:15.366642    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-716dc-7d7ffc79d4-2bdn6" podUID="161566e9-254c-4c2d-aa6c-351f0d666a07" containerName="switch-graph-716dc" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 31 23:57:16.426825 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:57:16.426790    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-32541-predictor-7ff4c79d55-9mbgd" podUID="fbac4361-0369-4cea-ab12-03a4e4ea1171" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.32:8080: connect: connection refused"
Jul 31 23:57:17.544141 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:57:17.544103    2578 generic.go:356] "Generic (PLEG): container finished" podID="161566e9-254c-4c2d-aa6c-351f0d666a07" containerID="2d3c68192a8740af48c4833efc21b8ae557dd7d4825998826521c85cdd3778d9" exitCode=0
Jul 31 23:57:17.544556 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:57:17.544193    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/switch-graph-716dc-7d7ffc79d4-2bdn6" event={"ID":"161566e9-254c-4c2d-aa6c-351f0d666a07","Type":"ContainerDied","Data":"2d3c68192a8740af48c4833efc21b8ae557dd7d4825998826521c85cdd3778d9"}
Jul 31 23:57:17.682964 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:57:17.682936    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/switch-graph-716dc-7d7ffc79d4-2bdn6"
Jul 31 23:57:17.806798 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:57:17.806712    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/161566e9-254c-4c2d-aa6c-351f0d666a07-openshift-service-ca-bundle\") pod \"161566e9-254c-4c2d-aa6c-351f0d666a07\" (UID: \"161566e9-254c-4c2d-aa6c-351f0d666a07\") "
Jul 31 23:57:17.806974 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:57:17.806803    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/161566e9-254c-4c2d-aa6c-351f0d666a07-proxy-tls\") pod \"161566e9-254c-4c2d-aa6c-351f0d666a07\" (UID: \"161566e9-254c-4c2d-aa6c-351f0d666a07\") "
Jul 31 23:57:17.807154 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:57:17.807126    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/161566e9-254c-4c2d-aa6c-351f0d666a07-openshift-service-ca-bundle" (OuterVolumeSpecName: "openshift-service-ca-bundle") pod "161566e9-254c-4c2d-aa6c-351f0d666a07" (UID: "161566e9-254c-4c2d-aa6c-351f0d666a07"). InnerVolumeSpecName "openshift-service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 31 23:57:17.809011 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:57:17.808986    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/161566e9-254c-4c2d-aa6c-351f0d666a07-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "161566e9-254c-4c2d-aa6c-351f0d666a07" (UID: "161566e9-254c-4c2d-aa6c-351f0d666a07"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 31 23:57:17.907887 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:57:17.907848    2578 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/161566e9-254c-4c2d-aa6c-351f0d666a07-openshift-service-ca-bundle\") on node \"ip-10-0-136-108.ec2.internal\" DevicePath \"\""
Jul 31 23:57:17.907887 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:57:17.907890    2578 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/161566e9-254c-4c2d-aa6c-351f0d666a07-proxy-tls\") on node \"ip-10-0-136-108.ec2.internal\" DevicePath \"\""
Jul 31 23:57:18.550355 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:57:18.550328    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/switch-graph-716dc-7d7ffc79d4-2bdn6"
Jul 31 23:57:18.550749 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:57:18.550322    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/switch-graph-716dc-7d7ffc79d4-2bdn6" event={"ID":"161566e9-254c-4c2d-aa6c-351f0d666a07","Type":"ContainerDied","Data":"7a49c123f04da46f4ab9cb0b72546216eb761e352e21c52c68e09812060eeb89"}
Jul 31 23:57:18.550749 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:57:18.550459    2578 scope.go:117] "RemoveContainer" containerID="2d3c68192a8740af48c4833efc21b8ae557dd7d4825998826521c85cdd3778d9"
Jul 31 23:57:18.564062 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:57:18.564035    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/switch-graph-716dc-7d7ffc79d4-2bdn6"]
Jul 31 23:57:18.566100 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:57:18.566080    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/switch-graph-716dc-7d7ffc79d4-2bdn6"]
Jul 31 23:57:20.489248 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:57:20.489211    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="161566e9-254c-4c2d-aa6c-351f0d666a07" path="/var/lib/kubelet/pods/161566e9-254c-4c2d-aa6c-351f0d666a07/volumes"
Jul 31 23:57:26.426139 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:57:26.426102    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-32541-predictor-7ff4c79d55-9mbgd" podUID="fbac4361-0369-4cea-ab12-03a4e4ea1171" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.32:8080: connect: connection refused"
Jul 31 23:57:26.903830 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:57:26.903793    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gttdm_b32e6d42-e4e0-4113-b8ce-f5d5c5dc13f7/konnectivity-agent/0.log" limit=1024
Jul 31 23:57:26.904339 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:57:26.904319    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/model-chainer-65d5b44868-4ffg2"]
Jul 31 23:57:26.904681 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:57:26.904670    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="161566e9-254c-4c2d-aa6c-351f0d666a07" containerName="switch-graph-716dc"
Jul 31 23:57:26.904722 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:57:26.904683    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="161566e9-254c-4c2d-aa6c-351f0d666a07" containerName="switch-graph-716dc"
Jul 31 23:57:26.904722 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:57:26.904696    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3d9b874f-5a27-498b-93bc-d0771624ffd9" containerName="kube-rbac-proxy"
Jul 31 23:57:26.904722 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:57:26.904701    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="3d9b874f-5a27-498b-93bc-d0771624ffd9" containerName="kube-rbac-proxy"
Jul 31 23:57:26.904722 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:57:26.904709    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3d9b874f-5a27-498b-93bc-d0771624ffd9" containerName="kserve-container"
Jul 31 23:57:26.904722 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:57:26.904715    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="3d9b874f-5a27-498b-93bc-d0771624ffd9" containerName="kserve-container"
Jul 31 23:57:26.904902 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:57:26.904767    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="3d9b874f-5a27-498b-93bc-d0771624ffd9" containerName="kserve-container"
Jul 31 23:57:26.904902 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:57:26.904777    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="3d9b874f-5a27-498b-93bc-d0771624ffd9" containerName="kube-rbac-proxy"
Jul 31 23:57:26.904902 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:57:26.904784    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="161566e9-254c-4c2d-aa6c-351f0d666a07" containerName="switch-graph-716dc"
Jul 31 23:57:26.908984 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:57:26.908967    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/model-chainer-65d5b44868-4ffg2"
Jul 31 23:57:26.910957 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:57:26.910925    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"model-chainer-serving-cert\""
Jul 31 23:57:26.911073 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:57:26.910999    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"model-chainer-kube-rbac-proxy-sar-config\""
Jul 31 23:57:26.918007 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:57:26.917980    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/model-chainer-65d5b44868-4ffg2"]
Jul 31 23:57:26.989770 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:57:26.989733    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e33a2914-6477-4923-8476-3987cd98783a-openshift-service-ca-bundle\") pod \"model-chainer-65d5b44868-4ffg2\" (UID: \"e33a2914-6477-4923-8476-3987cd98783a\") " pod="kserve-ci-e2e-test/model-chainer-65d5b44868-4ffg2"
Jul 31 23:57:26.989933 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:57:26.989816    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e33a2914-6477-4923-8476-3987cd98783a-proxy-tls\") pod \"model-chainer-65d5b44868-4ffg2\" (UID: \"e33a2914-6477-4923-8476-3987cd98783a\") " pod="kserve-ci-e2e-test/model-chainer-65d5b44868-4ffg2"
Jul 31 23:57:27.090676 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:57:27.090638    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e33a2914-6477-4923-8476-3987cd98783a-openshift-service-ca-bundle\") pod \"model-chainer-65d5b44868-4ffg2\" (UID: \"e33a2914-6477-4923-8476-3987cd98783a\") " pod="kserve-ci-e2e-test/model-chainer-65d5b44868-4ffg2"
Jul 31 23:57:27.090852 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:57:27.090733    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e33a2914-6477-4923-8476-3987cd98783a-proxy-tls\") pod \"model-chainer-65d5b44868-4ffg2\" (UID: \"e33a2914-6477-4923-8476-3987cd98783a\") " pod="kserve-ci-e2e-test/model-chainer-65d5b44868-4ffg2"
Jul 31 23:57:27.091414 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:57:27.091387    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e33a2914-6477-4923-8476-3987cd98783a-openshift-service-ca-bundle\") pod \"model-chainer-65d5b44868-4ffg2\" (UID: \"e33a2914-6477-4923-8476-3987cd98783a\") " pod="kserve-ci-e2e-test/model-chainer-65d5b44868-4ffg2"
Jul 31 23:57:27.093219 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:57:27.093199    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e33a2914-6477-4923-8476-3987cd98783a-proxy-tls\") pod \"model-chainer-65d5b44868-4ffg2\" (UID: \"e33a2914-6477-4923-8476-3987cd98783a\") " pod="kserve-ci-e2e-test/model-chainer-65d5b44868-4ffg2"
Jul 31 23:57:27.098987 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:57:27.098945    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gttdm_b32e6d42-e4e0-4113-b8ce-f5d5c5dc13f7/konnectivity-agent/0.log" limit=1024
Jul 31 23:57:27.221639 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:57:27.221548    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/model-chainer-65d5b44868-4ffg2"
Jul 31 23:57:27.350784 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:57:27.350693    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/model-chainer-65d5b44868-4ffg2"]
Jul 31 23:57:27.353503 ip-10-0-136-108 kubenswrapper[2578]: W0731 23:57:27.353473    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode33a2914_6477_4923_8476_3987cd98783a.slice/crio-ecb8e3665ea10f8c7214be4fe321d5be722c59bb8898dbb62825273af92f750f WatchSource:0}: Error finding container ecb8e3665ea10f8c7214be4fe321d5be722c59bb8898dbb62825273af92f750f: Status 404 returned error can't find the container with id ecb8e3665ea10f8c7214be4fe321d5be722c59bb8898dbb62825273af92f750f
Jul 31 23:57:27.594255 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:57:27.594211    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/model-chainer-65d5b44868-4ffg2" event={"ID":"e33a2914-6477-4923-8476-3987cd98783a","Type":"ContainerStarted","Data":"fd21722b6dfd89a80b00c6e0ad7cc005cb138f6d9a1853bf80aad249d647ea99"}
Jul 31 23:57:27.594713 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:57:27.594297    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/model-chainer-65d5b44868-4ffg2" event={"ID":"e33a2914-6477-4923-8476-3987cd98783a","Type":"ContainerStarted","Data":"ecb8e3665ea10f8c7214be4fe321d5be722c59bb8898dbb62825273af92f750f"}
Jul 31 23:57:27.594713 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:57:27.594319    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/model-chainer-65d5b44868-4ffg2"
Jul 31 23:57:27.609680 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:57:27.609625    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/model-chainer-65d5b44868-4ffg2" podStartSLOduration=1.609606547 podStartE2EDuration="1.609606547s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 23:57:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 23:57:27.608792096 +0000 UTC m=+671.576542855" watchObservedRunningTime="2026-07-31 23:57:27.609606547 +0000 UTC m=+671.577357313"
Jul 31 23:57:33.604493 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:57:33.604465    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/model-chainer-65d5b44868-4ffg2"
Jul 31 23:57:36.427235 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:57:36.427203    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-32541-predictor-7ff4c79d55-9mbgd"
Jul 31 23:57:37.010786 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:57:37.010755    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/model-chainer-65d5b44868-4ffg2"]
Jul 31 23:57:37.010953 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:57:37.010925    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/model-chainer-65d5b44868-4ffg2" podUID="e33a2914-6477-4923-8476-3987cd98783a" containerName="model-chainer" containerID="cri-o://fd21722b6dfd89a80b00c6e0ad7cc005cb138f6d9a1853bf80aad249d647ea99" gracePeriod=30
Jul 31 23:57:37.049409 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:57:37.049377    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gttdm_b32e6d42-e4e0-4113-b8ce-f5d5c5dc13f7/konnectivity-agent/0.log" limit=1024
Jul 31 23:57:37.141887 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:57:37.141854    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gttdm_b32e6d42-e4e0-4113-b8ce-f5d5c5dc13f7/konnectivity-agent/0.log" limit=1024
Jul 31 23:57:37.222236 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:57:37.222200    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gttdm_b32e6d42-e4e0-4113-b8ce-f5d5c5dc13f7/konnectivity-agent/0.log" limit=1024
Jul 31 23:57:37.226613 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:57:37.226583    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-a3ab7-predictor-7544d96d6f-m4qb4"]
Jul 31 23:57:37.230319 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:57:37.230299    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-a3ab7-predictor-7544d96d6f-m4qb4"
Jul 31 23:57:37.232382 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:57:37.232361    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-a3ab7-kube-rbac-proxy-sar-config\""
Jul 31 23:57:37.232633 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:57:37.232618    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-a3ab7-predictor-serving-cert\""
Jul 31 23:57:37.238915 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:57:37.238892    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-a3ab7-predictor-7544d96d6f-m4qb4"]
Jul 31 23:57:37.321030 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:57:37.320946    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gttdm_b32e6d42-e4e0-4113-b8ce-f5d5c5dc13f7/konnectivity-agent/0.log" limit=1024
Jul 31 23:57:37.385440 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:57:37.385406    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-twb84\" (UniqueName: \"kubernetes.io/projected/15f7d851-03e8-46bc-b59c-1b9ef87d6847-kube-api-access-twb84\") pod \"success-200-isvc-a3ab7-predictor-7544d96d6f-m4qb4\" (UID: \"15f7d851-03e8-46bc-b59c-1b9ef87d6847\") " pod="kserve-ci-e2e-test/success-200-isvc-a3ab7-predictor-7544d96d6f-m4qb4"
Jul 31 23:57:37.385613 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:57:37.385446    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/15f7d851-03e8-46bc-b59c-1b9ef87d6847-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-a3ab7-predictor-7544d96d6f-m4qb4\" (UID: \"15f7d851-03e8-46bc-b59c-1b9ef87d6847\") " pod="kserve-ci-e2e-test/success-200-isvc-a3ab7-predictor-7544d96d6f-m4qb4"
Jul 31 23:57:37.385613 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:57:37.385517    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/15f7d851-03e8-46bc-b59c-1b9ef87d6847-proxy-tls\") pod \"success-200-isvc-a3ab7-predictor-7544d96d6f-m4qb4\" (UID: \"15f7d851-03e8-46bc-b59c-1b9ef87d6847\") " pod="kserve-ci-e2e-test/success-200-isvc-a3ab7-predictor-7544d96d6f-m4qb4"
Jul 31 23:57:37.418090 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:57:37.418052    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gttdm_b32e6d42-e4e0-4113-b8ce-f5d5c5dc13f7/konnectivity-agent/0.log" limit=1024
Jul 31 23:57:37.485916 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:57:37.485881    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/15f7d851-03e8-46bc-b59c-1b9ef87d6847-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-a3ab7-predictor-7544d96d6f-m4qb4\" (UID: \"15f7d851-03e8-46bc-b59c-1b9ef87d6847\") " pod="kserve-ci-e2e-test/success-200-isvc-a3ab7-predictor-7544d96d6f-m4qb4"
Jul 31 23:57:37.486369 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:57:37.485944    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/15f7d851-03e8-46bc-b59c-1b9ef87d6847-proxy-tls\") pod \"success-200-isvc-a3ab7-predictor-7544d96d6f-m4qb4\" (UID: \"15f7d851-03e8-46bc-b59c-1b9ef87d6847\") " pod="kserve-ci-e2e-test/success-200-isvc-a3ab7-predictor-7544d96d6f-m4qb4"
Jul 31 23:57:37.486369 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:57:37.486008    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-twb84\" (UniqueName: \"kubernetes.io/projected/15f7d851-03e8-46bc-b59c-1b9ef87d6847-kube-api-access-twb84\") pod \"success-200-isvc-a3ab7-predictor-7544d96d6f-m4qb4\" (UID: \"15f7d851-03e8-46bc-b59c-1b9ef87d6847\") " pod="kserve-ci-e2e-test/success-200-isvc-a3ab7-predictor-7544d96d6f-m4qb4"
Jul 31 23:57:37.486369 ip-10-0-136-108 kubenswrapper[2578]: E0731 23:57:37.486279    2578 secret.go:189] Couldn't get secret kserve-ci-e2e-test/success-200-isvc-a3ab7-predictor-serving-cert: secret "success-200-isvc-a3ab7-predictor-serving-cert" not found
Jul 31 23:57:37.486369 ip-10-0-136-108 kubenswrapper[2578]: E0731 23:57:37.486347    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/15f7d851-03e8-46bc-b59c-1b9ef87d6847-proxy-tls podName:15f7d851-03e8-46bc-b59c-1b9ef87d6847 nodeName:}" failed. No retries permitted until 2026-07-31 23:57:37.986328962 +0000 UTC m=+681.954079713 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/15f7d851-03e8-46bc-b59c-1b9ef87d6847-proxy-tls") pod "success-200-isvc-a3ab7-predictor-7544d96d6f-m4qb4" (UID: "15f7d851-03e8-46bc-b59c-1b9ef87d6847") : secret "success-200-isvc-a3ab7-predictor-serving-cert" not found
Jul 31 23:57:37.486678 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:57:37.486658    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/15f7d851-03e8-46bc-b59c-1b9ef87d6847-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-a3ab7-predictor-7544d96d6f-m4qb4\" (UID: \"15f7d851-03e8-46bc-b59c-1b9ef87d6847\") " pod="kserve-ci-e2e-test/success-200-isvc-a3ab7-predictor-7544d96d6f-m4qb4"
Jul 31 23:57:37.494098 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:57:37.494068    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-twb84\" (UniqueName: \"kubernetes.io/projected/15f7d851-03e8-46bc-b59c-1b9ef87d6847-kube-api-access-twb84\") pod \"success-200-isvc-a3ab7-predictor-7544d96d6f-m4qb4\" (UID: \"15f7d851-03e8-46bc-b59c-1b9ef87d6847\") " pod="kserve-ci-e2e-test/success-200-isvc-a3ab7-predictor-7544d96d6f-m4qb4"
Jul 31 23:57:37.584367 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:57:37.584278    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gttdm_b32e6d42-e4e0-4113-b8ce-f5d5c5dc13f7/konnectivity-agent/0.log" limit=1024
Jul 31 23:57:37.692304 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:57:37.692270    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gttdm_b32e6d42-e4e0-4113-b8ce-f5d5c5dc13f7/konnectivity-agent/0.log" limit=1024
Jul 31 23:57:37.991097 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:57:37.991068    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/15f7d851-03e8-46bc-b59c-1b9ef87d6847-proxy-tls\") pod \"success-200-isvc-a3ab7-predictor-7544d96d6f-m4qb4\" (UID: \"15f7d851-03e8-46bc-b59c-1b9ef87d6847\") " pod="kserve-ci-e2e-test/success-200-isvc-a3ab7-predictor-7544d96d6f-m4qb4"
Jul 31 23:57:37.993432 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:57:37.993409    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/15f7d851-03e8-46bc-b59c-1b9ef87d6847-proxy-tls\") pod \"success-200-isvc-a3ab7-predictor-7544d96d6f-m4qb4\" (UID: \"15f7d851-03e8-46bc-b59c-1b9ef87d6847\") " pod="kserve-ci-e2e-test/success-200-isvc-a3ab7-predictor-7544d96d6f-m4qb4"
Jul 31 23:57:38.142604 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:57:38.142563    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-a3ab7-predictor-7544d96d6f-m4qb4"
Jul 31 23:57:38.275991 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:57:38.275964    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-a3ab7-predictor-7544d96d6f-m4qb4"]
Jul 31 23:57:38.278745 ip-10-0-136-108 kubenswrapper[2578]: W0731 23:57:38.278716    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod15f7d851_03e8_46bc_b59c_1b9ef87d6847.slice/crio-3516dfdfc698661746e098ef94a99916d89bd6f269b1bd3560d3fcf83b7fa87d WatchSource:0}: Error finding container 3516dfdfc698661746e098ef94a99916d89bd6f269b1bd3560d3fcf83b7fa87d: Status 404 returned error can't find the container with id 3516dfdfc698661746e098ef94a99916d89bd6f269b1bd3560d3fcf83b7fa87d
Jul 31 23:57:38.603325 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:57:38.603234    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/model-chainer-65d5b44868-4ffg2" podUID="e33a2914-6477-4923-8476-3987cd98783a" containerName="model-chainer" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 31 23:57:38.645144 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:57:38.645101    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-a3ab7-predictor-7544d96d6f-m4qb4" event={"ID":"15f7d851-03e8-46bc-b59c-1b9ef87d6847","Type":"ContainerStarted","Data":"0a18629e275bb4ca18cab9a1409d92eae27932bebc9c4d9d6d96a0be0d9d5536"}
Jul 31 23:57:38.645353 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:57:38.645199    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-a3ab7-predictor-7544d96d6f-m4qb4" event={"ID":"15f7d851-03e8-46bc-b59c-1b9ef87d6847","Type":"ContainerStarted","Data":"363f3c3e5337665b01e370fcafa690c62c213158ef92b3a06525bec94021012c"}
Jul 31 23:57:38.645353 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:57:38.645229    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-a3ab7-predictor-7544d96d6f-m4qb4" event={"ID":"15f7d851-03e8-46bc-b59c-1b9ef87d6847","Type":"ContainerStarted","Data":"3516dfdfc698661746e098ef94a99916d89bd6f269b1bd3560d3fcf83b7fa87d"}
Jul 31 23:57:38.645353 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:57:38.645262    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-a3ab7-predictor-7544d96d6f-m4qb4"
Jul 31 23:57:38.645353 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:57:38.645284    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-a3ab7-predictor-7544d96d6f-m4qb4"
Jul 31 23:57:38.646756 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:57:38.646734    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-a3ab7-predictor-7544d96d6f-m4qb4" podUID="15f7d851-03e8-46bc-b59c-1b9ef87d6847" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.34:8080: connect: connection refused"
Jul 31 23:57:38.661628 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:57:38.661582    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/success-200-isvc-a3ab7-predictor-7544d96d6f-m4qb4" podStartSLOduration=1.6615662 podStartE2EDuration="1.6615662s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 23:57:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 23:57:38.66041021 +0000 UTC m=+682.628160967" watchObservedRunningTime="2026-07-31 23:57:38.6615662 +0000 UTC m=+682.629316958"
Jul 31 23:57:39.649417 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:57:39.649380    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-a3ab7-predictor-7544d96d6f-m4qb4" podUID="15f7d851-03e8-46bc-b59c-1b9ef87d6847" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.34:8080: connect: connection refused"
Jul 31 23:57:43.602659 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:57:43.602618    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/model-chainer-65d5b44868-4ffg2" podUID="e33a2914-6477-4923-8476-3987cd98783a" containerName="model-chainer" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 31 23:57:44.654038 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:57:44.654010    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-a3ab7-predictor-7544d96d6f-m4qb4"
Jul 31 23:57:44.654470 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:57:44.654445    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-a3ab7-predictor-7544d96d6f-m4qb4" podUID="15f7d851-03e8-46bc-b59c-1b9ef87d6847" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.34:8080: connect: connection refused"
Jul 31 23:57:47.275385 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:57:47.275349    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/switch-graph-32541-775bd66797-x4tlp"]
Jul 31 23:57:47.278783 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:57:47.278756    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/switch-graph-32541-775bd66797-x4tlp"
Jul 31 23:57:47.280131 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:57:47.280102    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gttdm_b32e6d42-e4e0-4113-b8ce-f5d5c5dc13f7/konnectivity-agent/0.log" limit=1024
Jul 31 23:57:47.280688 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:57:47.280671    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"switch-graph-32541-serving-cert\""
Jul 31 23:57:47.280767 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:57:47.280671    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"switch-graph-32541-kube-rbac-proxy-sar-config\""
Jul 31 23:57:47.288694 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:57:47.288668    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/switch-graph-32541-775bd66797-x4tlp"]
Jul 31 23:57:47.377115 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:57:47.377084    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/28872143-734e-478c-9000-be449b6d1734-proxy-tls\") pod \"switch-graph-32541-775bd66797-x4tlp\" (UID: \"28872143-734e-478c-9000-be449b6d1734\") " pod="kserve-ci-e2e-test/switch-graph-32541-775bd66797-x4tlp"
Jul 31 23:57:47.377349 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:57:47.377247    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/28872143-734e-478c-9000-be449b6d1734-openshift-service-ca-bundle\") pod \"switch-graph-32541-775bd66797-x4tlp\" (UID: \"28872143-734e-478c-9000-be449b6d1734\") " pod="kserve-ci-e2e-test/switch-graph-32541-775bd66797-x4tlp"
Jul 31 23:57:47.377522 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:57:47.377464    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gttdm_b32e6d42-e4e0-4113-b8ce-f5d5c5dc13f7/konnectivity-agent/0.log" limit=1024
Jul 31 23:57:47.477948 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:57:47.477911    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/28872143-734e-478c-9000-be449b6d1734-openshift-service-ca-bundle\") pod \"switch-graph-32541-775bd66797-x4tlp\" (UID: \"28872143-734e-478c-9000-be449b6d1734\") " pod="kserve-ci-e2e-test/switch-graph-32541-775bd66797-x4tlp"
Jul 31 23:57:47.478130 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:57:47.477992    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/28872143-734e-478c-9000-be449b6d1734-proxy-tls\") pod \"switch-graph-32541-775bd66797-x4tlp\" (UID: \"28872143-734e-478c-9000-be449b6d1734\") " pod="kserve-ci-e2e-test/switch-graph-32541-775bd66797-x4tlp"
Jul 31 23:57:47.478622 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:57:47.478594    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/28872143-734e-478c-9000-be449b6d1734-openshift-service-ca-bundle\") pod \"switch-graph-32541-775bd66797-x4tlp\" (UID: \"28872143-734e-478c-9000-be449b6d1734\") " pod="kserve-ci-e2e-test/switch-graph-32541-775bd66797-x4tlp"
Jul 31 23:57:47.480607 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:57:47.480583    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/28872143-734e-478c-9000-be449b6d1734-proxy-tls\") pod \"switch-graph-32541-775bd66797-x4tlp\" (UID: \"28872143-734e-478c-9000-be449b6d1734\") " pod="kserve-ci-e2e-test/switch-graph-32541-775bd66797-x4tlp"
Jul 31 23:57:47.590882 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:57:47.590779    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/switch-graph-32541-775bd66797-x4tlp"
Jul 31 23:57:47.728875 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:57:47.728853    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/switch-graph-32541-775bd66797-x4tlp"]
Jul 31 23:57:47.731602 ip-10-0-136-108 kubenswrapper[2578]: W0731 23:57:47.731574    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod28872143_734e_478c_9000_be449b6d1734.slice/crio-8ec487a3493d6e6f8f0a2b5dfaac6c599943845fea53033ef0db77ec13215cdd WatchSource:0}: Error finding container 8ec487a3493d6e6f8f0a2b5dfaac6c599943845fea53033ef0db77ec13215cdd: Status 404 returned error can't find the container with id 8ec487a3493d6e6f8f0a2b5dfaac6c599943845fea53033ef0db77ec13215cdd
Jul 31 23:57:48.602875 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:57:48.602836    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/model-chainer-65d5b44868-4ffg2" podUID="e33a2914-6477-4923-8476-3987cd98783a" containerName="model-chainer" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 31 23:57:48.603320 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:57:48.602938    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/model-chainer-65d5b44868-4ffg2"
Jul 31 23:57:48.693470 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:57:48.693424    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/switch-graph-32541-775bd66797-x4tlp" event={"ID":"28872143-734e-478c-9000-be449b6d1734","Type":"ContainerStarted","Data":"d07591f6cbd3e05e2f2930dd03517d209f99c82f76b7f0cacbc7dd7c166bf090"}
Jul 31 23:57:48.693659 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:57:48.693492    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/switch-graph-32541-775bd66797-x4tlp" event={"ID":"28872143-734e-478c-9000-be449b6d1734","Type":"ContainerStarted","Data":"8ec487a3493d6e6f8f0a2b5dfaac6c599943845fea53033ef0db77ec13215cdd"}
Jul 31 23:57:48.693659 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:57:48.693542    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/switch-graph-32541-775bd66797-x4tlp"
Jul 31 23:57:48.710289 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:57:48.710238    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/switch-graph-32541-775bd66797-x4tlp" podStartSLOduration=1.710219542 podStartE2EDuration="1.710219542s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 23:57:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 23:57:48.709406391 +0000 UTC m=+692.677157148" watchObservedRunningTime="2026-07-31 23:57:48.710219542 +0000 UTC m=+692.677970299"
Jul 31 23:57:53.603197 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:57:53.603145    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/model-chainer-65d5b44868-4ffg2" podUID="e33a2914-6477-4923-8476-3987cd98783a" containerName="model-chainer" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 31 23:57:54.654856 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:57:54.654819    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-a3ab7-predictor-7544d96d6f-m4qb4" podUID="15f7d851-03e8-46bc-b59c-1b9ef87d6847" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.34:8080: connect: connection refused"
Jul 31 23:57:54.704581 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:57:54.704553    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/switch-graph-32541-775bd66797-x4tlp"
Jul 31 23:57:58.603156 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:57:58.603109    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/model-chainer-65d5b44868-4ffg2" podUID="e33a2914-6477-4923-8476-3987cd98783a" containerName="model-chainer" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 31 23:58:03.602765 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:58:03.602717    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/model-chainer-65d5b44868-4ffg2" podUID="e33a2914-6477-4923-8476-3987cd98783a" containerName="model-chainer" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 31 23:58:04.655374 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:58:04.655334    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-a3ab7-predictor-7544d96d6f-m4qb4" podUID="15f7d851-03e8-46bc-b59c-1b9ef87d6847" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.34:8080: connect: connection refused"
Jul 31 23:58:07.658282 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:58:07.658256    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/model-chainer-65d5b44868-4ffg2"
Jul 31 23:58:07.768728 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:58:07.768686    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e33a2914-6477-4923-8476-3987cd98783a-openshift-service-ca-bundle\") pod \"e33a2914-6477-4923-8476-3987cd98783a\" (UID: \"e33a2914-6477-4923-8476-3987cd98783a\") "
Jul 31 23:58:07.768933 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:58:07.768829    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e33a2914-6477-4923-8476-3987cd98783a-proxy-tls\") pod \"e33a2914-6477-4923-8476-3987cd98783a\" (UID: \"e33a2914-6477-4923-8476-3987cd98783a\") "
Jul 31 23:58:07.769068 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:58:07.769043    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e33a2914-6477-4923-8476-3987cd98783a-openshift-service-ca-bundle" (OuterVolumeSpecName: "openshift-service-ca-bundle") pod "e33a2914-6477-4923-8476-3987cd98783a" (UID: "e33a2914-6477-4923-8476-3987cd98783a"). InnerVolumeSpecName "openshift-service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 31 23:58:07.769161 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:58:07.769138    2578 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e33a2914-6477-4923-8476-3987cd98783a-openshift-service-ca-bundle\") on node \"ip-10-0-136-108.ec2.internal\" DevicePath \"\""
Jul 31 23:58:07.770942 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:58:07.770914    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e33a2914-6477-4923-8476-3987cd98783a-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "e33a2914-6477-4923-8476-3987cd98783a" (UID: "e33a2914-6477-4923-8476-3987cd98783a"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 31 23:58:07.781403 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:58:07.781377    2578 generic.go:356] "Generic (PLEG): container finished" podID="e33a2914-6477-4923-8476-3987cd98783a" containerID="fd21722b6dfd89a80b00c6e0ad7cc005cb138f6d9a1853bf80aad249d647ea99" exitCode=0
Jul 31 23:58:07.781538 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:58:07.781423    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/model-chainer-65d5b44868-4ffg2"
Jul 31 23:58:07.781538 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:58:07.781466    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/model-chainer-65d5b44868-4ffg2" event={"ID":"e33a2914-6477-4923-8476-3987cd98783a","Type":"ContainerDied","Data":"fd21722b6dfd89a80b00c6e0ad7cc005cb138f6d9a1853bf80aad249d647ea99"}
Jul 31 23:58:07.781538 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:58:07.781532    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/model-chainer-65d5b44868-4ffg2" event={"ID":"e33a2914-6477-4923-8476-3987cd98783a","Type":"ContainerDied","Data":"ecb8e3665ea10f8c7214be4fe321d5be722c59bb8898dbb62825273af92f750f"}
Jul 31 23:58:07.781719 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:58:07.781565    2578 scope.go:117] "RemoveContainer" containerID="fd21722b6dfd89a80b00c6e0ad7cc005cb138f6d9a1853bf80aad249d647ea99"
Jul 31 23:58:07.790749 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:58:07.790728    2578 scope.go:117] "RemoveContainer" containerID="fd21722b6dfd89a80b00c6e0ad7cc005cb138f6d9a1853bf80aad249d647ea99"
Jul 31 23:58:07.791021 ip-10-0-136-108 kubenswrapper[2578]: E0731 23:58:07.791000    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fd21722b6dfd89a80b00c6e0ad7cc005cb138f6d9a1853bf80aad249d647ea99\": container with ID starting with fd21722b6dfd89a80b00c6e0ad7cc005cb138f6d9a1853bf80aad249d647ea99 not found: ID does not exist" containerID="fd21722b6dfd89a80b00c6e0ad7cc005cb138f6d9a1853bf80aad249d647ea99"
Jul 31 23:58:07.791066 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:58:07.791034    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fd21722b6dfd89a80b00c6e0ad7cc005cb138f6d9a1853bf80aad249d647ea99"} err="failed to get container status \"fd21722b6dfd89a80b00c6e0ad7cc005cb138f6d9a1853bf80aad249d647ea99\": rpc error: code = NotFound desc = could not find container \"fd21722b6dfd89a80b00c6e0ad7cc005cb138f6d9a1853bf80aad249d647ea99\": container with ID starting with fd21722b6dfd89a80b00c6e0ad7cc005cb138f6d9a1853bf80aad249d647ea99 not found: ID does not exist"
Jul 31 23:58:07.801740 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:58:07.801713    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/model-chainer-65d5b44868-4ffg2"]
Jul 31 23:58:07.805321 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:58:07.805302    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/model-chainer-65d5b44868-4ffg2"]
Jul 31 23:58:07.869958 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:58:07.869926    2578 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e33a2914-6477-4923-8476-3987cd98783a-proxy-tls\") on node \"ip-10-0-136-108.ec2.internal\" DevicePath \"\""
Jul 31 23:58:08.489432 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:58:08.489396    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e33a2914-6477-4923-8476-3987cd98783a" path="/var/lib/kubelet/pods/e33a2914-6477-4923-8476-3987cd98783a/volumes"
Jul 31 23:58:14.654665 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:58:14.654627    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-a3ab7-predictor-7544d96d6f-m4qb4" podUID="15f7d851-03e8-46bc-b59c-1b9ef87d6847" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.34:8080: connect: connection refused"
Jul 31 23:58:24.655701 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:58:24.655666    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-a3ab7-predictor-7544d96d6f-m4qb4"
Jul 31 23:58:37.262916 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:58:37.262885    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/sequence-graph-a3ab7-5cc8864447-tbfbv"]
Jul 31 23:58:37.263388 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:58:37.263272    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e33a2914-6477-4923-8476-3987cd98783a" containerName="model-chainer"
Jul 31 23:58:37.263388 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:58:37.263283    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="e33a2914-6477-4923-8476-3987cd98783a" containerName="model-chainer"
Jul 31 23:58:37.263388 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:58:37.263340    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="e33a2914-6477-4923-8476-3987cd98783a" containerName="model-chainer"
Jul 31 23:58:37.267523 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:58:37.267504    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sequence-graph-a3ab7-5cc8864447-tbfbv"
Jul 31 23:58:37.271612 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:58:37.271592    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"sequence-graph-a3ab7-serving-cert\""
Jul 31 23:58:37.272023 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:58:37.272007    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"sequence-graph-a3ab7-kube-rbac-proxy-sar-config\""
Jul 31 23:58:37.273963 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:58:37.273935    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gttdm_b32e6d42-e4e0-4113-b8ce-f5d5c5dc13f7/konnectivity-agent/0.log" limit=1024
Jul 31 23:58:37.278732 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:58:37.278709    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-a3ab7-5cc8864447-tbfbv"]
Jul 31 23:58:37.326377 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:58:37.326341    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fcfc7c13-06ba-4fb4-a473-c3323bc9e4b8-openshift-service-ca-bundle\") pod \"sequence-graph-a3ab7-5cc8864447-tbfbv\" (UID: \"fcfc7c13-06ba-4fb4-a473-c3323bc9e4b8\") " pod="kserve-ci-e2e-test/sequence-graph-a3ab7-5cc8864447-tbfbv"
Jul 31 23:58:37.326574 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:58:37.326456    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fcfc7c13-06ba-4fb4-a473-c3323bc9e4b8-proxy-tls\") pod \"sequence-graph-a3ab7-5cc8864447-tbfbv\" (UID: \"fcfc7c13-06ba-4fb4-a473-c3323bc9e4b8\") " pod="kserve-ci-e2e-test/sequence-graph-a3ab7-5cc8864447-tbfbv"
Jul 31 23:58:37.427135 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:58:37.427093    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fcfc7c13-06ba-4fb4-a473-c3323bc9e4b8-openshift-service-ca-bundle\") pod \"sequence-graph-a3ab7-5cc8864447-tbfbv\" (UID: \"fcfc7c13-06ba-4fb4-a473-c3323bc9e4b8\") " pod="kserve-ci-e2e-test/sequence-graph-a3ab7-5cc8864447-tbfbv"
Jul 31 23:58:37.427328 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:58:37.427231    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fcfc7c13-06ba-4fb4-a473-c3323bc9e4b8-proxy-tls\") pod \"sequence-graph-a3ab7-5cc8864447-tbfbv\" (UID: \"fcfc7c13-06ba-4fb4-a473-c3323bc9e4b8\") " pod="kserve-ci-e2e-test/sequence-graph-a3ab7-5cc8864447-tbfbv"
Jul 31 23:58:37.427376 ip-10-0-136-108 kubenswrapper[2578]: E0731 23:58:37.427352    2578 secret.go:189] Couldn't get secret kserve-ci-e2e-test/sequence-graph-a3ab7-serving-cert: secret "sequence-graph-a3ab7-serving-cert" not found
Jul 31 23:58:37.427471 ip-10-0-136-108 kubenswrapper[2578]: E0731 23:58:37.427457    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fcfc7c13-06ba-4fb4-a473-c3323bc9e4b8-proxy-tls podName:fcfc7c13-06ba-4fb4-a473-c3323bc9e4b8 nodeName:}" failed. No retries permitted until 2026-07-31 23:58:37.927434581 +0000 UTC m=+741.895185337 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/fcfc7c13-06ba-4fb4-a473-c3323bc9e4b8-proxy-tls") pod "sequence-graph-a3ab7-5cc8864447-tbfbv" (UID: "fcfc7c13-06ba-4fb4-a473-c3323bc9e4b8") : secret "sequence-graph-a3ab7-serving-cert" not found
Jul 31 23:58:37.427919 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:58:37.427900    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fcfc7c13-06ba-4fb4-a473-c3323bc9e4b8-openshift-service-ca-bundle\") pod \"sequence-graph-a3ab7-5cc8864447-tbfbv\" (UID: \"fcfc7c13-06ba-4fb4-a473-c3323bc9e4b8\") " pod="kserve-ci-e2e-test/sequence-graph-a3ab7-5cc8864447-tbfbv"
Jul 31 23:58:37.867996 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:58:37.867951    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gttdm_b32e6d42-e4e0-4113-b8ce-f5d5c5dc13f7/konnectivity-agent/0.log" limit=1024
Jul 31 23:58:37.931492 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:58:37.931460    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fcfc7c13-06ba-4fb4-a473-c3323bc9e4b8-proxy-tls\") pod \"sequence-graph-a3ab7-5cc8864447-tbfbv\" (UID: \"fcfc7c13-06ba-4fb4-a473-c3323bc9e4b8\") " pod="kserve-ci-e2e-test/sequence-graph-a3ab7-5cc8864447-tbfbv"
Jul 31 23:58:37.933807 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:58:37.933789    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fcfc7c13-06ba-4fb4-a473-c3323bc9e4b8-proxy-tls\") pod \"sequence-graph-a3ab7-5cc8864447-tbfbv\" (UID: \"fcfc7c13-06ba-4fb4-a473-c3323bc9e4b8\") " pod="kserve-ci-e2e-test/sequence-graph-a3ab7-5cc8864447-tbfbv"
Jul 31 23:58:37.968607 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:58:37.968572    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gttdm_b32e6d42-e4e0-4113-b8ce-f5d5c5dc13f7/konnectivity-agent/0.log" limit=1024
Jul 31 23:58:38.178087 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:58:38.177992    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sequence-graph-a3ab7-5cc8864447-tbfbv"
Jul 31 23:58:38.310802 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:58:38.310774    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-a3ab7-5cc8864447-tbfbv"]
Jul 31 23:58:38.313243 ip-10-0-136-108 kubenswrapper[2578]: W0731 23:58:38.313209    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfcfc7c13_06ba_4fb4_a473_c3323bc9e4b8.slice/crio-225161d1f0e2188d003c3d9035a945036bd107b7b8b7002ef73d9b719c9a865e WatchSource:0}: Error finding container 225161d1f0e2188d003c3d9035a945036bd107b7b8b7002ef73d9b719c9a865e: Status 404 returned error can't find the container with id 225161d1f0e2188d003c3d9035a945036bd107b7b8b7002ef73d9b719c9a865e
Jul 31 23:58:38.345826 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:58:38.345796    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gttdm_b32e6d42-e4e0-4113-b8ce-f5d5c5dc13f7/konnectivity-agent/0.log" limit=1024
Jul 31 23:58:38.936245 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:58:38.936121    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sequence-graph-a3ab7-5cc8864447-tbfbv" event={"ID":"fcfc7c13-06ba-4fb4-a473-c3323bc9e4b8","Type":"ContainerStarted","Data":"e68e8bf27a2ddc3ff68ba9ecef9aab5e1bf8c6f5bf22f497ed519db9147324e0"}
Jul 31 23:58:38.936439 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:58:38.936287    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sequence-graph-a3ab7-5cc8864447-tbfbv" event={"ID":"fcfc7c13-06ba-4fb4-a473-c3323bc9e4b8","Type":"ContainerStarted","Data":"225161d1f0e2188d003c3d9035a945036bd107b7b8b7002ef73d9b719c9a865e"}
Jul 31 23:58:38.936504 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:58:38.936474    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/sequence-graph-a3ab7-5cc8864447-tbfbv"
Jul 31 23:58:38.952497 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:58:38.952449    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/sequence-graph-a3ab7-5cc8864447-tbfbv" podStartSLOduration=1.9524339450000001 podStartE2EDuration="1.952433945s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 23:58:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 23:58:38.950990025 +0000 UTC m=+742.918740784" watchObservedRunningTime="2026-07-31 23:58:38.952433945 +0000 UTC m=+742.920184703"
Jul 31 23:58:44.945654 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:58:44.945621    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/sequence-graph-a3ab7-5cc8864447-tbfbv"
Jul 31 23:58:58.998877 ip-10-0-136-108 kubenswrapper[2578]: I0731 23:58:58.998833    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gttdm_b32e6d42-e4e0-4113-b8ce-f5d5c5dc13f7/konnectivity-agent/0.log" limit=1024
Aug 01 00:02:24.755559 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:02:24.755506    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gttdm_b32e6d42-e4e0-4113-b8ce-f5d5c5dc13f7/konnectivity-agent/0.log" limit=1024
Aug 01 00:04:35.257684 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:04:35.257594    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gttdm_b32e6d42-e4e0-4113-b8ce-f5d5c5dc13f7/konnectivity-agent/0.log" limit=1024
Aug 01 00:06:02.104138 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:06:02.104101    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/switch-graph-32541-775bd66797-x4tlp"]
Aug 01 00:06:02.104726 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:06:02.104370    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/switch-graph-32541-775bd66797-x4tlp" podUID="28872143-734e-478c-9000-be449b6d1734" containerName="switch-graph-32541" containerID="cri-o://d07591f6cbd3e05e2f2930dd03517d209f99c82f76b7f0cacbc7dd7c166bf090" gracePeriod=30
Aug 01 00:06:02.132415 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:06:02.132379    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gttdm_b32e6d42-e4e0-4113-b8ce-f5d5c5dc13f7/konnectivity-agent/0.log" limit=1024
Aug 01 00:06:02.237571 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:06:02.237523    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-32541-predictor-7ff4c79d55-9mbgd"]
Aug 01 00:06:02.237834 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:06:02.237805    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-32541-predictor-7ff4c79d55-9mbgd" podUID="fbac4361-0369-4cea-ab12-03a4e4ea1171" containerName="kserve-container" containerID="cri-o://dd6ffc8a649db82dc9b5264b31424cb8cf0eb3d7b80a7dcab67190bcb7f19ce3" gracePeriod=30
Aug 01 00:06:02.237951 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:06:02.237850    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-32541-predictor-7ff4c79d55-9mbgd" podUID="fbac4361-0369-4cea-ab12-03a4e4ea1171" containerName="kube-rbac-proxy" containerID="cri-o://1e2705e8be8dc7917c32c232fbb3032eeea23fc7787005fce270947a90260746" gracePeriod=30
Aug 01 00:06:02.260424 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:06:02.260394    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gttdm_b32e6d42-e4e0-4113-b8ce-f5d5c5dc13f7/konnectivity-agent/0.log" limit=1024
Aug 01 00:06:02.324420 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:06:02.324378    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gttdm_b32e6d42-e4e0-4113-b8ce-f5d5c5dc13f7/konnectivity-agent/0.log" limit=1024
Aug 01 00:06:02.360132 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:06:02.360102    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-fe2ca-predictor-698655d7c5-6h276"]
Aug 01 00:06:02.363911 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:06:02.363888    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-fe2ca-predictor-698655d7c5-6h276"
Aug 01 00:06:02.366053 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:06:02.366027    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-fe2ca-kube-rbac-proxy-sar-config\""
Aug 01 00:06:02.366355 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:06:02.366334    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-fe2ca-predictor-serving-cert\""
Aug 01 00:06:02.371727 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:06:02.371700    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-fe2ca-predictor-698655d7c5-6h276"]
Aug 01 00:06:02.427041 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:06:02.426998    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gttdm_b32e6d42-e4e0-4113-b8ce-f5d5c5dc13f7/konnectivity-agent/0.log" limit=1024
Aug 01 00:06:02.481929 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:06:02.481896    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mp7s4\" (UniqueName: \"kubernetes.io/projected/cde6e7e4-4775-491c-8965-25508d23d619-kube-api-access-mp7s4\") pod \"success-200-isvc-fe2ca-predictor-698655d7c5-6h276\" (UID: \"cde6e7e4-4775-491c-8965-25508d23d619\") " pod="kserve-ci-e2e-test/success-200-isvc-fe2ca-predictor-698655d7c5-6h276"
Aug 01 00:06:02.482157 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:06:02.482056    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/cde6e7e4-4775-491c-8965-25508d23d619-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-fe2ca-predictor-698655d7c5-6h276\" (UID: \"cde6e7e4-4775-491c-8965-25508d23d619\") " pod="kserve-ci-e2e-test/success-200-isvc-fe2ca-predictor-698655d7c5-6h276"
Aug 01 00:06:02.482157 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:06:02.482128    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/cde6e7e4-4775-491c-8965-25508d23d619-proxy-tls\") pod \"success-200-isvc-fe2ca-predictor-698655d7c5-6h276\" (UID: \"cde6e7e4-4775-491c-8965-25508d23d619\") " pod="kserve-ci-e2e-test/success-200-isvc-fe2ca-predictor-698655d7c5-6h276"
Aug 01 00:06:02.525529 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:06:02.525491    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gttdm_b32e6d42-e4e0-4113-b8ce-f5d5c5dc13f7/konnectivity-agent/0.log" limit=1024
Aug 01 00:06:02.583429 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:06:02.583391    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mp7s4\" (UniqueName: \"kubernetes.io/projected/cde6e7e4-4775-491c-8965-25508d23d619-kube-api-access-mp7s4\") pod \"success-200-isvc-fe2ca-predictor-698655d7c5-6h276\" (UID: \"cde6e7e4-4775-491c-8965-25508d23d619\") " pod="kserve-ci-e2e-test/success-200-isvc-fe2ca-predictor-698655d7c5-6h276"
Aug 01 00:06:02.583629 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:06:02.583547    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/cde6e7e4-4775-491c-8965-25508d23d619-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-fe2ca-predictor-698655d7c5-6h276\" (UID: \"cde6e7e4-4775-491c-8965-25508d23d619\") " pod="kserve-ci-e2e-test/success-200-isvc-fe2ca-predictor-698655d7c5-6h276"
Aug 01 00:06:02.583629 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:06:02.583585    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/cde6e7e4-4775-491c-8965-25508d23d619-proxy-tls\") pod \"success-200-isvc-fe2ca-predictor-698655d7c5-6h276\" (UID: \"cde6e7e4-4775-491c-8965-25508d23d619\") " pod="kserve-ci-e2e-test/success-200-isvc-fe2ca-predictor-698655d7c5-6h276"
Aug 01 00:06:02.584273 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:06:02.584250    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/cde6e7e4-4775-491c-8965-25508d23d619-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-fe2ca-predictor-698655d7c5-6h276\" (UID: \"cde6e7e4-4775-491c-8965-25508d23d619\") " pod="kserve-ci-e2e-test/success-200-isvc-fe2ca-predictor-698655d7c5-6h276"
Aug 01 00:06:02.586259 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:06:02.586233    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/cde6e7e4-4775-491c-8965-25508d23d619-proxy-tls\") pod \"success-200-isvc-fe2ca-predictor-698655d7c5-6h276\" (UID: \"cde6e7e4-4775-491c-8965-25508d23d619\") " pod="kserve-ci-e2e-test/success-200-isvc-fe2ca-predictor-698655d7c5-6h276"
Aug 01 00:06:02.591387 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:06:02.591363    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mp7s4\" (UniqueName: \"kubernetes.io/projected/cde6e7e4-4775-491c-8965-25508d23d619-kube-api-access-mp7s4\") pod \"success-200-isvc-fe2ca-predictor-698655d7c5-6h276\" (UID: \"cde6e7e4-4775-491c-8965-25508d23d619\") " pod="kserve-ci-e2e-test/success-200-isvc-fe2ca-predictor-698655d7c5-6h276"
Aug 01 00:06:02.649555 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:06:02.649439    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gttdm_b32e6d42-e4e0-4113-b8ce-f5d5c5dc13f7/konnectivity-agent/0.log" limit=1024
Aug 01 00:06:02.677071 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:06:02.677026    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-fe2ca-predictor-698655d7c5-6h276"
Aug 01 00:06:02.822759 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:06:02.822737    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-fe2ca-predictor-698655d7c5-6h276"]
Aug 01 00:06:02.824980 ip-10-0-136-108 kubenswrapper[2578]: W0801 00:06:02.824956    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcde6e7e4_4775_491c_8965_25508d23d619.slice/crio-d577f75913f67a453e42aeb0f958ba2738ddef62b7e843e60353a7f0c7bcb684 WatchSource:0}: Error finding container d577f75913f67a453e42aeb0f958ba2738ddef62b7e843e60353a7f0c7bcb684: Status 404 returned error can't find the container with id d577f75913f67a453e42aeb0f958ba2738ddef62b7e843e60353a7f0c7bcb684
Aug 01 00:06:02.994785 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:06:02.994747    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-fe2ca-predictor-698655d7c5-6h276" event={"ID":"cde6e7e4-4775-491c-8965-25508d23d619","Type":"ContainerStarted","Data":"d126be418a54e67d940d68ff4a1211511d6ade1c8f1b80e632e3bf8ea2f731d0"}
Aug 01 00:06:02.994972 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:06:02.994815    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-fe2ca-predictor-698655d7c5-6h276" event={"ID":"cde6e7e4-4775-491c-8965-25508d23d619","Type":"ContainerStarted","Data":"7b9772fd84e7f4a2119c7b6e3521f8854d89c955f2580ec7441481976f2be9bc"}
Aug 01 00:06:02.994972 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:06:02.994855    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-fe2ca-predictor-698655d7c5-6h276"
Aug 01 00:06:02.994972 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:06:02.994881    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-fe2ca-predictor-698655d7c5-6h276" event={"ID":"cde6e7e4-4775-491c-8965-25508d23d619","Type":"ContainerStarted","Data":"d577f75913f67a453e42aeb0f958ba2738ddef62b7e843e60353a7f0c7bcb684"}
Aug 01 00:06:02.996715 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:06:02.996686    2578 generic.go:356] "Generic (PLEG): container finished" podID="fbac4361-0369-4cea-ab12-03a4e4ea1171" containerID="1e2705e8be8dc7917c32c232fbb3032eeea23fc7787005fce270947a90260746" exitCode=2
Aug 01 00:06:02.996815 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:06:02.996761    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-32541-predictor-7ff4c79d55-9mbgd" event={"ID":"fbac4361-0369-4cea-ab12-03a4e4ea1171","Type":"ContainerDied","Data":"1e2705e8be8dc7917c32c232fbb3032eeea23fc7787005fce270947a90260746"}
Aug 01 00:06:03.010886 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:06:03.010821    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/success-200-isvc-fe2ca-predictor-698655d7c5-6h276" podStartSLOduration=1.010801582 podStartE2EDuration="1.010801582s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-01 00:06:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-01 00:06:03.010600056 +0000 UTC m=+1186.978350814" watchObservedRunningTime="2026-08-01 00:06:03.010801582 +0000 UTC m=+1186.978552341"
Aug 01 00:06:04.001219 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:06:04.001189    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-fe2ca-predictor-698655d7c5-6h276"
Aug 01 00:06:04.002661 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:06:04.002636    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-fe2ca-predictor-698655d7c5-6h276" podUID="cde6e7e4-4775-491c-8965-25508d23d619" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.37:8080: connect: connection refused"
Aug 01 00:06:04.702846 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:06:04.702808    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-32541-775bd66797-x4tlp" podUID="28872143-734e-478c-9000-be449b6d1734" containerName="switch-graph-32541" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Aug 01 00:06:05.005463 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:06:05.005420    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-fe2ca-predictor-698655d7c5-6h276" podUID="cde6e7e4-4775-491c-8965-25508d23d619" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.37:8080: connect: connection refused"
Aug 01 00:06:05.601073 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:06:05.601048    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-32541-predictor-7ff4c79d55-9mbgd"
Aug 01 00:06:05.710274 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:06:05.710183    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fbac4361-0369-4cea-ab12-03a4e4ea1171-proxy-tls\") pod \"fbac4361-0369-4cea-ab12-03a4e4ea1171\" (UID: \"fbac4361-0369-4cea-ab12-03a4e4ea1171\") "
Aug 01 00:06:05.710274 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:06:05.710228    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-z5fvl\" (UniqueName: \"kubernetes.io/projected/fbac4361-0369-4cea-ab12-03a4e4ea1171-kube-api-access-z5fvl\") pod \"fbac4361-0369-4cea-ab12-03a4e4ea1171\" (UID: \"fbac4361-0369-4cea-ab12-03a4e4ea1171\") "
Aug 01 00:06:05.710511 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:06:05.710353    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fbac4361-0369-4cea-ab12-03a4e4ea1171-kube-rbac-proxy-sar-config\") pod \"fbac4361-0369-4cea-ab12-03a4e4ea1171\" (UID: \"fbac4361-0369-4cea-ab12-03a4e4ea1171\") "
Aug 01 00:06:05.710736 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:06:05.710708    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fbac4361-0369-4cea-ab12-03a4e4ea1171-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "fbac4361-0369-4cea-ab12-03a4e4ea1171" (UID: "fbac4361-0369-4cea-ab12-03a4e4ea1171"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 01 00:06:05.712326 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:06:05.712302    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fbac4361-0369-4cea-ab12-03a4e4ea1171-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "fbac4361-0369-4cea-ab12-03a4e4ea1171" (UID: "fbac4361-0369-4cea-ab12-03a4e4ea1171"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 01 00:06:05.712400 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:06:05.712379    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fbac4361-0369-4cea-ab12-03a4e4ea1171-kube-api-access-z5fvl" (OuterVolumeSpecName: "kube-api-access-z5fvl") pod "fbac4361-0369-4cea-ab12-03a4e4ea1171" (UID: "fbac4361-0369-4cea-ab12-03a4e4ea1171"). InnerVolumeSpecName "kube-api-access-z5fvl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 01 00:06:05.812028 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:06:05.811988    2578 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fbac4361-0369-4cea-ab12-03a4e4ea1171-kube-rbac-proxy-sar-config\") on node \"ip-10-0-136-108.ec2.internal\" DevicePath \"\""
Aug 01 00:06:05.812028 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:06:05.812019    2578 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fbac4361-0369-4cea-ab12-03a4e4ea1171-proxy-tls\") on node \"ip-10-0-136-108.ec2.internal\" DevicePath \"\""
Aug 01 00:06:05.812028 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:06:05.812033    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z5fvl\" (UniqueName: \"kubernetes.io/projected/fbac4361-0369-4cea-ab12-03a4e4ea1171-kube-api-access-z5fvl\") on node \"ip-10-0-136-108.ec2.internal\" DevicePath \"\""
Aug 01 00:06:06.011847 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:06:06.011809    2578 generic.go:356] "Generic (PLEG): container finished" podID="fbac4361-0369-4cea-ab12-03a4e4ea1171" containerID="dd6ffc8a649db82dc9b5264b31424cb8cf0eb3d7b80a7dcab67190bcb7f19ce3" exitCode=0
Aug 01 00:06:06.012291 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:06:06.011858    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-32541-predictor-7ff4c79d55-9mbgd"
Aug 01 00:06:06.012291 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:06:06.011862    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-32541-predictor-7ff4c79d55-9mbgd" event={"ID":"fbac4361-0369-4cea-ab12-03a4e4ea1171","Type":"ContainerDied","Data":"dd6ffc8a649db82dc9b5264b31424cb8cf0eb3d7b80a7dcab67190bcb7f19ce3"}
Aug 01 00:06:06.012291 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:06:06.011914    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-32541-predictor-7ff4c79d55-9mbgd" event={"ID":"fbac4361-0369-4cea-ab12-03a4e4ea1171","Type":"ContainerDied","Data":"08b3b302d4a426b0787e2668071094e755b45096966634284dcc8a357a74010e"}
Aug 01 00:06:06.012291 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:06:06.011941    2578 scope.go:117] "RemoveContainer" containerID="1e2705e8be8dc7917c32c232fbb3032eeea23fc7787005fce270947a90260746"
Aug 01 00:06:06.021278 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:06:06.021253    2578 scope.go:117] "RemoveContainer" containerID="dd6ffc8a649db82dc9b5264b31424cb8cf0eb3d7b80a7dcab67190bcb7f19ce3"
Aug 01 00:06:06.029440 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:06:06.029417    2578 scope.go:117] "RemoveContainer" containerID="1e2705e8be8dc7917c32c232fbb3032eeea23fc7787005fce270947a90260746"
Aug 01 00:06:06.029688 ip-10-0-136-108 kubenswrapper[2578]: E0801 00:06:06.029672    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1e2705e8be8dc7917c32c232fbb3032eeea23fc7787005fce270947a90260746\": container with ID starting with 1e2705e8be8dc7917c32c232fbb3032eeea23fc7787005fce270947a90260746 not found: ID does not exist" containerID="1e2705e8be8dc7917c32c232fbb3032eeea23fc7787005fce270947a90260746"
Aug 01 00:06:06.029747 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:06:06.029698    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1e2705e8be8dc7917c32c232fbb3032eeea23fc7787005fce270947a90260746"} err="failed to get container status \"1e2705e8be8dc7917c32c232fbb3032eeea23fc7787005fce270947a90260746\": rpc error: code = NotFound desc = could not find container \"1e2705e8be8dc7917c32c232fbb3032eeea23fc7787005fce270947a90260746\": container with ID starting with 1e2705e8be8dc7917c32c232fbb3032eeea23fc7787005fce270947a90260746 not found: ID does not exist"
Aug 01 00:06:06.029747 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:06:06.029713    2578 scope.go:117] "RemoveContainer" containerID="dd6ffc8a649db82dc9b5264b31424cb8cf0eb3d7b80a7dcab67190bcb7f19ce3"
Aug 01 00:06:06.029975 ip-10-0-136-108 kubenswrapper[2578]: E0801 00:06:06.029956    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dd6ffc8a649db82dc9b5264b31424cb8cf0eb3d7b80a7dcab67190bcb7f19ce3\": container with ID starting with dd6ffc8a649db82dc9b5264b31424cb8cf0eb3d7b80a7dcab67190bcb7f19ce3 not found: ID does not exist" containerID="dd6ffc8a649db82dc9b5264b31424cb8cf0eb3d7b80a7dcab67190bcb7f19ce3"
Aug 01 00:06:06.030030 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:06:06.029985    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dd6ffc8a649db82dc9b5264b31424cb8cf0eb3d7b80a7dcab67190bcb7f19ce3"} err="failed to get container status \"dd6ffc8a649db82dc9b5264b31424cb8cf0eb3d7b80a7dcab67190bcb7f19ce3\": rpc error: code = NotFound desc = could not find container \"dd6ffc8a649db82dc9b5264b31424cb8cf0eb3d7b80a7dcab67190bcb7f19ce3\": container with ID starting with dd6ffc8a649db82dc9b5264b31424cb8cf0eb3d7b80a7dcab67190bcb7f19ce3 not found: ID does not exist"
Aug 01 00:06:06.032866 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:06:06.032839    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-32541-predictor-7ff4c79d55-9mbgd"]
Aug 01 00:06:06.035045 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:06:06.035025    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-32541-predictor-7ff4c79d55-9mbgd"]
Aug 01 00:06:06.491377 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:06:06.491346    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fbac4361-0369-4cea-ab12-03a4e4ea1171" path="/var/lib/kubelet/pods/fbac4361-0369-4cea-ab12-03a4e4ea1171/volumes"
Aug 01 00:06:09.702924 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:06:09.702880    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-32541-775bd66797-x4tlp" podUID="28872143-734e-478c-9000-be449b6d1734" containerName="switch-graph-32541" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Aug 01 00:06:10.010522 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:06:10.010497    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-fe2ca-predictor-698655d7c5-6h276"
Aug 01 00:06:10.011031 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:06:10.011000    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-fe2ca-predictor-698655d7c5-6h276" podUID="cde6e7e4-4775-491c-8965-25508d23d619" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.37:8080: connect: connection refused"
Aug 01 00:06:14.702661 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:06:14.702620    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-32541-775bd66797-x4tlp" podUID="28872143-734e-478c-9000-be449b6d1734" containerName="switch-graph-32541" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Aug 01 00:06:14.703065 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:06:14.702741    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/switch-graph-32541-775bd66797-x4tlp"
Aug 01 00:06:19.703055 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:06:19.703013    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-32541-775bd66797-x4tlp" podUID="28872143-734e-478c-9000-be449b6d1734" containerName="switch-graph-32541" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Aug 01 00:06:20.011092 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:06:20.011056    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-fe2ca-predictor-698655d7c5-6h276" podUID="cde6e7e4-4775-491c-8965-25508d23d619" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.37:8080: connect: connection refused"
Aug 01 00:06:24.702752 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:06:24.702708    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-32541-775bd66797-x4tlp" podUID="28872143-734e-478c-9000-be449b6d1734" containerName="switch-graph-32541" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Aug 01 00:06:29.703367 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:06:29.703324    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-32541-775bd66797-x4tlp" podUID="28872143-734e-478c-9000-be449b6d1734" containerName="switch-graph-32541" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Aug 01 00:06:30.011589 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:06:30.011549    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-fe2ca-predictor-698655d7c5-6h276" podUID="cde6e7e4-4775-491c-8965-25508d23d619" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.37:8080: connect: connection refused"
Aug 01 00:06:32.133275 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:06:32.133241    2578 generic.go:356] "Generic (PLEG): container finished" podID="28872143-734e-478c-9000-be449b6d1734" containerID="d07591f6cbd3e05e2f2930dd03517d209f99c82f76b7f0cacbc7dd7c166bf090" exitCode=0
Aug 01 00:06:32.133721 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:06:32.133311    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/switch-graph-32541-775bd66797-x4tlp" event={"ID":"28872143-734e-478c-9000-be449b6d1734","Type":"ContainerDied","Data":"d07591f6cbd3e05e2f2930dd03517d209f99c82f76b7f0cacbc7dd7c166bf090"}
Aug 01 00:06:32.263040 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:06:32.263015    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/switch-graph-32541-775bd66797-x4tlp"
Aug 01 00:06:32.342916 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:06:32.342878    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/28872143-734e-478c-9000-be449b6d1734-proxy-tls\") pod \"28872143-734e-478c-9000-be449b6d1734\" (UID: \"28872143-734e-478c-9000-be449b6d1734\") "
Aug 01 00:06:32.342916 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:06:32.342922    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/28872143-734e-478c-9000-be449b6d1734-openshift-service-ca-bundle\") pod \"28872143-734e-478c-9000-be449b6d1734\" (UID: \"28872143-734e-478c-9000-be449b6d1734\") "
Aug 01 00:06:32.343328 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:06:32.343303    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/28872143-734e-478c-9000-be449b6d1734-openshift-service-ca-bundle" (OuterVolumeSpecName: "openshift-service-ca-bundle") pod "28872143-734e-478c-9000-be449b6d1734" (UID: "28872143-734e-478c-9000-be449b6d1734"). InnerVolumeSpecName "openshift-service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 01 00:06:32.344909 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:06:32.344884    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/28872143-734e-478c-9000-be449b6d1734-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "28872143-734e-478c-9000-be449b6d1734" (UID: "28872143-734e-478c-9000-be449b6d1734"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 01 00:06:32.443571 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:06:32.443483    2578 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/28872143-734e-478c-9000-be449b6d1734-proxy-tls\") on node \"ip-10-0-136-108.ec2.internal\" DevicePath \"\""
Aug 01 00:06:32.443571 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:06:32.443515    2578 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/28872143-734e-478c-9000-be449b6d1734-openshift-service-ca-bundle\") on node \"ip-10-0-136-108.ec2.internal\" DevicePath \"\""
Aug 01 00:06:33.139159 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:06:33.139123    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/switch-graph-32541-775bd66797-x4tlp" event={"ID":"28872143-734e-478c-9000-be449b6d1734","Type":"ContainerDied","Data":"8ec487a3493d6e6f8f0a2b5dfaac6c599943845fea53033ef0db77ec13215cdd"}
Aug 01 00:06:33.139159 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:06:33.139146    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/switch-graph-32541-775bd66797-x4tlp"
Aug 01 00:06:33.139679 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:06:33.139233    2578 scope.go:117] "RemoveContainer" containerID="d07591f6cbd3e05e2f2930dd03517d209f99c82f76b7f0cacbc7dd7c166bf090"
Aug 01 00:06:33.154305 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:06:33.154277    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/switch-graph-32541-775bd66797-x4tlp"]
Aug 01 00:06:33.157992 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:06:33.157967    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/switch-graph-32541-775bd66797-x4tlp"]
Aug 01 00:06:34.489478 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:06:34.489438    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="28872143-734e-478c-9000-be449b6d1734" path="/var/lib/kubelet/pods/28872143-734e-478c-9000-be449b6d1734/volumes"
Aug 01 00:06:40.011216 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:06:40.011152    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-fe2ca-predictor-698655d7c5-6h276" podUID="cde6e7e4-4775-491c-8965-25508d23d619" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.37:8080: connect: connection refused"
Aug 01 00:06:50.012008 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:06:50.011975    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-fe2ca-predictor-698655d7c5-6h276"
Aug 01 00:06:52.004796 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:06:52.004762    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-a3ab7-5cc8864447-tbfbv"]
Aug 01 00:06:52.005282 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:06:52.004978    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/sequence-graph-a3ab7-5cc8864447-tbfbv" podUID="fcfc7c13-06ba-4fb4-a473-c3323bc9e4b8" containerName="sequence-graph-a3ab7" containerID="cri-o://e68e8bf27a2ddc3ff68ba9ecef9aab5e1bf8c6f5bf22f497ed519db9147324e0" gracePeriod=30
Aug 01 00:06:52.024038 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:06:52.023994    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gttdm_b32e6d42-e4e0-4113-b8ce-f5d5c5dc13f7/konnectivity-agent/0.log" limit=1024
Aug 01 00:06:52.110954 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:06:52.110917    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-a3ab7-predictor-7544d96d6f-m4qb4"]
Aug 01 00:06:52.111163 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:06:52.111136    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-a3ab7-predictor-7544d96d6f-m4qb4" podUID="15f7d851-03e8-46bc-b59c-1b9ef87d6847" containerName="kserve-container" containerID="cri-o://363f3c3e5337665b01e370fcafa690c62c213158ef92b3a06525bec94021012c" gracePeriod=30
Aug 01 00:06:52.111251 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:06:52.111219    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-a3ab7-predictor-7544d96d6f-m4qb4" podUID="15f7d851-03e8-46bc-b59c-1b9ef87d6847" containerName="kube-rbac-proxy" containerID="cri-o://0a18629e275bb4ca18cab9a1409d92eae27932bebc9c4d9d6d96a0be0d9d5536" gracePeriod=30
Aug 01 00:06:52.132736 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:06:52.132703    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gttdm_b32e6d42-e4e0-4113-b8ce-f5d5c5dc13f7/konnectivity-agent/0.log" limit=1024
Aug 01 00:06:52.198768 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:06:52.198738    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gttdm_b32e6d42-e4e0-4113-b8ce-f5d5c5dc13f7/konnectivity-agent/0.log" limit=1024
Aug 01 00:06:52.202906 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:06:52.202879    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-df362-predictor-ddc8d9667-jd4hz"]
Aug 01 00:06:52.203286 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:06:52.203273    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fbac4361-0369-4cea-ab12-03a4e4ea1171" containerName="kube-rbac-proxy"
Aug 01 00:06:52.203336 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:06:52.203288    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="fbac4361-0369-4cea-ab12-03a4e4ea1171" containerName="kube-rbac-proxy"
Aug 01 00:06:52.203336 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:06:52.203301    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fbac4361-0369-4cea-ab12-03a4e4ea1171" containerName="kserve-container"
Aug 01 00:06:52.203336 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:06:52.203307    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="fbac4361-0369-4cea-ab12-03a4e4ea1171" containerName="kserve-container"
Aug 01 00:06:52.203336 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:06:52.203321    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="28872143-734e-478c-9000-be449b6d1734" containerName="switch-graph-32541"
Aug 01 00:06:52.203336 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:06:52.203327    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="28872143-734e-478c-9000-be449b6d1734" containerName="switch-graph-32541"
Aug 01 00:06:52.203490 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:06:52.203384    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="fbac4361-0369-4cea-ab12-03a4e4ea1171" containerName="kserve-container"
Aug 01 00:06:52.203490 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:06:52.203391    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="fbac4361-0369-4cea-ab12-03a4e4ea1171" containerName="kube-rbac-proxy"
Aug 01 00:06:52.203490 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:06:52.203398    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="28872143-734e-478c-9000-be449b6d1734" containerName="switch-graph-32541"
Aug 01 00:06:52.206648 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:06:52.206629    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-df362-predictor-ddc8d9667-jd4hz"
Aug 01 00:06:52.208696 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:06:52.208676    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-df362-predictor-serving-cert\""
Aug 01 00:06:52.208818 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:06:52.208723    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-df362-kube-rbac-proxy-sar-config\""
Aug 01 00:06:52.217353 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:06:52.217325    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-df362-predictor-ddc8d9667-jd4hz"]
Aug 01 00:06:52.234641 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:06:52.234603    2578 generic.go:356] "Generic (PLEG): container finished" podID="15f7d851-03e8-46bc-b59c-1b9ef87d6847" containerID="0a18629e275bb4ca18cab9a1409d92eae27932bebc9c4d9d6d96a0be0d9d5536" exitCode=2
Aug 01 00:06:52.234821 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:06:52.234715    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-a3ab7-predictor-7544d96d6f-m4qb4" event={"ID":"15f7d851-03e8-46bc-b59c-1b9ef87d6847","Type":"ContainerDied","Data":"0a18629e275bb4ca18cab9a1409d92eae27932bebc9c4d9d6d96a0be0d9d5536"}
Aug 01 00:06:52.304046 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:06:52.303936    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gttdm_b32e6d42-e4e0-4113-b8ce-f5d5c5dc13f7/konnectivity-agent/0.log" limit=1024
Aug 01 00:06:52.317026 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:06:52.316992    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8cql2\" (UniqueName: \"kubernetes.io/projected/b5f4558a-d013-46cb-98b6-84f50810cdba-kube-api-access-8cql2\") pod \"success-200-isvc-df362-predictor-ddc8d9667-jd4hz\" (UID: \"b5f4558a-d013-46cb-98b6-84f50810cdba\") " pod="kserve-ci-e2e-test/success-200-isvc-df362-predictor-ddc8d9667-jd4hz"
Aug 01 00:06:52.317193 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:06:52.317075    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b5f4558a-d013-46cb-98b6-84f50810cdba-proxy-tls\") pod \"success-200-isvc-df362-predictor-ddc8d9667-jd4hz\" (UID: \"b5f4558a-d013-46cb-98b6-84f50810cdba\") " pod="kserve-ci-e2e-test/success-200-isvc-df362-predictor-ddc8d9667-jd4hz"
Aug 01 00:06:52.317193 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:06:52.317182    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b5f4558a-d013-46cb-98b6-84f50810cdba-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-df362-predictor-ddc8d9667-jd4hz\" (UID: \"b5f4558a-d013-46cb-98b6-84f50810cdba\") " pod="kserve-ci-e2e-test/success-200-isvc-df362-predictor-ddc8d9667-jd4hz"
Aug 01 00:06:52.418542 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:06:52.418501    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b5f4558a-d013-46cb-98b6-84f50810cdba-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-df362-predictor-ddc8d9667-jd4hz\" (UID: \"b5f4558a-d013-46cb-98b6-84f50810cdba\") " pod="kserve-ci-e2e-test/success-200-isvc-df362-predictor-ddc8d9667-jd4hz"
Aug 01 00:06:52.418726 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:06:52.418557    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8cql2\" (UniqueName: \"kubernetes.io/projected/b5f4558a-d013-46cb-98b6-84f50810cdba-kube-api-access-8cql2\") pod \"success-200-isvc-df362-predictor-ddc8d9667-jd4hz\" (UID: \"b5f4558a-d013-46cb-98b6-84f50810cdba\") " pod="kserve-ci-e2e-test/success-200-isvc-df362-predictor-ddc8d9667-jd4hz"
Aug 01 00:06:52.418726 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:06:52.418580    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b5f4558a-d013-46cb-98b6-84f50810cdba-proxy-tls\") pod \"success-200-isvc-df362-predictor-ddc8d9667-jd4hz\" (UID: \"b5f4558a-d013-46cb-98b6-84f50810cdba\") " pod="kserve-ci-e2e-test/success-200-isvc-df362-predictor-ddc8d9667-jd4hz"
Aug 01 00:06:52.418726 ip-10-0-136-108 kubenswrapper[2578]: E0801 00:06:52.418689    2578 secret.go:189] Couldn't get secret kserve-ci-e2e-test/success-200-isvc-df362-predictor-serving-cert: secret "success-200-isvc-df362-predictor-serving-cert" not found
Aug 01 00:06:52.418869 ip-10-0-136-108 kubenswrapper[2578]: E0801 00:06:52.418757    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b5f4558a-d013-46cb-98b6-84f50810cdba-proxy-tls podName:b5f4558a-d013-46cb-98b6-84f50810cdba nodeName:}" failed. No retries permitted until 2026-08-01 00:06:52.918739802 +0000 UTC m=+1236.886490542 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/b5f4558a-d013-46cb-98b6-84f50810cdba-proxy-tls") pod "success-200-isvc-df362-predictor-ddc8d9667-jd4hz" (UID: "b5f4558a-d013-46cb-98b6-84f50810cdba") : secret "success-200-isvc-df362-predictor-serving-cert" not found
Aug 01 00:06:52.419459 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:06:52.419432    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b5f4558a-d013-46cb-98b6-84f50810cdba-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-df362-predictor-ddc8d9667-jd4hz\" (UID: \"b5f4558a-d013-46cb-98b6-84f50810cdba\") " pod="kserve-ci-e2e-test/success-200-isvc-df362-predictor-ddc8d9667-jd4hz"
Aug 01 00:06:52.428611 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:06:52.428586    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8cql2\" (UniqueName: \"kubernetes.io/projected/b5f4558a-d013-46cb-98b6-84f50810cdba-kube-api-access-8cql2\") pod \"success-200-isvc-df362-predictor-ddc8d9667-jd4hz\" (UID: \"b5f4558a-d013-46cb-98b6-84f50810cdba\") " pod="kserve-ci-e2e-test/success-200-isvc-df362-predictor-ddc8d9667-jd4hz"
Aug 01 00:06:52.536498 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:06:52.536451    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gttdm_b32e6d42-e4e0-4113-b8ce-f5d5c5dc13f7/konnectivity-agent/0.log" limit=1024
Aug 01 00:06:52.864982 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:06:52.864944    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gttdm_b32e6d42-e4e0-4113-b8ce-f5d5c5dc13f7/konnectivity-agent/0.log" limit=1024
Aug 01 00:06:52.923130 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:06:52.923089    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b5f4558a-d013-46cb-98b6-84f50810cdba-proxy-tls\") pod \"success-200-isvc-df362-predictor-ddc8d9667-jd4hz\" (UID: \"b5f4558a-d013-46cb-98b6-84f50810cdba\") " pod="kserve-ci-e2e-test/success-200-isvc-df362-predictor-ddc8d9667-jd4hz"
Aug 01 00:06:52.925597 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:06:52.925578    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b5f4558a-d013-46cb-98b6-84f50810cdba-proxy-tls\") pod \"success-200-isvc-df362-predictor-ddc8d9667-jd4hz\" (UID: \"b5f4558a-d013-46cb-98b6-84f50810cdba\") " pod="kserve-ci-e2e-test/success-200-isvc-df362-predictor-ddc8d9667-jd4hz"
Aug 01 00:06:53.118668 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:06:53.118578    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-df362-predictor-ddc8d9667-jd4hz"
Aug 01 00:06:53.252528 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:06:53.252504    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-df362-predictor-ddc8d9667-jd4hz"]
Aug 01 00:06:53.255102 ip-10-0-136-108 kubenswrapper[2578]: W0801 00:06:53.255075    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb5f4558a_d013_46cb_98b6_84f50810cdba.slice/crio-b01795bfd26869f4ea209c0273e8b32e6ba00a34b34ba693b425ade296abbd16 WatchSource:0}: Error finding container b01795bfd26869f4ea209c0273e8b32e6ba00a34b34ba693b425ade296abbd16: Status 404 returned error can't find the container with id b01795bfd26869f4ea209c0273e8b32e6ba00a34b34ba693b425ade296abbd16
Aug 01 00:06:54.246774 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:06:54.246734    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-df362-predictor-ddc8d9667-jd4hz" event={"ID":"b5f4558a-d013-46cb-98b6-84f50810cdba","Type":"ContainerStarted","Data":"49a9532e43b0046ef5c23597b1862e6bd5460a717320fb1e1e0ff40eac9a330f"}
Aug 01 00:06:54.247196 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:06:54.246794    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-df362-predictor-ddc8d9667-jd4hz" event={"ID":"b5f4558a-d013-46cb-98b6-84f50810cdba","Type":"ContainerStarted","Data":"c9209c344361dcc6bbdb6d31a8efbbf83182e645b9b2e953aa0786e9d395d934"}
Aug 01 00:06:54.247196 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:06:54.246814    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-df362-predictor-ddc8d9667-jd4hz" event={"ID":"b5f4558a-d013-46cb-98b6-84f50810cdba","Type":"ContainerStarted","Data":"b01795bfd26869f4ea209c0273e8b32e6ba00a34b34ba693b425ade296abbd16"}
Aug 01 00:06:54.247196 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:06:54.246938    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-df362-predictor-ddc8d9667-jd4hz"
Aug 01 00:06:54.265006 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:06:54.264951    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/success-200-isvc-df362-predictor-ddc8d9667-jd4hz" podStartSLOduration=2.264933982 podStartE2EDuration="2.264933982s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-01 00:06:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-01 00:06:54.262642008 +0000 UTC m=+1238.230392765" watchObservedRunningTime="2026-08-01 00:06:54.264933982 +0000 UTC m=+1238.232684740"
Aug 01 00:06:54.650295 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:06:54.650202    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-a3ab7-predictor-7544d96d6f-m4qb4" podUID="15f7d851-03e8-46bc-b59c-1b9ef87d6847" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.34:8643/healthz\": dial tcp 10.133.0.34:8643: connect: connection refused"
Aug 01 00:06:54.654601 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:06:54.654564    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-a3ab7-predictor-7544d96d6f-m4qb4" podUID="15f7d851-03e8-46bc-b59c-1b9ef87d6847" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.34:8080: connect: connection refused"
Aug 01 00:06:54.944596 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:06:54.944506    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-a3ab7-5cc8864447-tbfbv" podUID="fcfc7c13-06ba-4fb4-a473-c3323bc9e4b8" containerName="sequence-graph-a3ab7" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Aug 01 00:06:55.251622 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:06:55.251592    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-df362-predictor-ddc8d9667-jd4hz"
Aug 01 00:06:55.252887 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:06:55.252859    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-df362-predictor-ddc8d9667-jd4hz" podUID="b5f4558a-d013-46cb-98b6-84f50810cdba" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.38:8080: connect: connection refused"
Aug 01 00:06:55.569933 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:06:55.569902    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-a3ab7-predictor-7544d96d6f-m4qb4"
Aug 01 00:06:55.647128 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:06:55.647090    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/15f7d851-03e8-46bc-b59c-1b9ef87d6847-proxy-tls\") pod \"15f7d851-03e8-46bc-b59c-1b9ef87d6847\" (UID: \"15f7d851-03e8-46bc-b59c-1b9ef87d6847\") "
Aug 01 00:06:55.647328 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:06:55.647206    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-twb84\" (UniqueName: \"kubernetes.io/projected/15f7d851-03e8-46bc-b59c-1b9ef87d6847-kube-api-access-twb84\") pod \"15f7d851-03e8-46bc-b59c-1b9ef87d6847\" (UID: \"15f7d851-03e8-46bc-b59c-1b9ef87d6847\") "
Aug 01 00:06:55.647328 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:06:55.647257    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/15f7d851-03e8-46bc-b59c-1b9ef87d6847-kube-rbac-proxy-sar-config\") pod \"15f7d851-03e8-46bc-b59c-1b9ef87d6847\" (UID: \"15f7d851-03e8-46bc-b59c-1b9ef87d6847\") "
Aug 01 00:06:55.647660 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:06:55.647635    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/15f7d851-03e8-46bc-b59c-1b9ef87d6847-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "15f7d851-03e8-46bc-b59c-1b9ef87d6847" (UID: "15f7d851-03e8-46bc-b59c-1b9ef87d6847"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 01 00:06:55.649253 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:06:55.649226    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/15f7d851-03e8-46bc-b59c-1b9ef87d6847-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "15f7d851-03e8-46bc-b59c-1b9ef87d6847" (UID: "15f7d851-03e8-46bc-b59c-1b9ef87d6847"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 01 00:06:55.649314 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:06:55.649245    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/15f7d851-03e8-46bc-b59c-1b9ef87d6847-kube-api-access-twb84" (OuterVolumeSpecName: "kube-api-access-twb84") pod "15f7d851-03e8-46bc-b59c-1b9ef87d6847" (UID: "15f7d851-03e8-46bc-b59c-1b9ef87d6847"). InnerVolumeSpecName "kube-api-access-twb84". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 01 00:06:55.748836 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:06:55.748796    2578 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/15f7d851-03e8-46bc-b59c-1b9ef87d6847-proxy-tls\") on node \"ip-10-0-136-108.ec2.internal\" DevicePath \"\""
Aug 01 00:06:55.748836 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:06:55.748834    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-twb84\" (UniqueName: \"kubernetes.io/projected/15f7d851-03e8-46bc-b59c-1b9ef87d6847-kube-api-access-twb84\") on node \"ip-10-0-136-108.ec2.internal\" DevicePath \"\""
Aug 01 00:06:55.749015 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:06:55.748848    2578 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/15f7d851-03e8-46bc-b59c-1b9ef87d6847-kube-rbac-proxy-sar-config\") on node \"ip-10-0-136-108.ec2.internal\" DevicePath \"\""
Aug 01 00:06:56.257936 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:06:56.257903    2578 generic.go:356] "Generic (PLEG): container finished" podID="15f7d851-03e8-46bc-b59c-1b9ef87d6847" containerID="363f3c3e5337665b01e370fcafa690c62c213158ef92b3a06525bec94021012c" exitCode=0
Aug 01 00:06:56.258372 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:06:56.257959    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-a3ab7-predictor-7544d96d6f-m4qb4"
Aug 01 00:06:56.258372 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:06:56.257993    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-a3ab7-predictor-7544d96d6f-m4qb4" event={"ID":"15f7d851-03e8-46bc-b59c-1b9ef87d6847","Type":"ContainerDied","Data":"363f3c3e5337665b01e370fcafa690c62c213158ef92b3a06525bec94021012c"}
Aug 01 00:06:56.258372 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:06:56.258065    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-a3ab7-predictor-7544d96d6f-m4qb4" event={"ID":"15f7d851-03e8-46bc-b59c-1b9ef87d6847","Type":"ContainerDied","Data":"3516dfdfc698661746e098ef94a99916d89bd6f269b1bd3560d3fcf83b7fa87d"}
Aug 01 00:06:56.258372 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:06:56.258105    2578 scope.go:117] "RemoveContainer" containerID="0a18629e275bb4ca18cab9a1409d92eae27932bebc9c4d9d6d96a0be0d9d5536"
Aug 01 00:06:56.258745 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:06:56.258716    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-df362-predictor-ddc8d9667-jd4hz" podUID="b5f4558a-d013-46cb-98b6-84f50810cdba" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.38:8080: connect: connection refused"
Aug 01 00:06:56.267007 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:06:56.266991    2578 scope.go:117] "RemoveContainer" containerID="363f3c3e5337665b01e370fcafa690c62c213158ef92b3a06525bec94021012c"
Aug 01 00:06:56.274700 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:06:56.274672    2578 scope.go:117] "RemoveContainer" containerID="0a18629e275bb4ca18cab9a1409d92eae27932bebc9c4d9d6d96a0be0d9d5536"
Aug 01 00:06:56.274914 ip-10-0-136-108 kubenswrapper[2578]: E0801 00:06:56.274894    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0a18629e275bb4ca18cab9a1409d92eae27932bebc9c4d9d6d96a0be0d9d5536\": container with ID starting with 0a18629e275bb4ca18cab9a1409d92eae27932bebc9c4d9d6d96a0be0d9d5536 not found: ID does not exist" containerID="0a18629e275bb4ca18cab9a1409d92eae27932bebc9c4d9d6d96a0be0d9d5536"
Aug 01 00:06:56.274967 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:06:56.274924    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0a18629e275bb4ca18cab9a1409d92eae27932bebc9c4d9d6d96a0be0d9d5536"} err="failed to get container status \"0a18629e275bb4ca18cab9a1409d92eae27932bebc9c4d9d6d96a0be0d9d5536\": rpc error: code = NotFound desc = could not find container \"0a18629e275bb4ca18cab9a1409d92eae27932bebc9c4d9d6d96a0be0d9d5536\": container with ID starting with 0a18629e275bb4ca18cab9a1409d92eae27932bebc9c4d9d6d96a0be0d9d5536 not found: ID does not exist"
Aug 01 00:06:56.274967 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:06:56.274949    2578 scope.go:117] "RemoveContainer" containerID="363f3c3e5337665b01e370fcafa690c62c213158ef92b3a06525bec94021012c"
Aug 01 00:06:56.275213 ip-10-0-136-108 kubenswrapper[2578]: E0801 00:06:56.275191    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"363f3c3e5337665b01e370fcafa690c62c213158ef92b3a06525bec94021012c\": container with ID starting with 363f3c3e5337665b01e370fcafa690c62c213158ef92b3a06525bec94021012c not found: ID does not exist" containerID="363f3c3e5337665b01e370fcafa690c62c213158ef92b3a06525bec94021012c"
Aug 01 00:06:56.275302 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:06:56.275216    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"363f3c3e5337665b01e370fcafa690c62c213158ef92b3a06525bec94021012c"} err="failed to get container status \"363f3c3e5337665b01e370fcafa690c62c213158ef92b3a06525bec94021012c\": rpc error: code = NotFound desc = could not find container \"363f3c3e5337665b01e370fcafa690c62c213158ef92b3a06525bec94021012c\": container with ID starting with 363f3c3e5337665b01e370fcafa690c62c213158ef92b3a06525bec94021012c not found: ID does not exist"
Aug 01 00:06:56.293486 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:06:56.293454    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-a3ab7-predictor-7544d96d6f-m4qb4"]
Aug 01 00:06:56.303127 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:06:56.303104    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-a3ab7-predictor-7544d96d6f-m4qb4"]
Aug 01 00:06:56.491090 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:06:56.491056    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="15f7d851-03e8-46bc-b59c-1b9ef87d6847" path="/var/lib/kubelet/pods/15f7d851-03e8-46bc-b59c-1b9ef87d6847/volumes"
Aug 01 00:06:59.944762 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:06:59.944716    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-a3ab7-5cc8864447-tbfbv" podUID="fcfc7c13-06ba-4fb4-a473-c3323bc9e4b8" containerName="sequence-graph-a3ab7" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Aug 01 00:07:01.262919 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:07:01.262892    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-df362-predictor-ddc8d9667-jd4hz"
Aug 01 00:07:01.263426 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:07:01.263398    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-df362-predictor-ddc8d9667-jd4hz" podUID="b5f4558a-d013-46cb-98b6-84f50810cdba" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.38:8080: connect: connection refused"
Aug 01 00:07:02.365958 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:07:02.365912    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/ensemble-graph-fe2ca-7b566d4bb8-77nlb"]
Aug 01 00:07:02.366477 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:07:02.366456    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="15f7d851-03e8-46bc-b59c-1b9ef87d6847" containerName="kube-rbac-proxy"
Aug 01 00:07:02.366527 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:07:02.366481    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="15f7d851-03e8-46bc-b59c-1b9ef87d6847" containerName="kube-rbac-proxy"
Aug 01 00:07:02.366527 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:07:02.366513    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="15f7d851-03e8-46bc-b59c-1b9ef87d6847" containerName="kserve-container"
Aug 01 00:07:02.366527 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:07:02.366522    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="15f7d851-03e8-46bc-b59c-1b9ef87d6847" containerName="kserve-container"
Aug 01 00:07:02.366616 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:07:02.366607    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="15f7d851-03e8-46bc-b59c-1b9ef87d6847" containerName="kserve-container"
Aug 01 00:07:02.366652 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:07:02.366621    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="15f7d851-03e8-46bc-b59c-1b9ef87d6847" containerName="kube-rbac-proxy"
Aug 01 00:07:02.371433 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:07:02.371407    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/ensemble-graph-fe2ca-7b566d4bb8-77nlb"
Aug 01 00:07:02.372364 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:07:02.372334    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gttdm_b32e6d42-e4e0-4113-b8ce-f5d5c5dc13f7/konnectivity-agent/0.log" limit=1024
Aug 01 00:07:02.373429 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:07:02.373408    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"ensemble-graph-fe2ca-kube-rbac-proxy-sar-config\""
Aug 01 00:07:02.373556 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:07:02.373412    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"ensemble-graph-fe2ca-serving-cert\""
Aug 01 00:07:02.379589 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:07:02.379569    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/ensemble-graph-fe2ca-7b566d4bb8-77nlb"]
Aug 01 00:07:02.483433 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:07:02.483399    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gttdm_b32e6d42-e4e0-4113-b8ce-f5d5c5dc13f7/konnectivity-agent/0.log" limit=1024
Aug 01 00:07:02.506959 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:07:02.506921    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/00818bdd-5ee9-4a26-ba26-0ecadbf181c8-openshift-service-ca-bundle\") pod \"ensemble-graph-fe2ca-7b566d4bb8-77nlb\" (UID: \"00818bdd-5ee9-4a26-ba26-0ecadbf181c8\") " pod="kserve-ci-e2e-test/ensemble-graph-fe2ca-7b566d4bb8-77nlb"
Aug 01 00:07:02.507148 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:07:02.506969    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/00818bdd-5ee9-4a26-ba26-0ecadbf181c8-proxy-tls\") pod \"ensemble-graph-fe2ca-7b566d4bb8-77nlb\" (UID: \"00818bdd-5ee9-4a26-ba26-0ecadbf181c8\") " pod="kserve-ci-e2e-test/ensemble-graph-fe2ca-7b566d4bb8-77nlb"
Aug 01 00:07:02.608455 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:07:02.608409    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/00818bdd-5ee9-4a26-ba26-0ecadbf181c8-openshift-service-ca-bundle\") pod \"ensemble-graph-fe2ca-7b566d4bb8-77nlb\" (UID: \"00818bdd-5ee9-4a26-ba26-0ecadbf181c8\") " pod="kserve-ci-e2e-test/ensemble-graph-fe2ca-7b566d4bb8-77nlb"
Aug 01 00:07:02.608455 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:07:02.608461    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/00818bdd-5ee9-4a26-ba26-0ecadbf181c8-proxy-tls\") pod \"ensemble-graph-fe2ca-7b566d4bb8-77nlb\" (UID: \"00818bdd-5ee9-4a26-ba26-0ecadbf181c8\") " pod="kserve-ci-e2e-test/ensemble-graph-fe2ca-7b566d4bb8-77nlb"
Aug 01 00:07:02.609200 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:07:02.609160    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/00818bdd-5ee9-4a26-ba26-0ecadbf181c8-openshift-service-ca-bundle\") pod \"ensemble-graph-fe2ca-7b566d4bb8-77nlb\" (UID: \"00818bdd-5ee9-4a26-ba26-0ecadbf181c8\") " pod="kserve-ci-e2e-test/ensemble-graph-fe2ca-7b566d4bb8-77nlb"
Aug 01 00:07:02.610854 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:07:02.610836    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/00818bdd-5ee9-4a26-ba26-0ecadbf181c8-proxy-tls\") pod \"ensemble-graph-fe2ca-7b566d4bb8-77nlb\" (UID: \"00818bdd-5ee9-4a26-ba26-0ecadbf181c8\") " pod="kserve-ci-e2e-test/ensemble-graph-fe2ca-7b566d4bb8-77nlb"
Aug 01 00:07:02.683556 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:07:02.683463    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/ensemble-graph-fe2ca-7b566d4bb8-77nlb"
Aug 01 00:07:02.821014 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:07:02.820993    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/ensemble-graph-fe2ca-7b566d4bb8-77nlb"]
Aug 01 00:07:02.823601 ip-10-0-136-108 kubenswrapper[2578]: W0801 00:07:02.823576    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod00818bdd_5ee9_4a26_ba26_0ecadbf181c8.slice/crio-3c75058f82ea1bf0180d8a074343c274f147fe478763625d474ad8ff768ef57d WatchSource:0}: Error finding container 3c75058f82ea1bf0180d8a074343c274f147fe478763625d474ad8ff768ef57d: Status 404 returned error can't find the container with id 3c75058f82ea1bf0180d8a074343c274f147fe478763625d474ad8ff768ef57d
Aug 01 00:07:03.293007 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:07:03.292965    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/ensemble-graph-fe2ca-7b566d4bb8-77nlb" event={"ID":"00818bdd-5ee9-4a26-ba26-0ecadbf181c8","Type":"ContainerStarted","Data":"c3ff822e674b816bc29982eb4e5b651233bec32cf991c68d7ef826ea88972647"}
Aug 01 00:07:03.293202 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:07:03.293046    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/ensemble-graph-fe2ca-7b566d4bb8-77nlb" event={"ID":"00818bdd-5ee9-4a26-ba26-0ecadbf181c8","Type":"ContainerStarted","Data":"3c75058f82ea1bf0180d8a074343c274f147fe478763625d474ad8ff768ef57d"}
Aug 01 00:07:03.293202 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:07:03.293102    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/ensemble-graph-fe2ca-7b566d4bb8-77nlb"
Aug 01 00:07:03.308003 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:07:03.307946    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/ensemble-graph-fe2ca-7b566d4bb8-77nlb" podStartSLOduration=1.307924257 podStartE2EDuration="1.307924257s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-01 00:07:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-01 00:07:03.307205616 +0000 UTC m=+1247.274956373" watchObservedRunningTime="2026-08-01 00:07:03.307924257 +0000 UTC m=+1247.275675021"
Aug 01 00:07:04.945193 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:07:04.945129    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-a3ab7-5cc8864447-tbfbv" podUID="fcfc7c13-06ba-4fb4-a473-c3323bc9e4b8" containerName="sequence-graph-a3ab7" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Aug 01 00:07:04.945564 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:07:04.945271    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/sequence-graph-a3ab7-5cc8864447-tbfbv"
Aug 01 00:07:09.304216 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:07:09.304118    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/ensemble-graph-fe2ca-7b566d4bb8-77nlb"
Aug 01 00:07:09.944207 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:07:09.944153    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-a3ab7-5cc8864447-tbfbv" podUID="fcfc7c13-06ba-4fb4-a473-c3323bc9e4b8" containerName="sequence-graph-a3ab7" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Aug 01 00:07:11.264281 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:07:11.264241    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-df362-predictor-ddc8d9667-jd4hz" podUID="b5f4558a-d013-46cb-98b6-84f50810cdba" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.38:8080: connect: connection refused"
Aug 01 00:07:12.462239 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:07:12.462138    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/ensemble-graph-fe2ca-7b566d4bb8-77nlb"]
Aug 01 00:07:12.462679 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:07:12.462416    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/ensemble-graph-fe2ca-7b566d4bb8-77nlb" podUID="00818bdd-5ee9-4a26-ba26-0ecadbf181c8" containerName="ensemble-graph-fe2ca" containerID="cri-o://c3ff822e674b816bc29982eb4e5b651233bec32cf991c68d7ef826ea88972647" gracePeriod=30
Aug 01 00:07:12.482672 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:07:12.482627    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gttdm_b32e6d42-e4e0-4113-b8ce-f5d5c5dc13f7/konnectivity-agent/0.log" limit=1024
Aug 01 00:07:12.568983 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:07:12.568951    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-fe2ca-predictor-698655d7c5-6h276"]
Aug 01 00:07:12.569294 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:07:12.569263    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-fe2ca-predictor-698655d7c5-6h276" podUID="cde6e7e4-4775-491c-8965-25508d23d619" containerName="kserve-container" containerID="cri-o://7b9772fd84e7f4a2119c7b6e3521f8854d89c955f2580ec7441481976f2be9bc" gracePeriod=30
Aug 01 00:07:12.569688 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:07:12.569658    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-fe2ca-predictor-698655d7c5-6h276" podUID="cde6e7e4-4775-491c-8965-25508d23d619" containerName="kube-rbac-proxy" containerID="cri-o://d126be418a54e67d940d68ff4a1211511d6ade1c8f1b80e632e3bf8ea2f731d0" gracePeriod=30
Aug 01 00:07:12.597428 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:07:12.597389    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gttdm_b32e6d42-e4e0-4113-b8ce-f5d5c5dc13f7/konnectivity-agent/0.log" limit=1024
Aug 01 00:07:12.666097 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:07:12.666066    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-01c7a-predictor-774957f645-hvd2w"]
Aug 01 00:07:12.667443 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:07:12.667413    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gttdm_b32e6d42-e4e0-4113-b8ce-f5d5c5dc13f7/konnectivity-agent/0.log" limit=1024
Aug 01 00:07:12.669898 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:07:12.669878    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-01c7a-predictor-774957f645-hvd2w"
Aug 01 00:07:12.671826 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:07:12.671802    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-01c7a-predictor-serving-cert\""
Aug 01 00:07:12.671929 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:07:12.671809    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-01c7a-kube-rbac-proxy-sar-config\""
Aug 01 00:07:12.679361 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:07:12.679336    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-01c7a-predictor-774957f645-hvd2w"]
Aug 01 00:07:12.694758 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:07:12.694724    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b6eed2b1-cef9-457a-b42d-99effb74a9a4-proxy-tls\") pod \"success-200-isvc-01c7a-predictor-774957f645-hvd2w\" (UID: \"b6eed2b1-cef9-457a-b42d-99effb74a9a4\") " pod="kserve-ci-e2e-test/success-200-isvc-01c7a-predictor-774957f645-hvd2w"
Aug 01 00:07:12.694896 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:07:12.694774    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b6eed2b1-cef9-457a-b42d-99effb74a9a4-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-01c7a-predictor-774957f645-hvd2w\" (UID: \"b6eed2b1-cef9-457a-b42d-99effb74a9a4\") " pod="kserve-ci-e2e-test/success-200-isvc-01c7a-predictor-774957f645-hvd2w"
Aug 01 00:07:12.694896 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:07:12.694851    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hwc8f\" (UniqueName: \"kubernetes.io/projected/b6eed2b1-cef9-457a-b42d-99effb74a9a4-kube-api-access-hwc8f\") pod \"success-200-isvc-01c7a-predictor-774957f645-hvd2w\" (UID: \"b6eed2b1-cef9-457a-b42d-99effb74a9a4\") " pod="kserve-ci-e2e-test/success-200-isvc-01c7a-predictor-774957f645-hvd2w"
Aug 01 00:07:12.764893 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:07:12.764844    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gttdm_b32e6d42-e4e0-4113-b8ce-f5d5c5dc13f7/konnectivity-agent/0.log" limit=1024
Aug 01 00:07:12.796124 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:07:12.796086    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hwc8f\" (UniqueName: \"kubernetes.io/projected/b6eed2b1-cef9-457a-b42d-99effb74a9a4-kube-api-access-hwc8f\") pod \"success-200-isvc-01c7a-predictor-774957f645-hvd2w\" (UID: \"b6eed2b1-cef9-457a-b42d-99effb74a9a4\") " pod="kserve-ci-e2e-test/success-200-isvc-01c7a-predictor-774957f645-hvd2w"
Aug 01 00:07:12.796338 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:07:12.796149    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b6eed2b1-cef9-457a-b42d-99effb74a9a4-proxy-tls\") pod \"success-200-isvc-01c7a-predictor-774957f645-hvd2w\" (UID: \"b6eed2b1-cef9-457a-b42d-99effb74a9a4\") " pod="kserve-ci-e2e-test/success-200-isvc-01c7a-predictor-774957f645-hvd2w"
Aug 01 00:07:12.796338 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:07:12.796222    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b6eed2b1-cef9-457a-b42d-99effb74a9a4-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-01c7a-predictor-774957f645-hvd2w\" (UID: \"b6eed2b1-cef9-457a-b42d-99effb74a9a4\") " pod="kserve-ci-e2e-test/success-200-isvc-01c7a-predictor-774957f645-hvd2w"
Aug 01 00:07:12.796338 ip-10-0-136-108 kubenswrapper[2578]: E0801 00:07:12.796277    2578 secret.go:189] Couldn't get secret kserve-ci-e2e-test/success-200-isvc-01c7a-predictor-serving-cert: secret "success-200-isvc-01c7a-predictor-serving-cert" not found
Aug 01 00:07:12.796468 ip-10-0-136-108 kubenswrapper[2578]: E0801 00:07:12.796343    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b6eed2b1-cef9-457a-b42d-99effb74a9a4-proxy-tls podName:b6eed2b1-cef9-457a-b42d-99effb74a9a4 nodeName:}" failed. No retries permitted until 2026-08-01 00:07:13.296324224 +0000 UTC m=+1257.264074960 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/b6eed2b1-cef9-457a-b42d-99effb74a9a4-proxy-tls") pod "success-200-isvc-01c7a-predictor-774957f645-hvd2w" (UID: "b6eed2b1-cef9-457a-b42d-99effb74a9a4") : secret "success-200-isvc-01c7a-predictor-serving-cert" not found
Aug 01 00:07:12.797017 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:07:12.796971    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b6eed2b1-cef9-457a-b42d-99effb74a9a4-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-01c7a-predictor-774957f645-hvd2w\" (UID: \"b6eed2b1-cef9-457a-b42d-99effb74a9a4\") " pod="kserve-ci-e2e-test/success-200-isvc-01c7a-predictor-774957f645-hvd2w"
Aug 01 00:07:12.806585 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:07:12.806557    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hwc8f\" (UniqueName: \"kubernetes.io/projected/b6eed2b1-cef9-457a-b42d-99effb74a9a4-kube-api-access-hwc8f\") pod \"success-200-isvc-01c7a-predictor-774957f645-hvd2w\" (UID: \"b6eed2b1-cef9-457a-b42d-99effb74a9a4\") " pod="kserve-ci-e2e-test/success-200-isvc-01c7a-predictor-774957f645-hvd2w"
Aug 01 00:07:12.866642 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:07:12.866611    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gttdm_b32e6d42-e4e0-4113-b8ce-f5d5c5dc13f7/konnectivity-agent/0.log" limit=1024
Aug 01 00:07:13.299958 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:07:13.299929    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b6eed2b1-cef9-457a-b42d-99effb74a9a4-proxy-tls\") pod \"success-200-isvc-01c7a-predictor-774957f645-hvd2w\" (UID: \"b6eed2b1-cef9-457a-b42d-99effb74a9a4\") " pod="kserve-ci-e2e-test/success-200-isvc-01c7a-predictor-774957f645-hvd2w"
Aug 01 00:07:13.302361 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:07:13.302344    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b6eed2b1-cef9-457a-b42d-99effb74a9a4-proxy-tls\") pod \"success-200-isvc-01c7a-predictor-774957f645-hvd2w\" (UID: \"b6eed2b1-cef9-457a-b42d-99effb74a9a4\") " pod="kserve-ci-e2e-test/success-200-isvc-01c7a-predictor-774957f645-hvd2w"
Aug 01 00:07:13.340523 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:07:13.340472    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gttdm_b32e6d42-e4e0-4113-b8ce-f5d5c5dc13f7/konnectivity-agent/0.log" limit=1024
Aug 01 00:07:13.342413 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:07:13.342381    2578 generic.go:356] "Generic (PLEG): container finished" podID="cde6e7e4-4775-491c-8965-25508d23d619" containerID="d126be418a54e67d940d68ff4a1211511d6ade1c8f1b80e632e3bf8ea2f731d0" exitCode=2
Aug 01 00:07:13.342532 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:07:13.342455    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-fe2ca-predictor-698655d7c5-6h276" event={"ID":"cde6e7e4-4775-491c-8965-25508d23d619","Type":"ContainerDied","Data":"d126be418a54e67d940d68ff4a1211511d6ade1c8f1b80e632e3bf8ea2f731d0"}
Aug 01 00:07:13.582542 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:07:13.582443    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-01c7a-predictor-774957f645-hvd2w"
Aug 01 00:07:13.716118 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:07:13.716047    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-01c7a-predictor-774957f645-hvd2w"]
Aug 01 00:07:13.718212 ip-10-0-136-108 kubenswrapper[2578]: W0801 00:07:13.718164    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb6eed2b1_cef9_457a_b42d_99effb74a9a4.slice/crio-f7b73f50788d6da970f1f72f78620c2ec2b1df08c94809c070ea18280ef40183 WatchSource:0}: Error finding container f7b73f50788d6da970f1f72f78620c2ec2b1df08c94809c070ea18280ef40183: Status 404 returned error can't find the container with id f7b73f50788d6da970f1f72f78620c2ec2b1df08c94809c070ea18280ef40183
Aug 01 00:07:14.302808 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:07:14.302758    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/ensemble-graph-fe2ca-7b566d4bb8-77nlb" podUID="00818bdd-5ee9-4a26-ba26-0ecadbf181c8" containerName="ensemble-graph-fe2ca" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Aug 01 00:07:14.349716 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:07:14.349678    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-01c7a-predictor-774957f645-hvd2w" event={"ID":"b6eed2b1-cef9-457a-b42d-99effb74a9a4","Type":"ContainerStarted","Data":"363ec9505eccd6a0334afa9166e39a7a4fa9ab1902e290b111f75cfc04579a26"}
Aug 01 00:07:14.349908 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:07:14.349743    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-01c7a-predictor-774957f645-hvd2w" event={"ID":"b6eed2b1-cef9-457a-b42d-99effb74a9a4","Type":"ContainerStarted","Data":"4920053b85982f0695f43f20352d96e21ee486f9311931a59836ff92e0338fbe"}
Aug 01 00:07:14.349908 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:07:14.349764    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-01c7a-predictor-774957f645-hvd2w" event={"ID":"b6eed2b1-cef9-457a-b42d-99effb74a9a4","Type":"ContainerStarted","Data":"f7b73f50788d6da970f1f72f78620c2ec2b1df08c94809c070ea18280ef40183"}
Aug 01 00:07:14.349908 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:07:14.349795    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-01c7a-predictor-774957f645-hvd2w"
Aug 01 00:07:14.350072 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:07:14.349910    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-01c7a-predictor-774957f645-hvd2w"
Aug 01 00:07:14.351477 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:07:14.351441    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-01c7a-predictor-774957f645-hvd2w" podUID="b6eed2b1-cef9-457a-b42d-99effb74a9a4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.40:8080: connect: connection refused"
Aug 01 00:07:14.365080 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:07:14.365036    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/success-200-isvc-01c7a-predictor-774957f645-hvd2w" podStartSLOduration=2.365019081 podStartE2EDuration="2.365019081s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-01 00:07:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-01 00:07:14.364593146 +0000 UTC m=+1258.332343907" watchObservedRunningTime="2026-08-01 00:07:14.365019081 +0000 UTC m=+1258.332769839"
Aug 01 00:07:14.944292 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:07:14.944253    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-a3ab7-5cc8864447-tbfbv" podUID="fcfc7c13-06ba-4fb4-a473-c3323bc9e4b8" containerName="sequence-graph-a3ab7" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Aug 01 00:07:15.006144 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:07:15.006096    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-fe2ca-predictor-698655d7c5-6h276" podUID="cde6e7e4-4775-491c-8965-25508d23d619" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.37:8643/healthz\": dial tcp 10.133.0.37:8643: connect: connection refused"
Aug 01 00:07:15.356190 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:07:15.356125    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-01c7a-predictor-774957f645-hvd2w" podUID="b6eed2b1-cef9-457a-b42d-99effb74a9a4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.40:8080: connect: connection refused"
Aug 01 00:07:16.040079 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:07:16.040054    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-fe2ca-predictor-698655d7c5-6h276"
Aug 01 00:07:16.120499 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:07:16.120405    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/cde6e7e4-4775-491c-8965-25508d23d619-proxy-tls\") pod \"cde6e7e4-4775-491c-8965-25508d23d619\" (UID: \"cde6e7e4-4775-491c-8965-25508d23d619\") "
Aug 01 00:07:16.120499 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:07:16.120476    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mp7s4\" (UniqueName: \"kubernetes.io/projected/cde6e7e4-4775-491c-8965-25508d23d619-kube-api-access-mp7s4\") pod \"cde6e7e4-4775-491c-8965-25508d23d619\" (UID: \"cde6e7e4-4775-491c-8965-25508d23d619\") "
Aug 01 00:07:16.120499 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:07:16.120495    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/cde6e7e4-4775-491c-8965-25508d23d619-kube-rbac-proxy-sar-config\") pod \"cde6e7e4-4775-491c-8965-25508d23d619\" (UID: \"cde6e7e4-4775-491c-8965-25508d23d619\") "
Aug 01 00:07:16.120903 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:07:16.120878    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cde6e7e4-4775-491c-8965-25508d23d619-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "cde6e7e4-4775-491c-8965-25508d23d619" (UID: "cde6e7e4-4775-491c-8965-25508d23d619"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 01 00:07:16.122527 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:07:16.122507    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cde6e7e4-4775-491c-8965-25508d23d619-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "cde6e7e4-4775-491c-8965-25508d23d619" (UID: "cde6e7e4-4775-491c-8965-25508d23d619"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 01 00:07:16.122631 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:07:16.122615    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cde6e7e4-4775-491c-8965-25508d23d619-kube-api-access-mp7s4" (OuterVolumeSpecName: "kube-api-access-mp7s4") pod "cde6e7e4-4775-491c-8965-25508d23d619" (UID: "cde6e7e4-4775-491c-8965-25508d23d619"). InnerVolumeSpecName "kube-api-access-mp7s4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 01 00:07:16.221941 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:07:16.221905    2578 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/cde6e7e4-4775-491c-8965-25508d23d619-proxy-tls\") on node \"ip-10-0-136-108.ec2.internal\" DevicePath \"\""
Aug 01 00:07:16.221941 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:07:16.221936    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mp7s4\" (UniqueName: \"kubernetes.io/projected/cde6e7e4-4775-491c-8965-25508d23d619-kube-api-access-mp7s4\") on node \"ip-10-0-136-108.ec2.internal\" DevicePath \"\""
Aug 01 00:07:16.221941 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:07:16.221947    2578 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/cde6e7e4-4775-491c-8965-25508d23d619-kube-rbac-proxy-sar-config\") on node \"ip-10-0-136-108.ec2.internal\" DevicePath \"\""
Aug 01 00:07:16.368926 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:07:16.368895    2578 generic.go:356] "Generic (PLEG): container finished" podID="cde6e7e4-4775-491c-8965-25508d23d619" containerID="7b9772fd84e7f4a2119c7b6e3521f8854d89c955f2580ec7441481976f2be9bc" exitCode=0
Aug 01 00:07:16.369079 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:07:16.368939    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-fe2ca-predictor-698655d7c5-6h276" event={"ID":"cde6e7e4-4775-491c-8965-25508d23d619","Type":"ContainerDied","Data":"7b9772fd84e7f4a2119c7b6e3521f8854d89c955f2580ec7441481976f2be9bc"}
Aug 01 00:07:16.369079 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:07:16.368952    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-fe2ca-predictor-698655d7c5-6h276"
Aug 01 00:07:16.369079 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:07:16.369009    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-fe2ca-predictor-698655d7c5-6h276" event={"ID":"cde6e7e4-4775-491c-8965-25508d23d619","Type":"ContainerDied","Data":"d577f75913f67a453e42aeb0f958ba2738ddef62b7e843e60353a7f0c7bcb684"}
Aug 01 00:07:16.369079 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:07:16.369036    2578 scope.go:117] "RemoveContainer" containerID="d126be418a54e67d940d68ff4a1211511d6ade1c8f1b80e632e3bf8ea2f731d0"
Aug 01 00:07:16.378609 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:07:16.378586    2578 scope.go:117] "RemoveContainer" containerID="7b9772fd84e7f4a2119c7b6e3521f8854d89c955f2580ec7441481976f2be9bc"
Aug 01 00:07:16.386148 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:07:16.386128    2578 scope.go:117] "RemoveContainer" containerID="d126be418a54e67d940d68ff4a1211511d6ade1c8f1b80e632e3bf8ea2f731d0"
Aug 01 00:07:16.386456 ip-10-0-136-108 kubenswrapper[2578]: E0801 00:07:16.386436    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d126be418a54e67d940d68ff4a1211511d6ade1c8f1b80e632e3bf8ea2f731d0\": container with ID starting with d126be418a54e67d940d68ff4a1211511d6ade1c8f1b80e632e3bf8ea2f731d0 not found: ID does not exist" containerID="d126be418a54e67d940d68ff4a1211511d6ade1c8f1b80e632e3bf8ea2f731d0"
Aug 01 00:07:16.386509 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:07:16.386465    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d126be418a54e67d940d68ff4a1211511d6ade1c8f1b80e632e3bf8ea2f731d0"} err="failed to get container status \"d126be418a54e67d940d68ff4a1211511d6ade1c8f1b80e632e3bf8ea2f731d0\": rpc error: code = NotFound desc = could not find container \"d126be418a54e67d940d68ff4a1211511d6ade1c8f1b80e632e3bf8ea2f731d0\": container with ID starting with d126be418a54e67d940d68ff4a1211511d6ade1c8f1b80e632e3bf8ea2f731d0 not found: ID does not exist"
Aug 01 00:07:16.386509 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:07:16.386483    2578 scope.go:117] "RemoveContainer" containerID="7b9772fd84e7f4a2119c7b6e3521f8854d89c955f2580ec7441481976f2be9bc"
Aug 01 00:07:16.386751 ip-10-0-136-108 kubenswrapper[2578]: E0801 00:07:16.386734    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7b9772fd84e7f4a2119c7b6e3521f8854d89c955f2580ec7441481976f2be9bc\": container with ID starting with 7b9772fd84e7f4a2119c7b6e3521f8854d89c955f2580ec7441481976f2be9bc not found: ID does not exist" containerID="7b9772fd84e7f4a2119c7b6e3521f8854d89c955f2580ec7441481976f2be9bc"
Aug 01 00:07:16.386804 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:07:16.386757    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7b9772fd84e7f4a2119c7b6e3521f8854d89c955f2580ec7441481976f2be9bc"} err="failed to get container status \"7b9772fd84e7f4a2119c7b6e3521f8854d89c955f2580ec7441481976f2be9bc\": rpc error: code = NotFound desc = could not find container \"7b9772fd84e7f4a2119c7b6e3521f8854d89c955f2580ec7441481976f2be9bc\": container with ID starting with 7b9772fd84e7f4a2119c7b6e3521f8854d89c955f2580ec7441481976f2be9bc not found: ID does not exist"
Aug 01 00:07:16.389885 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:07:16.389865    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-fe2ca-predictor-698655d7c5-6h276"]
Aug 01 00:07:16.393556 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:07:16.393536    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-fe2ca-predictor-698655d7c5-6h276"]
Aug 01 00:07:16.490464 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:07:16.490432    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cde6e7e4-4775-491c-8965-25508d23d619" path="/var/lib/kubelet/pods/cde6e7e4-4775-491c-8965-25508d23d619/volumes"
Aug 01 00:07:19.302431 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:07:19.302390    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/ensemble-graph-fe2ca-7b566d4bb8-77nlb" podUID="00818bdd-5ee9-4a26-ba26-0ecadbf181c8" containerName="ensemble-graph-fe2ca" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Aug 01 00:07:19.944092 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:07:19.944050    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-a3ab7-5cc8864447-tbfbv" podUID="fcfc7c13-06ba-4fb4-a473-c3323bc9e4b8" containerName="sequence-graph-a3ab7" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Aug 01 00:07:20.360397 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:07:20.360372    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-01c7a-predictor-774957f645-hvd2w"
Aug 01 00:07:20.360877 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:07:20.360851    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-01c7a-predictor-774957f645-hvd2w" podUID="b6eed2b1-cef9-457a-b42d-99effb74a9a4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.40:8080: connect: connection refused"
Aug 01 00:07:21.263824 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:07:21.263784    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-df362-predictor-ddc8d9667-jd4hz" podUID="b5f4558a-d013-46cb-98b6-84f50810cdba" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.38:8080: connect: connection refused"
Aug 01 00:07:22.161779 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:07:22.161749    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sequence-graph-a3ab7-5cc8864447-tbfbv"
Aug 01 00:07:22.271643 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:07:22.271552    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fcfc7c13-06ba-4fb4-a473-c3323bc9e4b8-openshift-service-ca-bundle\") pod \"fcfc7c13-06ba-4fb4-a473-c3323bc9e4b8\" (UID: \"fcfc7c13-06ba-4fb4-a473-c3323bc9e4b8\") "
Aug 01 00:07:22.271643 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:07:22.271612    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fcfc7c13-06ba-4fb4-a473-c3323bc9e4b8-proxy-tls\") pod \"fcfc7c13-06ba-4fb4-a473-c3323bc9e4b8\" (UID: \"fcfc7c13-06ba-4fb4-a473-c3323bc9e4b8\") "
Aug 01 00:07:22.271949 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:07:22.271924    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fcfc7c13-06ba-4fb4-a473-c3323bc9e4b8-openshift-service-ca-bundle" (OuterVolumeSpecName: "openshift-service-ca-bundle") pod "fcfc7c13-06ba-4fb4-a473-c3323bc9e4b8" (UID: "fcfc7c13-06ba-4fb4-a473-c3323bc9e4b8"). InnerVolumeSpecName "openshift-service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 01 00:07:22.273707 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:07:22.273672    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fcfc7c13-06ba-4fb4-a473-c3323bc9e4b8-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "fcfc7c13-06ba-4fb4-a473-c3323bc9e4b8" (UID: "fcfc7c13-06ba-4fb4-a473-c3323bc9e4b8"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 01 00:07:22.372242 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:07:22.372201    2578 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fcfc7c13-06ba-4fb4-a473-c3323bc9e4b8-openshift-service-ca-bundle\") on node \"ip-10-0-136-108.ec2.internal\" DevicePath \"\""
Aug 01 00:07:22.372242 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:07:22.372233    2578 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fcfc7c13-06ba-4fb4-a473-c3323bc9e4b8-proxy-tls\") on node \"ip-10-0-136-108.ec2.internal\" DevicePath \"\""
Aug 01 00:07:22.398146 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:07:22.398110    2578 generic.go:356] "Generic (PLEG): container finished" podID="fcfc7c13-06ba-4fb4-a473-c3323bc9e4b8" containerID="e68e8bf27a2ddc3ff68ba9ecef9aab5e1bf8c6f5bf22f497ed519db9147324e0" exitCode=0
Aug 01 00:07:22.398325 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:07:22.398154    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sequence-graph-a3ab7-5cc8864447-tbfbv"
Aug 01 00:07:22.398325 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:07:22.398202    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sequence-graph-a3ab7-5cc8864447-tbfbv" event={"ID":"fcfc7c13-06ba-4fb4-a473-c3323bc9e4b8","Type":"ContainerDied","Data":"e68e8bf27a2ddc3ff68ba9ecef9aab5e1bf8c6f5bf22f497ed519db9147324e0"}
Aug 01 00:07:22.398325 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:07:22.398259    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sequence-graph-a3ab7-5cc8864447-tbfbv" event={"ID":"fcfc7c13-06ba-4fb4-a473-c3323bc9e4b8","Type":"ContainerDied","Data":"225161d1f0e2188d003c3d9035a945036bd107b7b8b7002ef73d9b719c9a865e"}
Aug 01 00:07:22.398325 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:07:22.398281    2578 scope.go:117] "RemoveContainer" containerID="e68e8bf27a2ddc3ff68ba9ecef9aab5e1bf8c6f5bf22f497ed519db9147324e0"
Aug 01 00:07:22.407453 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:07:22.407433    2578 scope.go:117] "RemoveContainer" containerID="e68e8bf27a2ddc3ff68ba9ecef9aab5e1bf8c6f5bf22f497ed519db9147324e0"
Aug 01 00:07:22.407692 ip-10-0-136-108 kubenswrapper[2578]: E0801 00:07:22.407675    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e68e8bf27a2ddc3ff68ba9ecef9aab5e1bf8c6f5bf22f497ed519db9147324e0\": container with ID starting with e68e8bf27a2ddc3ff68ba9ecef9aab5e1bf8c6f5bf22f497ed519db9147324e0 not found: ID does not exist" containerID="e68e8bf27a2ddc3ff68ba9ecef9aab5e1bf8c6f5bf22f497ed519db9147324e0"
Aug 01 00:07:22.407754 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:07:22.407700    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e68e8bf27a2ddc3ff68ba9ecef9aab5e1bf8c6f5bf22f497ed519db9147324e0"} err="failed to get container status \"e68e8bf27a2ddc3ff68ba9ecef9aab5e1bf8c6f5bf22f497ed519db9147324e0\": rpc error: code = NotFound desc = could not find container \"e68e8bf27a2ddc3ff68ba9ecef9aab5e1bf8c6f5bf22f497ed519db9147324e0\": container with ID starting with e68e8bf27a2ddc3ff68ba9ecef9aab5e1bf8c6f5bf22f497ed519db9147324e0 not found: ID does not exist"
Aug 01 00:07:22.418997 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:07:22.418974    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-a3ab7-5cc8864447-tbfbv"]
Aug 01 00:07:22.422741 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:07:22.422719    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-a3ab7-5cc8864447-tbfbv"]
Aug 01 00:07:22.490650 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:07:22.490618    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fcfc7c13-06ba-4fb4-a473-c3323bc9e4b8" path="/var/lib/kubelet/pods/fcfc7c13-06ba-4fb4-a473-c3323bc9e4b8/volumes"
Aug 01 00:07:24.302008 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:07:24.301963    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/ensemble-graph-fe2ca-7b566d4bb8-77nlb" podUID="00818bdd-5ee9-4a26-ba26-0ecadbf181c8" containerName="ensemble-graph-fe2ca" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Aug 01 00:07:24.302467 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:07:24.302075    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/ensemble-graph-fe2ca-7b566d4bb8-77nlb"
Aug 01 00:07:29.302039 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:07:29.302001    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/ensemble-graph-fe2ca-7b566d4bb8-77nlb" podUID="00818bdd-5ee9-4a26-ba26-0ecadbf181c8" containerName="ensemble-graph-fe2ca" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Aug 01 00:07:30.361332 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:07:30.361287    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-01c7a-predictor-774957f645-hvd2w" podUID="b6eed2b1-cef9-457a-b42d-99effb74a9a4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.40:8080: connect: connection refused"
Aug 01 00:07:31.263360 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:07:31.263320    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-df362-predictor-ddc8d9667-jd4hz" podUID="b5f4558a-d013-46cb-98b6-84f50810cdba" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.38:8080: connect: connection refused"
Aug 01 00:07:34.301749 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:07:34.301702    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/ensemble-graph-fe2ca-7b566d4bb8-77nlb" podUID="00818bdd-5ee9-4a26-ba26-0ecadbf181c8" containerName="ensemble-graph-fe2ca" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Aug 01 00:07:39.301934 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:07:39.301891    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/ensemble-graph-fe2ca-7b566d4bb8-77nlb" podUID="00818bdd-5ee9-4a26-ba26-0ecadbf181c8" containerName="ensemble-graph-fe2ca" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Aug 01 00:07:40.361679 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:07:40.361636    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-01c7a-predictor-774957f645-hvd2w" podUID="b6eed2b1-cef9-457a-b42d-99effb74a9a4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.40:8080: connect: connection refused"
Aug 01 00:07:41.264415 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:07:41.264368    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-df362-predictor-ddc8d9667-jd4hz"
Aug 01 00:07:42.661727 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:07:42.661702    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/ensemble-graph-fe2ca-7b566d4bb8-77nlb"
Aug 01 00:07:42.743112 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:07:42.743084    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/00818bdd-5ee9-4a26-ba26-0ecadbf181c8-openshift-service-ca-bundle\") pod \"00818bdd-5ee9-4a26-ba26-0ecadbf181c8\" (UID: \"00818bdd-5ee9-4a26-ba26-0ecadbf181c8\") "
Aug 01 00:07:42.743295 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:07:42.743143    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/00818bdd-5ee9-4a26-ba26-0ecadbf181c8-proxy-tls\") pod \"00818bdd-5ee9-4a26-ba26-0ecadbf181c8\" (UID: \"00818bdd-5ee9-4a26-ba26-0ecadbf181c8\") "
Aug 01 00:07:42.743522 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:07:42.743497    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/00818bdd-5ee9-4a26-ba26-0ecadbf181c8-openshift-service-ca-bundle" (OuterVolumeSpecName: "openshift-service-ca-bundle") pod "00818bdd-5ee9-4a26-ba26-0ecadbf181c8" (UID: "00818bdd-5ee9-4a26-ba26-0ecadbf181c8"). InnerVolumeSpecName "openshift-service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 01 00:07:42.745157 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:07:42.745119    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/00818bdd-5ee9-4a26-ba26-0ecadbf181c8-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "00818bdd-5ee9-4a26-ba26-0ecadbf181c8" (UID: "00818bdd-5ee9-4a26-ba26-0ecadbf181c8"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 01 00:07:42.844005 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:07:42.843962    2578 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/00818bdd-5ee9-4a26-ba26-0ecadbf181c8-openshift-service-ca-bundle\") on node \"ip-10-0-136-108.ec2.internal\" DevicePath \"\""
Aug 01 00:07:42.844005 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:07:42.843996    2578 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/00818bdd-5ee9-4a26-ba26-0ecadbf181c8-proxy-tls\") on node \"ip-10-0-136-108.ec2.internal\" DevicePath \"\""
Aug 01 00:07:43.496889 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:07:43.496850    2578 generic.go:356] "Generic (PLEG): container finished" podID="00818bdd-5ee9-4a26-ba26-0ecadbf181c8" containerID="c3ff822e674b816bc29982eb4e5b651233bec32cf991c68d7ef826ea88972647" exitCode=137
Aug 01 00:07:43.497079 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:07:43.496909    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/ensemble-graph-fe2ca-7b566d4bb8-77nlb"
Aug 01 00:07:43.497079 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:07:43.496932    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/ensemble-graph-fe2ca-7b566d4bb8-77nlb" event={"ID":"00818bdd-5ee9-4a26-ba26-0ecadbf181c8","Type":"ContainerDied","Data":"c3ff822e674b816bc29982eb4e5b651233bec32cf991c68d7ef826ea88972647"}
Aug 01 00:07:43.497079 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:07:43.496988    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/ensemble-graph-fe2ca-7b566d4bb8-77nlb" event={"ID":"00818bdd-5ee9-4a26-ba26-0ecadbf181c8","Type":"ContainerDied","Data":"3c75058f82ea1bf0180d8a074343c274f147fe478763625d474ad8ff768ef57d"}
Aug 01 00:07:43.497079 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:07:43.497019    2578 scope.go:117] "RemoveContainer" containerID="c3ff822e674b816bc29982eb4e5b651233bec32cf991c68d7ef826ea88972647"
Aug 01 00:07:43.506002 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:07:43.505987    2578 scope.go:117] "RemoveContainer" containerID="c3ff822e674b816bc29982eb4e5b651233bec32cf991c68d7ef826ea88972647"
Aug 01 00:07:43.506296 ip-10-0-136-108 kubenswrapper[2578]: E0801 00:07:43.506280    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c3ff822e674b816bc29982eb4e5b651233bec32cf991c68d7ef826ea88972647\": container with ID starting with c3ff822e674b816bc29982eb4e5b651233bec32cf991c68d7ef826ea88972647 not found: ID does not exist" containerID="c3ff822e674b816bc29982eb4e5b651233bec32cf991c68d7ef826ea88972647"
Aug 01 00:07:43.506342 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:07:43.506305    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c3ff822e674b816bc29982eb4e5b651233bec32cf991c68d7ef826ea88972647"} err="failed to get container status \"c3ff822e674b816bc29982eb4e5b651233bec32cf991c68d7ef826ea88972647\": rpc error: code = NotFound desc = could not find container \"c3ff822e674b816bc29982eb4e5b651233bec32cf991c68d7ef826ea88972647\": container with ID starting with c3ff822e674b816bc29982eb4e5b651233bec32cf991c68d7ef826ea88972647 not found: ID does not exist"
Aug 01 00:07:43.518382 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:07:43.518357    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/ensemble-graph-fe2ca-7b566d4bb8-77nlb"]
Aug 01 00:07:43.522393 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:07:43.522373    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/ensemble-graph-fe2ca-7b566d4bb8-77nlb"]
Aug 01 00:07:44.494661 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:07:44.494626    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="00818bdd-5ee9-4a26-ba26-0ecadbf181c8" path="/var/lib/kubelet/pods/00818bdd-5ee9-4a26-ba26-0ecadbf181c8/volumes"
Aug 01 00:07:50.361222 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:07:50.361164    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-01c7a-predictor-774957f645-hvd2w" podUID="b6eed2b1-cef9-457a-b42d-99effb74a9a4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.40:8080: connect: connection refused"
Aug 01 00:07:52.231536 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:07:52.231504    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/sequence-graph-df362-67945678d8-nr7bj"]
Aug 01 00:07:52.232004 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:07:52.231864    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cde6e7e4-4775-491c-8965-25508d23d619" containerName="kube-rbac-proxy"
Aug 01 00:07:52.232004 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:07:52.231876    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="cde6e7e4-4775-491c-8965-25508d23d619" containerName="kube-rbac-proxy"
Aug 01 00:07:52.232004 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:07:52.231886    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="00818bdd-5ee9-4a26-ba26-0ecadbf181c8" containerName="ensemble-graph-fe2ca"
Aug 01 00:07:52.232004 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:07:52.231892    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="00818bdd-5ee9-4a26-ba26-0ecadbf181c8" containerName="ensemble-graph-fe2ca"
Aug 01 00:07:52.232004 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:07:52.231907    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fcfc7c13-06ba-4fb4-a473-c3323bc9e4b8" containerName="sequence-graph-a3ab7"
Aug 01 00:07:52.232004 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:07:52.231913    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="fcfc7c13-06ba-4fb4-a473-c3323bc9e4b8" containerName="sequence-graph-a3ab7"
Aug 01 00:07:52.232004 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:07:52.231934    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cde6e7e4-4775-491c-8965-25508d23d619" containerName="kserve-container"
Aug 01 00:07:52.232004 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:07:52.231939    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="cde6e7e4-4775-491c-8965-25508d23d619" containerName="kserve-container"
Aug 01 00:07:52.232004 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:07:52.232002    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="fcfc7c13-06ba-4fb4-a473-c3323bc9e4b8" containerName="sequence-graph-a3ab7"
Aug 01 00:07:52.232341 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:07:52.232012    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="cde6e7e4-4775-491c-8965-25508d23d619" containerName="kube-rbac-proxy"
Aug 01 00:07:52.232341 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:07:52.232020    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="cde6e7e4-4775-491c-8965-25508d23d619" containerName="kserve-container"
Aug 01 00:07:52.232341 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:07:52.232027    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="00818bdd-5ee9-4a26-ba26-0ecadbf181c8" containerName="ensemble-graph-fe2ca"
Aug 01 00:07:52.234734 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:07:52.234713    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sequence-graph-df362-67945678d8-nr7bj"
Aug 01 00:07:52.236536 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:07:52.236495    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"sequence-graph-df362-serving-cert\""
Aug 01 00:07:52.236536 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:07:52.236514    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gttdm_b32e6d42-e4e0-4113-b8ce-f5d5c5dc13f7/konnectivity-agent/0.log" limit=1024
Aug 01 00:07:52.236730 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:07:52.236495    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"sequence-graph-df362-kube-rbac-proxy-sar-config\""
Aug 01 00:07:52.241998 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:07:52.241855    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-df362-67945678d8-nr7bj"]
Aug 01 00:07:52.326856 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:07:52.326823    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/65018b1f-2bdb-4b63-846c-99cdcc75aef5-proxy-tls\") pod \"sequence-graph-df362-67945678d8-nr7bj\" (UID: \"65018b1f-2bdb-4b63-846c-99cdcc75aef5\") " pod="kserve-ci-e2e-test/sequence-graph-df362-67945678d8-nr7bj"
Aug 01 00:07:52.327033 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:07:52.326893    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/65018b1f-2bdb-4b63-846c-99cdcc75aef5-openshift-service-ca-bundle\") pod \"sequence-graph-df362-67945678d8-nr7bj\" (UID: \"65018b1f-2bdb-4b63-846c-99cdcc75aef5\") " pod="kserve-ci-e2e-test/sequence-graph-df362-67945678d8-nr7bj"
Aug 01 00:07:52.427551 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:07:52.427510    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/65018b1f-2bdb-4b63-846c-99cdcc75aef5-proxy-tls\") pod \"sequence-graph-df362-67945678d8-nr7bj\" (UID: \"65018b1f-2bdb-4b63-846c-99cdcc75aef5\") " pod="kserve-ci-e2e-test/sequence-graph-df362-67945678d8-nr7bj"
Aug 01 00:07:52.427718 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:07:52.427614    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/65018b1f-2bdb-4b63-846c-99cdcc75aef5-openshift-service-ca-bundle\") pod \"sequence-graph-df362-67945678d8-nr7bj\" (UID: \"65018b1f-2bdb-4b63-846c-99cdcc75aef5\") " pod="kserve-ci-e2e-test/sequence-graph-df362-67945678d8-nr7bj"
Aug 01 00:07:52.427718 ip-10-0-136-108 kubenswrapper[2578]: E0801 00:07:52.427652    2578 secret.go:189] Couldn't get secret kserve-ci-e2e-test/sequence-graph-df362-serving-cert: secret "sequence-graph-df362-serving-cert" not found
Aug 01 00:07:52.427810 ip-10-0-136-108 kubenswrapper[2578]: E0801 00:07:52.427725    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/65018b1f-2bdb-4b63-846c-99cdcc75aef5-proxy-tls podName:65018b1f-2bdb-4b63-846c-99cdcc75aef5 nodeName:}" failed. No retries permitted until 2026-08-01 00:07:52.927708084 +0000 UTC m=+1296.895458820 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/65018b1f-2bdb-4b63-846c-99cdcc75aef5-proxy-tls") pod "sequence-graph-df362-67945678d8-nr7bj" (UID: "65018b1f-2bdb-4b63-846c-99cdcc75aef5") : secret "sequence-graph-df362-serving-cert" not found
Aug 01 00:07:52.428431 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:07:52.428409    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/65018b1f-2bdb-4b63-846c-99cdcc75aef5-openshift-service-ca-bundle\") pod \"sequence-graph-df362-67945678d8-nr7bj\" (UID: \"65018b1f-2bdb-4b63-846c-99cdcc75aef5\") " pod="kserve-ci-e2e-test/sequence-graph-df362-67945678d8-nr7bj"
Aug 01 00:07:52.932508 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:07:52.932459    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/65018b1f-2bdb-4b63-846c-99cdcc75aef5-proxy-tls\") pod \"sequence-graph-df362-67945678d8-nr7bj\" (UID: \"65018b1f-2bdb-4b63-846c-99cdcc75aef5\") " pod="kserve-ci-e2e-test/sequence-graph-df362-67945678d8-nr7bj"
Aug 01 00:07:52.932732 ip-10-0-136-108 kubenswrapper[2578]: E0801 00:07:52.932633    2578 secret.go:189] Couldn't get secret kserve-ci-e2e-test/sequence-graph-df362-serving-cert: secret "sequence-graph-df362-serving-cert" not found
Aug 01 00:07:52.932732 ip-10-0-136-108 kubenswrapper[2578]: E0801 00:07:52.932725    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/65018b1f-2bdb-4b63-846c-99cdcc75aef5-proxy-tls podName:65018b1f-2bdb-4b63-846c-99cdcc75aef5 nodeName:}" failed. No retries permitted until 2026-08-01 00:07:53.932704112 +0000 UTC m=+1297.900454862 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/65018b1f-2bdb-4b63-846c-99cdcc75aef5-proxy-tls") pod "sequence-graph-df362-67945678d8-nr7bj" (UID: "65018b1f-2bdb-4b63-846c-99cdcc75aef5") : secret "sequence-graph-df362-serving-cert" not found
Aug 01 00:07:53.254470 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:07:53.254436    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gttdm_b32e6d42-e4e0-4113-b8ce-f5d5c5dc13f7/konnectivity-agent/0.log" limit=1024
Aug 01 00:07:53.941827 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:07:53.941785    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/65018b1f-2bdb-4b63-846c-99cdcc75aef5-proxy-tls\") pod \"sequence-graph-df362-67945678d8-nr7bj\" (UID: \"65018b1f-2bdb-4b63-846c-99cdcc75aef5\") " pod="kserve-ci-e2e-test/sequence-graph-df362-67945678d8-nr7bj"
Aug 01 00:07:53.944229 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:07:53.944206    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/65018b1f-2bdb-4b63-846c-99cdcc75aef5-proxy-tls\") pod \"sequence-graph-df362-67945678d8-nr7bj\" (UID: \"65018b1f-2bdb-4b63-846c-99cdcc75aef5\") " pod="kserve-ci-e2e-test/sequence-graph-df362-67945678d8-nr7bj"
Aug 01 00:07:54.046085 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:07:54.046053    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sequence-graph-df362-67945678d8-nr7bj"
Aug 01 00:07:54.176649 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:07:54.176621    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-df362-67945678d8-nr7bj"]
Aug 01 00:07:54.178490 ip-10-0-136-108 kubenswrapper[2578]: W0801 00:07:54.178457    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod65018b1f_2bdb_4b63_846c_99cdcc75aef5.slice/crio-eb8167ba42ffb08aaee002096ddc1d56fe273ddbc56c52be79302d6be8fc3078 WatchSource:0}: Error finding container eb8167ba42ffb08aaee002096ddc1d56fe273ddbc56c52be79302d6be8fc3078: Status 404 returned error can't find the container with id eb8167ba42ffb08aaee002096ddc1d56fe273ddbc56c52be79302d6be8fc3078
Aug 01 00:07:54.550562 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:07:54.550525    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sequence-graph-df362-67945678d8-nr7bj" event={"ID":"65018b1f-2bdb-4b63-846c-99cdcc75aef5","Type":"ContainerStarted","Data":"22785f7879ac2c803891308d4de17ccfba4f9357f64c473751100f6c25e3c361"}
Aug 01 00:07:54.551012 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:07:54.550577    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sequence-graph-df362-67945678d8-nr7bj" event={"ID":"65018b1f-2bdb-4b63-846c-99cdcc75aef5","Type":"ContainerStarted","Data":"eb8167ba42ffb08aaee002096ddc1d56fe273ddbc56c52be79302d6be8fc3078"}
Aug 01 00:07:54.551012 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:07:54.550708    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/sequence-graph-df362-67945678d8-nr7bj"
Aug 01 00:07:54.565413 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:07:54.565356    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/sequence-graph-df362-67945678d8-nr7bj" podStartSLOduration=2.5653365299999997 podStartE2EDuration="2.56533653s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-01 00:07:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-01 00:07:54.56465657 +0000 UTC m=+1298.532407329" watchObservedRunningTime="2026-08-01 00:07:54.56533653 +0000 UTC m=+1298.533087288"
Aug 01 00:08:00.362050 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:08:00.362022    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-01c7a-predictor-774957f645-hvd2w"
Aug 01 00:08:00.563106 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:08:00.563078    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/sequence-graph-df362-67945678d8-nr7bj"
Aug 01 00:08:02.318197 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:08:02.316419    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-df362-67945678d8-nr7bj"]
Aug 01 00:08:02.318197 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:08:02.316766    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/sequence-graph-df362-67945678d8-nr7bj" podUID="65018b1f-2bdb-4b63-846c-99cdcc75aef5" containerName="sequence-graph-df362" containerID="cri-o://22785f7879ac2c803891308d4de17ccfba4f9357f64c473751100f6c25e3c361" gracePeriod=30
Aug 01 00:08:02.332942 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:08:02.332906    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gttdm_b32e6d42-e4e0-4113-b8ce-f5d5c5dc13f7/konnectivity-agent/0.log" limit=1024
Aug 01 00:08:02.437210 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:08:02.437137    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-df362-predictor-ddc8d9667-jd4hz"]
Aug 01 00:08:02.437432 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:08:02.437384    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-df362-predictor-ddc8d9667-jd4hz" podUID="b5f4558a-d013-46cb-98b6-84f50810cdba" containerName="kserve-container" containerID="cri-o://c9209c344361dcc6bbdb6d31a8efbbf83182e645b9b2e953aa0786e9d395d934" gracePeriod=30
Aug 01 00:08:02.437432 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:08:02.437425    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-df362-predictor-ddc8d9667-jd4hz" podUID="b5f4558a-d013-46cb-98b6-84f50810cdba" containerName="kube-rbac-proxy" containerID="cri-o://49a9532e43b0046ef5c23597b1862e6bd5460a717320fb1e1e0ff40eac9a330f" gracePeriod=30
Aug 01 00:08:02.460709 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:08:02.460663    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gttdm_b32e6d42-e4e0-4113-b8ce-f5d5c5dc13f7/konnectivity-agent/0.log" limit=1024
Aug 01 00:08:02.511346 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:08:02.511315    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-d99eb-predictor-5bf7f45f47-77mxx"]
Aug 01 00:08:02.515185 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:08:02.515154    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-d99eb-predictor-5bf7f45f47-77mxx"
Aug 01 00:08:02.517054 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:08:02.517026    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-d99eb-predictor-serving-cert\""
Aug 01 00:08:02.517195 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:08:02.517164    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-d99eb-kube-rbac-proxy-sar-config\""
Aug 01 00:08:02.522744 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:08:02.522579    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-d99eb-predictor-5bf7f45f47-77mxx"]
Aug 01 00:08:02.527958 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:08:02.527927    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gttdm_b32e6d42-e4e0-4113-b8ce-f5d5c5dc13f7/konnectivity-agent/0.log" limit=1024
Aug 01 00:08:02.593000 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:08:02.592914    2578 generic.go:356] "Generic (PLEG): container finished" podID="b5f4558a-d013-46cb-98b6-84f50810cdba" containerID="49a9532e43b0046ef5c23597b1862e6bd5460a717320fb1e1e0ff40eac9a330f" exitCode=2
Aug 01 00:08:02.593000 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:08:02.592988    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-df362-predictor-ddc8d9667-jd4hz" event={"ID":"b5f4558a-d013-46cb-98b6-84f50810cdba","Type":"ContainerDied","Data":"49a9532e43b0046ef5c23597b1862e6bd5460a717320fb1e1e0ff40eac9a330f"}
Aug 01 00:08:02.594438 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:08:02.594408    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gttdm_b32e6d42-e4e0-4113-b8ce-f5d5c5dc13f7/konnectivity-agent/0.log" limit=1024
Aug 01 00:08:02.620512 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:08:02.620468    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2da97b83-5b75-4fb7-912b-0e2dd8d1444d-proxy-tls\") pod \"success-200-isvc-d99eb-predictor-5bf7f45f47-77mxx\" (UID: \"2da97b83-5b75-4fb7-912b-0e2dd8d1444d\") " pod="kserve-ci-e2e-test/success-200-isvc-d99eb-predictor-5bf7f45f47-77mxx"
Aug 01 00:08:02.620713 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:08:02.620606    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2da97b83-5b75-4fb7-912b-0e2dd8d1444d-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-d99eb-predictor-5bf7f45f47-77mxx\" (UID: \"2da97b83-5b75-4fb7-912b-0e2dd8d1444d\") " pod="kserve-ci-e2e-test/success-200-isvc-d99eb-predictor-5bf7f45f47-77mxx"
Aug 01 00:08:02.620713 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:08:02.620645    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bf9g8\" (UniqueName: \"kubernetes.io/projected/2da97b83-5b75-4fb7-912b-0e2dd8d1444d-kube-api-access-bf9g8\") pod \"success-200-isvc-d99eb-predictor-5bf7f45f47-77mxx\" (UID: \"2da97b83-5b75-4fb7-912b-0e2dd8d1444d\") " pod="kserve-ci-e2e-test/success-200-isvc-d99eb-predictor-5bf7f45f47-77mxx"
Aug 01 00:08:02.721784 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:08:02.721751    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2da97b83-5b75-4fb7-912b-0e2dd8d1444d-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-d99eb-predictor-5bf7f45f47-77mxx\" (UID: \"2da97b83-5b75-4fb7-912b-0e2dd8d1444d\") " pod="kserve-ci-e2e-test/success-200-isvc-d99eb-predictor-5bf7f45f47-77mxx"
Aug 01 00:08:02.721980 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:08:02.721792    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bf9g8\" (UniqueName: \"kubernetes.io/projected/2da97b83-5b75-4fb7-912b-0e2dd8d1444d-kube-api-access-bf9g8\") pod \"success-200-isvc-d99eb-predictor-5bf7f45f47-77mxx\" (UID: \"2da97b83-5b75-4fb7-912b-0e2dd8d1444d\") " pod="kserve-ci-e2e-test/success-200-isvc-d99eb-predictor-5bf7f45f47-77mxx"
Aug 01 00:08:02.721980 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:08:02.721833    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2da97b83-5b75-4fb7-912b-0e2dd8d1444d-proxy-tls\") pod \"success-200-isvc-d99eb-predictor-5bf7f45f47-77mxx\" (UID: \"2da97b83-5b75-4fb7-912b-0e2dd8d1444d\") " pod="kserve-ci-e2e-test/success-200-isvc-d99eb-predictor-5bf7f45f47-77mxx"
Aug 01 00:08:02.721980 ip-10-0-136-108 kubenswrapper[2578]: E0801 00:08:02.721960    2578 secret.go:189] Couldn't get secret kserve-ci-e2e-test/success-200-isvc-d99eb-predictor-serving-cert: secret "success-200-isvc-d99eb-predictor-serving-cert" not found
Aug 01 00:08:02.722128 ip-10-0-136-108 kubenswrapper[2578]: E0801 00:08:02.722023    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2da97b83-5b75-4fb7-912b-0e2dd8d1444d-proxy-tls podName:2da97b83-5b75-4fb7-912b-0e2dd8d1444d nodeName:}" failed. No retries permitted until 2026-08-01 00:08:03.222007289 +0000 UTC m=+1307.189758025 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/2da97b83-5b75-4fb7-912b-0e2dd8d1444d-proxy-tls") pod "success-200-isvc-d99eb-predictor-5bf7f45f47-77mxx" (UID: "2da97b83-5b75-4fb7-912b-0e2dd8d1444d") : secret "success-200-isvc-d99eb-predictor-serving-cert" not found
Aug 01 00:08:02.722419 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:08:02.722399    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2da97b83-5b75-4fb7-912b-0e2dd8d1444d-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-d99eb-predictor-5bf7f45f47-77mxx\" (UID: \"2da97b83-5b75-4fb7-912b-0e2dd8d1444d\") " pod="kserve-ci-e2e-test/success-200-isvc-d99eb-predictor-5bf7f45f47-77mxx"
Aug 01 00:08:02.730235 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:08:02.730204    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bf9g8\" (UniqueName: \"kubernetes.io/projected/2da97b83-5b75-4fb7-912b-0e2dd8d1444d-kube-api-access-bf9g8\") pod \"success-200-isvc-d99eb-predictor-5bf7f45f47-77mxx\" (UID: \"2da97b83-5b75-4fb7-912b-0e2dd8d1444d\") " pod="kserve-ci-e2e-test/success-200-isvc-d99eb-predictor-5bf7f45f47-77mxx"
Aug 01 00:08:02.771963 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:08:02.771922    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gttdm_b32e6d42-e4e0-4113-b8ce-f5d5c5dc13f7/konnectivity-agent/0.log" limit=1024
Aug 01 00:08:02.874212 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:08:02.874109    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gttdm_b32e6d42-e4e0-4113-b8ce-f5d5c5dc13f7/konnectivity-agent/0.log" limit=1024
Aug 01 00:08:03.227278 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:08:03.227186    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2da97b83-5b75-4fb7-912b-0e2dd8d1444d-proxy-tls\") pod \"success-200-isvc-d99eb-predictor-5bf7f45f47-77mxx\" (UID: \"2da97b83-5b75-4fb7-912b-0e2dd8d1444d\") " pod="kserve-ci-e2e-test/success-200-isvc-d99eb-predictor-5bf7f45f47-77mxx"
Aug 01 00:08:03.229718 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:08:03.229691    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2da97b83-5b75-4fb7-912b-0e2dd8d1444d-proxy-tls\") pod \"success-200-isvc-d99eb-predictor-5bf7f45f47-77mxx\" (UID: \"2da97b83-5b75-4fb7-912b-0e2dd8d1444d\") " pod="kserve-ci-e2e-test/success-200-isvc-d99eb-predictor-5bf7f45f47-77mxx"
Aug 01 00:08:03.426578 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:08:03.426537    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-d99eb-predictor-5bf7f45f47-77mxx"
Aug 01 00:08:03.559583 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:08:03.559557    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-d99eb-predictor-5bf7f45f47-77mxx"]
Aug 01 00:08:03.561845 ip-10-0-136-108 kubenswrapper[2578]: W0801 00:08:03.561816    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2da97b83_5b75_4fb7_912b_0e2dd8d1444d.slice/crio-8f7f876364b6f7abafd91d6616a182c4ea473e8591481147f6e5dfab4e370595 WatchSource:0}: Error finding container 8f7f876364b6f7abafd91d6616a182c4ea473e8591481147f6e5dfab4e370595: Status 404 returned error can't find the container with id 8f7f876364b6f7abafd91d6616a182c4ea473e8591481147f6e5dfab4e370595
Aug 01 00:08:03.599336 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:08:03.599309    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-d99eb-predictor-5bf7f45f47-77mxx" event={"ID":"2da97b83-5b75-4fb7-912b-0e2dd8d1444d","Type":"ContainerStarted","Data":"8f7f876364b6f7abafd91d6616a182c4ea473e8591481147f6e5dfab4e370595"}
Aug 01 00:08:04.605302 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:08:04.605264    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-d99eb-predictor-5bf7f45f47-77mxx" event={"ID":"2da97b83-5b75-4fb7-912b-0e2dd8d1444d","Type":"ContainerStarted","Data":"f284b0f19771ebdc7b42558a59c4771b4ccb1ce292a8ae290d4e7292e535aeb4"}
Aug 01 00:08:04.605728 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:08:04.605331    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-d99eb-predictor-5bf7f45f47-77mxx" event={"ID":"2da97b83-5b75-4fb7-912b-0e2dd8d1444d","Type":"ContainerStarted","Data":"6e2e0b1a0abe31e3f6866b2990daca0f8b45bb58eaf8cfaaf7d5a3ad01687882"}
Aug 01 00:08:04.605728 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:08:04.605359    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-d99eb-predictor-5bf7f45f47-77mxx"
Aug 01 00:08:04.622953 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:08:04.622903    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/success-200-isvc-d99eb-predictor-5bf7f45f47-77mxx" podStartSLOduration=2.6228861930000003 podStartE2EDuration="2.622886193s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-01 00:08:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-01 00:08:04.621053408 +0000 UTC m=+1308.588804167" watchObservedRunningTime="2026-08-01 00:08:04.622886193 +0000 UTC m=+1308.590636950"
Aug 01 00:08:05.562221 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:08:05.562183    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-df362-67945678d8-nr7bj" podUID="65018b1f-2bdb-4b63-846c-99cdcc75aef5" containerName="sequence-graph-df362" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Aug 01 00:08:05.612843 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:08:05.612806    2578 generic.go:356] "Generic (PLEG): container finished" podID="b5f4558a-d013-46cb-98b6-84f50810cdba" containerID="c9209c344361dcc6bbdb6d31a8efbbf83182e645b9b2e953aa0786e9d395d934" exitCode=0
Aug 01 00:08:05.613212 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:08:05.612835    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-df362-predictor-ddc8d9667-jd4hz" event={"ID":"b5f4558a-d013-46cb-98b6-84f50810cdba","Type":"ContainerDied","Data":"c9209c344361dcc6bbdb6d31a8efbbf83182e645b9b2e953aa0786e9d395d934"}
Aug 01 00:08:05.613291 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:08:05.613210    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-d99eb-predictor-5bf7f45f47-77mxx"
Aug 01 00:08:05.614457 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:08:05.614433    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-d99eb-predictor-5bf7f45f47-77mxx" podUID="2da97b83-5b75-4fb7-912b-0e2dd8d1444d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.42:8080: connect: connection refused"
Aug 01 00:08:05.691384 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:08:05.691361    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-df362-predictor-ddc8d9667-jd4hz"
Aug 01 00:08:05.855401 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:08:05.855300    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8cql2\" (UniqueName: \"kubernetes.io/projected/b5f4558a-d013-46cb-98b6-84f50810cdba-kube-api-access-8cql2\") pod \"b5f4558a-d013-46cb-98b6-84f50810cdba\" (UID: \"b5f4558a-d013-46cb-98b6-84f50810cdba\") "
Aug 01 00:08:05.855401 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:08:05.855380    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b5f4558a-d013-46cb-98b6-84f50810cdba-proxy-tls\") pod \"b5f4558a-d013-46cb-98b6-84f50810cdba\" (UID: \"b5f4558a-d013-46cb-98b6-84f50810cdba\") "
Aug 01 00:08:05.855664 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:08:05.855439    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b5f4558a-d013-46cb-98b6-84f50810cdba-kube-rbac-proxy-sar-config\") pod \"b5f4558a-d013-46cb-98b6-84f50810cdba\" (UID: \"b5f4558a-d013-46cb-98b6-84f50810cdba\") "
Aug 01 00:08:05.855999 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:08:05.855968    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b5f4558a-d013-46cb-98b6-84f50810cdba-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "b5f4558a-d013-46cb-98b6-84f50810cdba" (UID: "b5f4558a-d013-46cb-98b6-84f50810cdba"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 01 00:08:05.857724 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:08:05.857694    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b5f4558a-d013-46cb-98b6-84f50810cdba-kube-api-access-8cql2" (OuterVolumeSpecName: "kube-api-access-8cql2") pod "b5f4558a-d013-46cb-98b6-84f50810cdba" (UID: "b5f4558a-d013-46cb-98b6-84f50810cdba"). InnerVolumeSpecName "kube-api-access-8cql2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 01 00:08:05.857808 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:08:05.857706    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b5f4558a-d013-46cb-98b6-84f50810cdba-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "b5f4558a-d013-46cb-98b6-84f50810cdba" (UID: "b5f4558a-d013-46cb-98b6-84f50810cdba"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 01 00:08:05.956907 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:08:05.956871    2578 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b5f4558a-d013-46cb-98b6-84f50810cdba-kube-rbac-proxy-sar-config\") on node \"ip-10-0-136-108.ec2.internal\" DevicePath \"\""
Aug 01 00:08:05.956907 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:08:05.956899    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8cql2\" (UniqueName: \"kubernetes.io/projected/b5f4558a-d013-46cb-98b6-84f50810cdba-kube-api-access-8cql2\") on node \"ip-10-0-136-108.ec2.internal\" DevicePath \"\""
Aug 01 00:08:05.956907 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:08:05.956909    2578 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b5f4558a-d013-46cb-98b6-84f50810cdba-proxy-tls\") on node \"ip-10-0-136-108.ec2.internal\" DevicePath \"\""
Aug 01 00:08:06.619055 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:08:06.619029    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-df362-predictor-ddc8d9667-jd4hz"
Aug 01 00:08:06.619055 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:08:06.619044    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-df362-predictor-ddc8d9667-jd4hz" event={"ID":"b5f4558a-d013-46cb-98b6-84f50810cdba","Type":"ContainerDied","Data":"b01795bfd26869f4ea209c0273e8b32e6ba00a34b34ba693b425ade296abbd16"}
Aug 01 00:08:06.619540 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:08:06.619122    2578 scope.go:117] "RemoveContainer" containerID="49a9532e43b0046ef5c23597b1862e6bd5460a717320fb1e1e0ff40eac9a330f"
Aug 01 00:08:06.619787 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:08:06.619750    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-d99eb-predictor-5bf7f45f47-77mxx" podUID="2da97b83-5b75-4fb7-912b-0e2dd8d1444d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.42:8080: connect: connection refused"
Aug 01 00:08:06.627867 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:08:06.627837    2578 scope.go:117] "RemoveContainer" containerID="c9209c344361dcc6bbdb6d31a8efbbf83182e645b9b2e953aa0786e9d395d934"
Aug 01 00:08:06.637271 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:08:06.637250    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-df362-predictor-ddc8d9667-jd4hz"]
Aug 01 00:08:06.644518 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:08:06.644488    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-df362-predictor-ddc8d9667-jd4hz"]
Aug 01 00:08:08.491434 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:08:08.491399    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b5f4558a-d013-46cb-98b6-84f50810cdba" path="/var/lib/kubelet/pods/b5f4558a-d013-46cb-98b6-84f50810cdba/volumes"
Aug 01 00:08:10.561251 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:08:10.561202    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-df362-67945678d8-nr7bj" podUID="65018b1f-2bdb-4b63-846c-99cdcc75aef5" containerName="sequence-graph-df362" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Aug 01 00:08:11.625498 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:08:11.625471    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-d99eb-predictor-5bf7f45f47-77mxx"
Aug 01 00:08:11.625892 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:08:11.625868    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-d99eb-predictor-5bf7f45f47-77mxx" podUID="2da97b83-5b75-4fb7-912b-0e2dd8d1444d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.42:8080: connect: connection refused"
Aug 01 00:08:12.775593 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:08:12.775537    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gttdm_b32e6d42-e4e0-4113-b8ce-f5d5c5dc13f7/konnectivity-agent/0.log" limit=1024
Aug 01 00:08:12.796690 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:08:12.796657    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/ensemble-graph-01c7a-7f54567bd4-96nwd"]
Aug 01 00:08:12.797069 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:08:12.797053    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b5f4558a-d013-46cb-98b6-84f50810cdba" containerName="kube-rbac-proxy"
Aug 01 00:08:12.797110 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:08:12.797071    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="b5f4558a-d013-46cb-98b6-84f50810cdba" containerName="kube-rbac-proxy"
Aug 01 00:08:12.797110 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:08:12.797087    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b5f4558a-d013-46cb-98b6-84f50810cdba" containerName="kserve-container"
Aug 01 00:08:12.797110 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:08:12.797093    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="b5f4558a-d013-46cb-98b6-84f50810cdba" containerName="kserve-container"
Aug 01 00:08:12.797221 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:08:12.797163    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="b5f4558a-d013-46cb-98b6-84f50810cdba" containerName="kube-rbac-proxy"
Aug 01 00:08:12.797221 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:08:12.797184    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="b5f4558a-d013-46cb-98b6-84f50810cdba" containerName="kserve-container"
Aug 01 00:08:12.801588 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:08:12.801569    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/ensemble-graph-01c7a-7f54567bd4-96nwd"
Aug 01 00:08:12.803895 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:08:12.803879    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"ensemble-graph-01c7a-serving-cert\""
Aug 01 00:08:12.803895 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:08:12.803888    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"ensemble-graph-01c7a-kube-rbac-proxy-sar-config\""
Aug 01 00:08:12.809108 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:08:12.809082    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/65eab692-528b-4c72-bc0b-4fdb2f5b4678-openshift-service-ca-bundle\") pod \"ensemble-graph-01c7a-7f54567bd4-96nwd\" (UID: \"65eab692-528b-4c72-bc0b-4fdb2f5b4678\") " pod="kserve-ci-e2e-test/ensemble-graph-01c7a-7f54567bd4-96nwd"
Aug 01 00:08:12.809306 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:08:12.809158    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/65eab692-528b-4c72-bc0b-4fdb2f5b4678-proxy-tls\") pod \"ensemble-graph-01c7a-7f54567bd4-96nwd\" (UID: \"65eab692-528b-4c72-bc0b-4fdb2f5b4678\") " pod="kserve-ci-e2e-test/ensemble-graph-01c7a-7f54567bd4-96nwd"
Aug 01 00:08:12.810854 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:08:12.810832    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/ensemble-graph-01c7a-7f54567bd4-96nwd"]
Aug 01 00:08:12.910028 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:08:12.909989    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/65eab692-528b-4c72-bc0b-4fdb2f5b4678-openshift-service-ca-bundle\") pod \"ensemble-graph-01c7a-7f54567bd4-96nwd\" (UID: \"65eab692-528b-4c72-bc0b-4fdb2f5b4678\") " pod="kserve-ci-e2e-test/ensemble-graph-01c7a-7f54567bd4-96nwd"
Aug 01 00:08:12.910256 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:08:12.910067    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/65eab692-528b-4c72-bc0b-4fdb2f5b4678-proxy-tls\") pod \"ensemble-graph-01c7a-7f54567bd4-96nwd\" (UID: \"65eab692-528b-4c72-bc0b-4fdb2f5b4678\") " pod="kserve-ci-e2e-test/ensemble-graph-01c7a-7f54567bd4-96nwd"
Aug 01 00:08:12.910256 ip-10-0-136-108 kubenswrapper[2578]: E0801 00:08:12.910237    2578 secret.go:189] Couldn't get secret kserve-ci-e2e-test/ensemble-graph-01c7a-serving-cert: secret "ensemble-graph-01c7a-serving-cert" not found
Aug 01 00:08:12.910353 ip-10-0-136-108 kubenswrapper[2578]: E0801 00:08:12.910311    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/65eab692-528b-4c72-bc0b-4fdb2f5b4678-proxy-tls podName:65eab692-528b-4c72-bc0b-4fdb2f5b4678 nodeName:}" failed. No retries permitted until 2026-08-01 00:08:13.410291728 +0000 UTC m=+1317.378042465 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/65eab692-528b-4c72-bc0b-4fdb2f5b4678-proxy-tls") pod "ensemble-graph-01c7a-7f54567bd4-96nwd" (UID: "65eab692-528b-4c72-bc0b-4fdb2f5b4678") : secret "ensemble-graph-01c7a-serving-cert" not found
Aug 01 00:08:12.910692 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:08:12.910672    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/65eab692-528b-4c72-bc0b-4fdb2f5b4678-openshift-service-ca-bundle\") pod \"ensemble-graph-01c7a-7f54567bd4-96nwd\" (UID: \"65eab692-528b-4c72-bc0b-4fdb2f5b4678\") " pod="kserve-ci-e2e-test/ensemble-graph-01c7a-7f54567bd4-96nwd"
Aug 01 00:08:13.198052 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:08:13.197938    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gttdm_b32e6d42-e4e0-4113-b8ce-f5d5c5dc13f7/konnectivity-agent/0.log" limit=1024
Aug 01 00:08:13.414785 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:08:13.414751    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/65eab692-528b-4c72-bc0b-4fdb2f5b4678-proxy-tls\") pod \"ensemble-graph-01c7a-7f54567bd4-96nwd\" (UID: \"65eab692-528b-4c72-bc0b-4fdb2f5b4678\") " pod="kserve-ci-e2e-test/ensemble-graph-01c7a-7f54567bd4-96nwd"
Aug 01 00:08:13.417205 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:08:13.417165    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/65eab692-528b-4c72-bc0b-4fdb2f5b4678-proxy-tls\") pod \"ensemble-graph-01c7a-7f54567bd4-96nwd\" (UID: \"65eab692-528b-4c72-bc0b-4fdb2f5b4678\") " pod="kserve-ci-e2e-test/ensemble-graph-01c7a-7f54567bd4-96nwd"
Aug 01 00:08:13.712975 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:08:13.712937    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/ensemble-graph-01c7a-7f54567bd4-96nwd"
Aug 01 00:08:13.846970 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:08:13.846945    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/ensemble-graph-01c7a-7f54567bd4-96nwd"]
Aug 01 00:08:13.848931 ip-10-0-136-108 kubenswrapper[2578]: W0801 00:08:13.848904    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod65eab692_528b_4c72_bc0b_4fdb2f5b4678.slice/crio-3ba013fdac2eb5e509f14a5d437adbfdfcd627b9c04fdfa29a45dc8e0747ed04 WatchSource:0}: Error finding container 3ba013fdac2eb5e509f14a5d437adbfdfcd627b9c04fdfa29a45dc8e0747ed04: Status 404 returned error can't find the container with id 3ba013fdac2eb5e509f14a5d437adbfdfcd627b9c04fdfa29a45dc8e0747ed04
Aug 01 00:08:14.664576 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:08:14.664539    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/ensemble-graph-01c7a-7f54567bd4-96nwd" event={"ID":"65eab692-528b-4c72-bc0b-4fdb2f5b4678","Type":"ContainerStarted","Data":"5cd959c5599107444c9c499a8263b32c8eaac0aac5fe224b05e8e9ee1b2a8ce8"}
Aug 01 00:08:14.664773 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:08:14.664588    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/ensemble-graph-01c7a-7f54567bd4-96nwd" event={"ID":"65eab692-528b-4c72-bc0b-4fdb2f5b4678","Type":"ContainerStarted","Data":"3ba013fdac2eb5e509f14a5d437adbfdfcd627b9c04fdfa29a45dc8e0747ed04"}
Aug 01 00:08:14.664773 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:08:14.664618    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/ensemble-graph-01c7a-7f54567bd4-96nwd"
Aug 01 00:08:14.680949 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:08:14.680893    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/ensemble-graph-01c7a-7f54567bd4-96nwd" podStartSLOduration=2.680874649 podStartE2EDuration="2.680874649s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-01 00:08:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-01 00:08:14.679358532 +0000 UTC m=+1318.647109289" watchObservedRunningTime="2026-08-01 00:08:14.680874649 +0000 UTC m=+1318.648625406"
Aug 01 00:08:15.561863 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:08:15.561824    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-df362-67945678d8-nr7bj" podUID="65018b1f-2bdb-4b63-846c-99cdcc75aef5" containerName="sequence-graph-df362" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Aug 01 00:08:15.562250 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:08:15.561921    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/sequence-graph-df362-67945678d8-nr7bj"
Aug 01 00:08:20.561976 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:08:20.561933    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-df362-67945678d8-nr7bj" podUID="65018b1f-2bdb-4b63-846c-99cdcc75aef5" containerName="sequence-graph-df362" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Aug 01 00:08:20.675233 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:08:20.675198    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/ensemble-graph-01c7a-7f54567bd4-96nwd"
Aug 01 00:08:21.626737 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:08:21.626694    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-d99eb-predictor-5bf7f45f47-77mxx" podUID="2da97b83-5b75-4fb7-912b-0e2dd8d1444d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.42:8080: connect: connection refused"
Aug 01 00:08:25.562161 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:08:25.562115    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-df362-67945678d8-nr7bj" podUID="65018b1f-2bdb-4b63-846c-99cdcc75aef5" containerName="sequence-graph-df362" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Aug 01 00:08:30.561537 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:08:30.561492    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-df362-67945678d8-nr7bj" podUID="65018b1f-2bdb-4b63-846c-99cdcc75aef5" containerName="sequence-graph-df362" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Aug 01 00:08:31.626322 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:08:31.626283    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-d99eb-predictor-5bf7f45f47-77mxx" podUID="2da97b83-5b75-4fb7-912b-0e2dd8d1444d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.42:8080: connect: connection refused"
Aug 01 00:08:32.462844 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:08:32.462815    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sequence-graph-df362-67945678d8-nr7bj"
Aug 01 00:08:32.575895 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:08:32.575796    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/65018b1f-2bdb-4b63-846c-99cdcc75aef5-openshift-service-ca-bundle\") pod \"65018b1f-2bdb-4b63-846c-99cdcc75aef5\" (UID: \"65018b1f-2bdb-4b63-846c-99cdcc75aef5\") "
Aug 01 00:08:32.575895 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:08:32.575862    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/65018b1f-2bdb-4b63-846c-99cdcc75aef5-proxy-tls\") pod \"65018b1f-2bdb-4b63-846c-99cdcc75aef5\" (UID: \"65018b1f-2bdb-4b63-846c-99cdcc75aef5\") "
Aug 01 00:08:32.576251 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:08:32.576227    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/65018b1f-2bdb-4b63-846c-99cdcc75aef5-openshift-service-ca-bundle" (OuterVolumeSpecName: "openshift-service-ca-bundle") pod "65018b1f-2bdb-4b63-846c-99cdcc75aef5" (UID: "65018b1f-2bdb-4b63-846c-99cdcc75aef5"). InnerVolumeSpecName "openshift-service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 01 00:08:32.577977 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:08:32.577953    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/65018b1f-2bdb-4b63-846c-99cdcc75aef5-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "65018b1f-2bdb-4b63-846c-99cdcc75aef5" (UID: "65018b1f-2bdb-4b63-846c-99cdcc75aef5"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 01 00:08:32.676734 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:08:32.676698    2578 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/65018b1f-2bdb-4b63-846c-99cdcc75aef5-openshift-service-ca-bundle\") on node \"ip-10-0-136-108.ec2.internal\" DevicePath \"\""
Aug 01 00:08:32.676734 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:08:32.676729    2578 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/65018b1f-2bdb-4b63-846c-99cdcc75aef5-proxy-tls\") on node \"ip-10-0-136-108.ec2.internal\" DevicePath \"\""
Aug 01 00:08:32.751899 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:08:32.751863    2578 generic.go:356] "Generic (PLEG): container finished" podID="65018b1f-2bdb-4b63-846c-99cdcc75aef5" containerID="22785f7879ac2c803891308d4de17ccfba4f9357f64c473751100f6c25e3c361" exitCode=0
Aug 01 00:08:32.752086 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:08:32.751914    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sequence-graph-df362-67945678d8-nr7bj"
Aug 01 00:08:32.752086 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:08:32.751942    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sequence-graph-df362-67945678d8-nr7bj" event={"ID":"65018b1f-2bdb-4b63-846c-99cdcc75aef5","Type":"ContainerDied","Data":"22785f7879ac2c803891308d4de17ccfba4f9357f64c473751100f6c25e3c361"}
Aug 01 00:08:32.752086 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:08:32.752006    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sequence-graph-df362-67945678d8-nr7bj" event={"ID":"65018b1f-2bdb-4b63-846c-99cdcc75aef5","Type":"ContainerDied","Data":"eb8167ba42ffb08aaee002096ddc1d56fe273ddbc56c52be79302d6be8fc3078"}
Aug 01 00:08:32.752086 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:08:32.752022    2578 scope.go:117] "RemoveContainer" containerID="22785f7879ac2c803891308d4de17ccfba4f9357f64c473751100f6c25e3c361"
Aug 01 00:08:32.760634 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:08:32.760610    2578 scope.go:117] "RemoveContainer" containerID="22785f7879ac2c803891308d4de17ccfba4f9357f64c473751100f6c25e3c361"
Aug 01 00:08:32.760939 ip-10-0-136-108 kubenswrapper[2578]: E0801 00:08:32.760920    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"22785f7879ac2c803891308d4de17ccfba4f9357f64c473751100f6c25e3c361\": container with ID starting with 22785f7879ac2c803891308d4de17ccfba4f9357f64c473751100f6c25e3c361 not found: ID does not exist" containerID="22785f7879ac2c803891308d4de17ccfba4f9357f64c473751100f6c25e3c361"
Aug 01 00:08:32.760996 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:08:32.760948    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"22785f7879ac2c803891308d4de17ccfba4f9357f64c473751100f6c25e3c361"} err="failed to get container status \"22785f7879ac2c803891308d4de17ccfba4f9357f64c473751100f6c25e3c361\": rpc error: code = NotFound desc = could not find container \"22785f7879ac2c803891308d4de17ccfba4f9357f64c473751100f6c25e3c361\": container with ID starting with 22785f7879ac2c803891308d4de17ccfba4f9357f64c473751100f6c25e3c361 not found: ID does not exist"
Aug 01 00:08:32.773369 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:08:32.773337    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-df362-67945678d8-nr7bj"]
Aug 01 00:08:32.776119 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:08:32.776091    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-df362-67945678d8-nr7bj"]
Aug 01 00:08:34.489955 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:08:34.489919    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="65018b1f-2bdb-4b63-846c-99cdcc75aef5" path="/var/lib/kubelet/pods/65018b1f-2bdb-4b63-846c-99cdcc75aef5/volumes"
Aug 01 00:08:41.626501 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:08:41.626407    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-d99eb-predictor-5bf7f45f47-77mxx" podUID="2da97b83-5b75-4fb7-912b-0e2dd8d1444d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.42:8080: connect: connection refused"
Aug 01 00:08:51.627097 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:08:51.627068    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-d99eb-predictor-5bf7f45f47-77mxx"
Aug 01 00:08:59.793289 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:08:59.793252    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gttdm_b32e6d42-e4e0-4113-b8ce-f5d5c5dc13f7/konnectivity-agent/0.log" limit=1024
Aug 01 00:09:02.576236 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:09:02.576199    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gttdm_b32e6d42-e4e0-4113-b8ce-f5d5c5dc13f7/konnectivity-agent/0.log" limit=1024
Aug 01 00:09:02.576616 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:09:02.576381    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/sequence-graph-d99eb-8bf569c97-62v27"]
Aug 01 00:09:02.576748 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:09:02.576735    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="65018b1f-2bdb-4b63-846c-99cdcc75aef5" containerName="sequence-graph-df362"
Aug 01 00:09:02.576790 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:09:02.576751    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="65018b1f-2bdb-4b63-846c-99cdcc75aef5" containerName="sequence-graph-df362"
Aug 01 00:09:02.576832 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:09:02.576823    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="65018b1f-2bdb-4b63-846c-99cdcc75aef5" containerName="sequence-graph-df362"
Aug 01 00:09:02.579925 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:09:02.579906    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sequence-graph-d99eb-8bf569c97-62v27"
Aug 01 00:09:02.581855 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:09:02.581828    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"sequence-graph-d99eb-kube-rbac-proxy-sar-config\""
Aug 01 00:09:02.581964 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:09:02.581830    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"sequence-graph-d99eb-serving-cert\""
Aug 01 00:09:02.587514 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:09:02.587495    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-d99eb-8bf569c97-62v27"]
Aug 01 00:09:02.708388 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:09:02.708341    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gttdm_b32e6d42-e4e0-4113-b8ce-f5d5c5dc13f7/konnectivity-agent/0.log" limit=1024
Aug 01 00:09:02.734525 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:09:02.734484    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fa70dcc9-b8a7-4b3c-a582-290cbec06a9a-openshift-service-ca-bundle\") pod \"sequence-graph-d99eb-8bf569c97-62v27\" (UID: \"fa70dcc9-b8a7-4b3c-a582-290cbec06a9a\") " pod="kserve-ci-e2e-test/sequence-graph-d99eb-8bf569c97-62v27"
Aug 01 00:09:02.734690 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:09:02.734537    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fa70dcc9-b8a7-4b3c-a582-290cbec06a9a-proxy-tls\") pod \"sequence-graph-d99eb-8bf569c97-62v27\" (UID: \"fa70dcc9-b8a7-4b3c-a582-290cbec06a9a\") " pod="kserve-ci-e2e-test/sequence-graph-d99eb-8bf569c97-62v27"
Aug 01 00:09:02.835480 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:09:02.835399    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fa70dcc9-b8a7-4b3c-a582-290cbec06a9a-proxy-tls\") pod \"sequence-graph-d99eb-8bf569c97-62v27\" (UID: \"fa70dcc9-b8a7-4b3c-a582-290cbec06a9a\") " pod="kserve-ci-e2e-test/sequence-graph-d99eb-8bf569c97-62v27"
Aug 01 00:09:02.835638 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:09:02.835498    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fa70dcc9-b8a7-4b3c-a582-290cbec06a9a-openshift-service-ca-bundle\") pod \"sequence-graph-d99eb-8bf569c97-62v27\" (UID: \"fa70dcc9-b8a7-4b3c-a582-290cbec06a9a\") " pod="kserve-ci-e2e-test/sequence-graph-d99eb-8bf569c97-62v27"
Aug 01 00:09:02.836058 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:09:02.836034    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fa70dcc9-b8a7-4b3c-a582-290cbec06a9a-openshift-service-ca-bundle\") pod \"sequence-graph-d99eb-8bf569c97-62v27\" (UID: \"fa70dcc9-b8a7-4b3c-a582-290cbec06a9a\") " pod="kserve-ci-e2e-test/sequence-graph-d99eb-8bf569c97-62v27"
Aug 01 00:09:02.837907 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:09:02.837888    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fa70dcc9-b8a7-4b3c-a582-290cbec06a9a-proxy-tls\") pod \"sequence-graph-d99eb-8bf569c97-62v27\" (UID: \"fa70dcc9-b8a7-4b3c-a582-290cbec06a9a\") " pod="kserve-ci-e2e-test/sequence-graph-d99eb-8bf569c97-62v27"
Aug 01 00:09:02.891066 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:09:02.891036    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sequence-graph-d99eb-8bf569c97-62v27"
Aug 01 00:09:03.035536 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:09:03.035499    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-d99eb-8bf569c97-62v27"]
Aug 01 00:09:03.038604 ip-10-0-136-108 kubenswrapper[2578]: W0801 00:09:03.038568    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfa70dcc9_b8a7_4b3c_a582_290cbec06a9a.slice/crio-c3f9ad04bb99cd83dc1e5122c142e9a513818b158f5dcf02fc635f71b634f113 WatchSource:0}: Error finding container c3f9ad04bb99cd83dc1e5122c142e9a513818b158f5dcf02fc635f71b634f113: Status 404 returned error can't find the container with id c3f9ad04bb99cd83dc1e5122c142e9a513818b158f5dcf02fc635f71b634f113
Aug 01 00:09:03.900646 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:09:03.900607    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sequence-graph-d99eb-8bf569c97-62v27" event={"ID":"fa70dcc9-b8a7-4b3c-a582-290cbec06a9a","Type":"ContainerStarted","Data":"4589c656a83d953a52229d055aae3b3adea404000ab6ec19d06ed243017d68f2"}
Aug 01 00:09:03.901013 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:09:03.900659    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sequence-graph-d99eb-8bf569c97-62v27" event={"ID":"fa70dcc9-b8a7-4b3c-a582-290cbec06a9a","Type":"ContainerStarted","Data":"c3f9ad04bb99cd83dc1e5122c142e9a513818b158f5dcf02fc635f71b634f113"}
Aug 01 00:09:03.901013 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:09:03.900688    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/sequence-graph-d99eb-8bf569c97-62v27"
Aug 01 00:09:03.919474 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:09:03.919426    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/sequence-graph-d99eb-8bf569c97-62v27" podStartSLOduration=1.919407989 podStartE2EDuration="1.919407989s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-01 00:09:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-01 00:09:03.918146068 +0000 UTC m=+1367.885896826" watchObservedRunningTime="2026-08-01 00:09:03.919407989 +0000 UTC m=+1367.887158748"
Aug 01 00:09:09.912000 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:09:09.911963    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/sequence-graph-d99eb-8bf569c97-62v27"
Aug 01 00:11:31.719682 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:11:31.719638    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gttdm_b32e6d42-e4e0-4113-b8ce-f5d5c5dc13f7/konnectivity-agent/0.log" limit=1024
Aug 01 00:12:40.088758 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:12:40.088716    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gttdm_b32e6d42-e4e0-4113-b8ce-f5d5c5dc13f7/konnectivity-agent/0.log" limit=1024
Aug 01 00:16:27.491993 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:16:27.491955    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/ensemble-graph-01c7a-7f54567bd4-96nwd"]
Aug 01 00:16:27.492537 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:16:27.492272    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/ensemble-graph-01c7a-7f54567bd4-96nwd" podUID="65eab692-528b-4c72-bc0b-4fdb2f5b4678" containerName="ensemble-graph-01c7a" containerID="cri-o://5cd959c5599107444c9c499a8263b32c8eaac0aac5fe224b05e8e9ee1b2a8ce8" gracePeriod=30
Aug 01 00:16:27.553624 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:16:27.553588    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gttdm_b32e6d42-e4e0-4113-b8ce-f5d5c5dc13f7/konnectivity-agent/0.log" limit=1024
Aug 01 00:16:27.640229 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:16:27.640200    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-01c7a-predictor-774957f645-hvd2w"]
Aug 01 00:16:27.640451 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:16:27.640423    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-01c7a-predictor-774957f645-hvd2w" podUID="b6eed2b1-cef9-457a-b42d-99effb74a9a4" containerName="kserve-container" containerID="cri-o://4920053b85982f0695f43f20352d96e21ee486f9311931a59836ff92e0338fbe" gracePeriod=30
Aug 01 00:16:27.640561 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:16:27.640450    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-01c7a-predictor-774957f645-hvd2w" podUID="b6eed2b1-cef9-457a-b42d-99effb74a9a4" containerName="kube-rbac-proxy" containerID="cri-o://363ec9505eccd6a0334afa9166e39a7a4fa9ab1902e290b111f75cfc04579a26" gracePeriod=30
Aug 01 00:16:27.702310 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:16:27.702275    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-d0074-predictor-5fc6bf94b7-fxc9g"]
Aug 01 00:16:27.705950 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:16:27.705933    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-d0074-predictor-5fc6bf94b7-fxc9g"
Aug 01 00:16:27.707872 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:16:27.707849    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-d0074-predictor-serving-cert\""
Aug 01 00:16:27.707986 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:16:27.707871    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-d0074-kube-rbac-proxy-sar-config\""
Aug 01 00:16:27.714719 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:16:27.714643    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-d0074-predictor-5fc6bf94b7-fxc9g"]
Aug 01 00:16:27.829786 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:16:27.829752    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fq76m\" (UniqueName: \"kubernetes.io/projected/9f543e2b-4d86-42f6-9859-dc6dab87f5b2-kube-api-access-fq76m\") pod \"success-200-isvc-d0074-predictor-5fc6bf94b7-fxc9g\" (UID: \"9f543e2b-4d86-42f6-9859-dc6dab87f5b2\") " pod="kserve-ci-e2e-test/success-200-isvc-d0074-predictor-5fc6bf94b7-fxc9g"
Aug 01 00:16:27.829786 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:16:27.829794    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9f543e2b-4d86-42f6-9859-dc6dab87f5b2-proxy-tls\") pod \"success-200-isvc-d0074-predictor-5fc6bf94b7-fxc9g\" (UID: \"9f543e2b-4d86-42f6-9859-dc6dab87f5b2\") " pod="kserve-ci-e2e-test/success-200-isvc-d0074-predictor-5fc6bf94b7-fxc9g"
Aug 01 00:16:27.830021 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:16:27.829831    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9f543e2b-4d86-42f6-9859-dc6dab87f5b2-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-d0074-predictor-5fc6bf94b7-fxc9g\" (UID: \"9f543e2b-4d86-42f6-9859-dc6dab87f5b2\") " pod="kserve-ci-e2e-test/success-200-isvc-d0074-predictor-5fc6bf94b7-fxc9g"
Aug 01 00:16:27.930913 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:16:27.930882    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9f543e2b-4d86-42f6-9859-dc6dab87f5b2-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-d0074-predictor-5fc6bf94b7-fxc9g\" (UID: \"9f543e2b-4d86-42f6-9859-dc6dab87f5b2\") " pod="kserve-ci-e2e-test/success-200-isvc-d0074-predictor-5fc6bf94b7-fxc9g"
Aug 01 00:16:27.931100 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:16:27.930968    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fq76m\" (UniqueName: \"kubernetes.io/projected/9f543e2b-4d86-42f6-9859-dc6dab87f5b2-kube-api-access-fq76m\") pod \"success-200-isvc-d0074-predictor-5fc6bf94b7-fxc9g\" (UID: \"9f543e2b-4d86-42f6-9859-dc6dab87f5b2\") " pod="kserve-ci-e2e-test/success-200-isvc-d0074-predictor-5fc6bf94b7-fxc9g"
Aug 01 00:16:27.931100 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:16:27.930998    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9f543e2b-4d86-42f6-9859-dc6dab87f5b2-proxy-tls\") pod \"success-200-isvc-d0074-predictor-5fc6bf94b7-fxc9g\" (UID: \"9f543e2b-4d86-42f6-9859-dc6dab87f5b2\") " pod="kserve-ci-e2e-test/success-200-isvc-d0074-predictor-5fc6bf94b7-fxc9g"
Aug 01 00:16:27.931644 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:16:27.931610    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9f543e2b-4d86-42f6-9859-dc6dab87f5b2-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-d0074-predictor-5fc6bf94b7-fxc9g\" (UID: \"9f543e2b-4d86-42f6-9859-dc6dab87f5b2\") " pod="kserve-ci-e2e-test/success-200-isvc-d0074-predictor-5fc6bf94b7-fxc9g"
Aug 01 00:16:27.933727 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:16:27.933701    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9f543e2b-4d86-42f6-9859-dc6dab87f5b2-proxy-tls\") pod \"success-200-isvc-d0074-predictor-5fc6bf94b7-fxc9g\" (UID: \"9f543e2b-4d86-42f6-9859-dc6dab87f5b2\") " pod="kserve-ci-e2e-test/success-200-isvc-d0074-predictor-5fc6bf94b7-fxc9g"
Aug 01 00:16:27.938613 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:16:27.938591    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fq76m\" (UniqueName: \"kubernetes.io/projected/9f543e2b-4d86-42f6-9859-dc6dab87f5b2-kube-api-access-fq76m\") pod \"success-200-isvc-d0074-predictor-5fc6bf94b7-fxc9g\" (UID: \"9f543e2b-4d86-42f6-9859-dc6dab87f5b2\") " pod="kserve-ci-e2e-test/success-200-isvc-d0074-predictor-5fc6bf94b7-fxc9g"
Aug 01 00:16:27.949153 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:16:27.949124    2578 generic.go:356] "Generic (PLEG): container finished" podID="b6eed2b1-cef9-457a-b42d-99effb74a9a4" containerID="363ec9505eccd6a0334afa9166e39a7a4fa9ab1902e290b111f75cfc04579a26" exitCode=2
Aug 01 00:16:27.949268 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:16:27.949192    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-01c7a-predictor-774957f645-hvd2w" event={"ID":"b6eed2b1-cef9-457a-b42d-99effb74a9a4","Type":"ContainerDied","Data":"363ec9505eccd6a0334afa9166e39a7a4fa9ab1902e290b111f75cfc04579a26"}
Aug 01 00:16:28.020289 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:16:28.020201    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-d0074-predictor-5fc6bf94b7-fxc9g"
Aug 01 00:16:28.153067 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:16:28.153042    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-d0074-predictor-5fc6bf94b7-fxc9g"]
Aug 01 00:16:28.155301 ip-10-0-136-108 kubenswrapper[2578]: W0801 00:16:28.155273    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9f543e2b_4d86_42f6_9859_dc6dab87f5b2.slice/crio-4f27fe938cb1428763a8c64a6665a452f2dde28d8ddddc07b645879c98923eba WatchSource:0}: Error finding container 4f27fe938cb1428763a8c64a6665a452f2dde28d8ddddc07b645879c98923eba: Status 404 returned error can't find the container with id 4f27fe938cb1428763a8c64a6665a452f2dde28d8ddddc07b645879c98923eba
Aug 01 00:16:28.563656 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:16:28.563614    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gttdm_b32e6d42-e4e0-4113-b8ce-f5d5c5dc13f7/konnectivity-agent/0.log" limit=1024
Aug 01 00:16:28.956103 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:16:28.956008    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-d0074-predictor-5fc6bf94b7-fxc9g" event={"ID":"9f543e2b-4d86-42f6-9859-dc6dab87f5b2","Type":"ContainerStarted","Data":"fe51d901790f32d56f2fb9efa1a220fdcb0137479b166b5ef865ec1e55b22f01"}
Aug 01 00:16:28.956279 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:16:28.956192    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-d0074-predictor-5fc6bf94b7-fxc9g"
Aug 01 00:16:28.956279 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:16:28.956230    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-d0074-predictor-5fc6bf94b7-fxc9g"
Aug 01 00:16:28.956279 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:16:28.956257    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-d0074-predictor-5fc6bf94b7-fxc9g" event={"ID":"9f543e2b-4d86-42f6-9859-dc6dab87f5b2","Type":"ContainerStarted","Data":"6a5c95b385d5d7a1e496b01533f936a74a51ed07d248af13e062fed1ea55b490"}
Aug 01 00:16:28.956401 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:16:28.956292    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-d0074-predictor-5fc6bf94b7-fxc9g" event={"ID":"9f543e2b-4d86-42f6-9859-dc6dab87f5b2","Type":"ContainerStarted","Data":"4f27fe938cb1428763a8c64a6665a452f2dde28d8ddddc07b645879c98923eba"}
Aug 01 00:16:28.957857 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:16:28.957834    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-d0074-predictor-5fc6bf94b7-fxc9g" podUID="9f543e2b-4d86-42f6-9859-dc6dab87f5b2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.45:8080: connect: connection refused"
Aug 01 00:16:28.972601 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:16:28.972552    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/success-200-isvc-d0074-predictor-5fc6bf94b7-fxc9g" podStartSLOduration=1.972537325 podStartE2EDuration="1.972537325s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-01 00:16:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-01 00:16:28.970106239 +0000 UTC m=+1812.937856997" watchObservedRunningTime="2026-08-01 00:16:28.972537325 +0000 UTC m=+1812.940288082"
Aug 01 00:16:29.961192 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:16:29.961130    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-d0074-predictor-5fc6bf94b7-fxc9g" podUID="9f543e2b-4d86-42f6-9859-dc6dab87f5b2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.45:8080: connect: connection refused"
Aug 01 00:16:30.357323 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:16:30.357274    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-01c7a-predictor-774957f645-hvd2w" podUID="b6eed2b1-cef9-457a-b42d-99effb74a9a4" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.40:8643/healthz\": dial tcp 10.133.0.40:8643: connect: connection refused"
Aug 01 00:16:30.361290 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:16:30.361260    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-01c7a-predictor-774957f645-hvd2w" podUID="b6eed2b1-cef9-457a-b42d-99effb74a9a4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.40:8080: connect: connection refused"
Aug 01 00:16:30.673262 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:16:30.673150    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/ensemble-graph-01c7a-7f54567bd4-96nwd" podUID="65eab692-528b-4c72-bc0b-4fdb2f5b4678" containerName="ensemble-graph-01c7a" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Aug 01 00:16:30.968400 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:16:30.968311    2578 generic.go:356] "Generic (PLEG): container finished" podID="b6eed2b1-cef9-457a-b42d-99effb74a9a4" containerID="4920053b85982f0695f43f20352d96e21ee486f9311931a59836ff92e0338fbe" exitCode=0
Aug 01 00:16:30.968400 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:16:30.968381    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-01c7a-predictor-774957f645-hvd2w" event={"ID":"b6eed2b1-cef9-457a-b42d-99effb74a9a4","Type":"ContainerDied","Data":"4920053b85982f0695f43f20352d96e21ee486f9311931a59836ff92e0338fbe"}
Aug 01 00:16:31.003193 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:16:31.003147    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-01c7a-predictor-774957f645-hvd2w"
Aug 01 00:16:31.160971 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:16:31.160933    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b6eed2b1-cef9-457a-b42d-99effb74a9a4-proxy-tls\") pod \"b6eed2b1-cef9-457a-b42d-99effb74a9a4\" (UID: \"b6eed2b1-cef9-457a-b42d-99effb74a9a4\") "
Aug 01 00:16:31.160971 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:16:31.160972    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hwc8f\" (UniqueName: \"kubernetes.io/projected/b6eed2b1-cef9-457a-b42d-99effb74a9a4-kube-api-access-hwc8f\") pod \"b6eed2b1-cef9-457a-b42d-99effb74a9a4\" (UID: \"b6eed2b1-cef9-457a-b42d-99effb74a9a4\") "
Aug 01 00:16:31.161270 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:16:31.161005    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b6eed2b1-cef9-457a-b42d-99effb74a9a4-kube-rbac-proxy-sar-config\") pod \"b6eed2b1-cef9-457a-b42d-99effb74a9a4\" (UID: \"b6eed2b1-cef9-457a-b42d-99effb74a9a4\") "
Aug 01 00:16:31.161405 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:16:31.161381    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b6eed2b1-cef9-457a-b42d-99effb74a9a4-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "b6eed2b1-cef9-457a-b42d-99effb74a9a4" (UID: "b6eed2b1-cef9-457a-b42d-99effb74a9a4"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 01 00:16:31.163188 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:16:31.163141    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b6eed2b1-cef9-457a-b42d-99effb74a9a4-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "b6eed2b1-cef9-457a-b42d-99effb74a9a4" (UID: "b6eed2b1-cef9-457a-b42d-99effb74a9a4"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 01 00:16:31.163301 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:16:31.163213    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b6eed2b1-cef9-457a-b42d-99effb74a9a4-kube-api-access-hwc8f" (OuterVolumeSpecName: "kube-api-access-hwc8f") pod "b6eed2b1-cef9-457a-b42d-99effb74a9a4" (UID: "b6eed2b1-cef9-457a-b42d-99effb74a9a4"). InnerVolumeSpecName "kube-api-access-hwc8f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 01 00:16:31.261784 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:16:31.261748    2578 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b6eed2b1-cef9-457a-b42d-99effb74a9a4-proxy-tls\") on node \"ip-10-0-136-108.ec2.internal\" DevicePath \"\""
Aug 01 00:16:31.261784 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:16:31.261783    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hwc8f\" (UniqueName: \"kubernetes.io/projected/b6eed2b1-cef9-457a-b42d-99effb74a9a4-kube-api-access-hwc8f\") on node \"ip-10-0-136-108.ec2.internal\" DevicePath \"\""
Aug 01 00:16:31.261993 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:16:31.261795    2578 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b6eed2b1-cef9-457a-b42d-99effb74a9a4-kube-rbac-proxy-sar-config\") on node \"ip-10-0-136-108.ec2.internal\" DevicePath \"\""
Aug 01 00:16:31.975082 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:16:31.975046    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-01c7a-predictor-774957f645-hvd2w"
Aug 01 00:16:31.975082 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:16:31.975052    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-01c7a-predictor-774957f645-hvd2w" event={"ID":"b6eed2b1-cef9-457a-b42d-99effb74a9a4","Type":"ContainerDied","Data":"f7b73f50788d6da970f1f72f78620c2ec2b1df08c94809c070ea18280ef40183"}
Aug 01 00:16:31.975642 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:16:31.975118    2578 scope.go:117] "RemoveContainer" containerID="363ec9505eccd6a0334afa9166e39a7a4fa9ab1902e290b111f75cfc04579a26"
Aug 01 00:16:31.983815 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:16:31.983795    2578 scope.go:117] "RemoveContainer" containerID="4920053b85982f0695f43f20352d96e21ee486f9311931a59836ff92e0338fbe"
Aug 01 00:16:31.995966 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:16:31.995942    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-01c7a-predictor-774957f645-hvd2w"]
Aug 01 00:16:32.002070 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:16:32.002049    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-01c7a-predictor-774957f645-hvd2w"]
Aug 01 00:16:32.490987 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:16:32.490950    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b6eed2b1-cef9-457a-b42d-99effb74a9a4" path="/var/lib/kubelet/pods/b6eed2b1-cef9-457a-b42d-99effb74a9a4/volumes"
Aug 01 00:16:34.966540 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:16:34.966512    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-d0074-predictor-5fc6bf94b7-fxc9g"
Aug 01 00:16:34.966979 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:16:34.966952    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-d0074-predictor-5fc6bf94b7-fxc9g" podUID="9f543e2b-4d86-42f6-9859-dc6dab87f5b2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.45:8080: connect: connection refused"
Aug 01 00:16:35.673093 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:16:35.673052    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/ensemble-graph-01c7a-7f54567bd4-96nwd" podUID="65eab692-528b-4c72-bc0b-4fdb2f5b4678" containerName="ensemble-graph-01c7a" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Aug 01 00:16:40.674226 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:16:40.674165    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/ensemble-graph-01c7a-7f54567bd4-96nwd" podUID="65eab692-528b-4c72-bc0b-4fdb2f5b4678" containerName="ensemble-graph-01c7a" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Aug 01 00:16:40.674647 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:16:40.674300    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/ensemble-graph-01c7a-7f54567bd4-96nwd"
Aug 01 00:16:44.967791 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:16:44.967746    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-d0074-predictor-5fc6bf94b7-fxc9g" podUID="9f543e2b-4d86-42f6-9859-dc6dab87f5b2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.45:8080: connect: connection refused"
Aug 01 00:16:45.672940 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:16:45.672898    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/ensemble-graph-01c7a-7f54567bd4-96nwd" podUID="65eab692-528b-4c72-bc0b-4fdb2f5b4678" containerName="ensemble-graph-01c7a" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Aug 01 00:16:50.673544 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:16:50.673500    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/ensemble-graph-01c7a-7f54567bd4-96nwd" podUID="65eab692-528b-4c72-bc0b-4fdb2f5b4678" containerName="ensemble-graph-01c7a" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Aug 01 00:16:54.967226 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:16:54.967167    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-d0074-predictor-5fc6bf94b7-fxc9g" podUID="9f543e2b-4d86-42f6-9859-dc6dab87f5b2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.45:8080: connect: connection refused"
Aug 01 00:16:55.673482 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:16:55.673443    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/ensemble-graph-01c7a-7f54567bd4-96nwd" podUID="65eab692-528b-4c72-bc0b-4fdb2f5b4678" containerName="ensemble-graph-01c7a" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Aug 01 00:16:57.632700 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:16:57.632672    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/ensemble-graph-01c7a-7f54567bd4-96nwd"
Aug 01 00:16:57.679818 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:16:57.679786    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/65eab692-528b-4c72-bc0b-4fdb2f5b4678-openshift-service-ca-bundle\") pod \"65eab692-528b-4c72-bc0b-4fdb2f5b4678\" (UID: \"65eab692-528b-4c72-bc0b-4fdb2f5b4678\") "
Aug 01 00:16:57.679975 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:16:57.679859    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/65eab692-528b-4c72-bc0b-4fdb2f5b4678-proxy-tls\") pod \"65eab692-528b-4c72-bc0b-4fdb2f5b4678\" (UID: \"65eab692-528b-4c72-bc0b-4fdb2f5b4678\") "
Aug 01 00:16:57.680187 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:16:57.680151    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/65eab692-528b-4c72-bc0b-4fdb2f5b4678-openshift-service-ca-bundle" (OuterVolumeSpecName: "openshift-service-ca-bundle") pod "65eab692-528b-4c72-bc0b-4fdb2f5b4678" (UID: "65eab692-528b-4c72-bc0b-4fdb2f5b4678"). InnerVolumeSpecName "openshift-service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 01 00:16:57.682024 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:16:57.682002    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/65eab692-528b-4c72-bc0b-4fdb2f5b4678-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "65eab692-528b-4c72-bc0b-4fdb2f5b4678" (UID: "65eab692-528b-4c72-bc0b-4fdb2f5b4678"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 01 00:16:57.781047 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:16:57.780956    2578 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/65eab692-528b-4c72-bc0b-4fdb2f5b4678-proxy-tls\") on node \"ip-10-0-136-108.ec2.internal\" DevicePath \"\""
Aug 01 00:16:57.781047 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:16:57.780986    2578 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/65eab692-528b-4c72-bc0b-4fdb2f5b4678-openshift-service-ca-bundle\") on node \"ip-10-0-136-108.ec2.internal\" DevicePath \"\""
Aug 01 00:16:58.103577 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:16:58.103491    2578 generic.go:356] "Generic (PLEG): container finished" podID="65eab692-528b-4c72-bc0b-4fdb2f5b4678" containerID="5cd959c5599107444c9c499a8263b32c8eaac0aac5fe224b05e8e9ee1b2a8ce8" exitCode=0
Aug 01 00:16:58.103577 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:16:58.103538    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/ensemble-graph-01c7a-7f54567bd4-96nwd"
Aug 01 00:16:58.103577 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:16:58.103552    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/ensemble-graph-01c7a-7f54567bd4-96nwd" event={"ID":"65eab692-528b-4c72-bc0b-4fdb2f5b4678","Type":"ContainerDied","Data":"5cd959c5599107444c9c499a8263b32c8eaac0aac5fe224b05e8e9ee1b2a8ce8"}
Aug 01 00:16:58.103792 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:16:58.103619    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/ensemble-graph-01c7a-7f54567bd4-96nwd" event={"ID":"65eab692-528b-4c72-bc0b-4fdb2f5b4678","Type":"ContainerDied","Data":"3ba013fdac2eb5e509f14a5d437adbfdfcd627b9c04fdfa29a45dc8e0747ed04"}
Aug 01 00:16:58.103792 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:16:58.103643    2578 scope.go:117] "RemoveContainer" containerID="5cd959c5599107444c9c499a8263b32c8eaac0aac5fe224b05e8e9ee1b2a8ce8"
Aug 01 00:16:58.112781 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:16:58.112764    2578 scope.go:117] "RemoveContainer" containerID="5cd959c5599107444c9c499a8263b32c8eaac0aac5fe224b05e8e9ee1b2a8ce8"
Aug 01 00:16:58.113020 ip-10-0-136-108 kubenswrapper[2578]: E0801 00:16:58.113001    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5cd959c5599107444c9c499a8263b32c8eaac0aac5fe224b05e8e9ee1b2a8ce8\": container with ID starting with 5cd959c5599107444c9c499a8263b32c8eaac0aac5fe224b05e8e9ee1b2a8ce8 not found: ID does not exist" containerID="5cd959c5599107444c9c499a8263b32c8eaac0aac5fe224b05e8e9ee1b2a8ce8"
Aug 01 00:16:58.113085 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:16:58.113029    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5cd959c5599107444c9c499a8263b32c8eaac0aac5fe224b05e8e9ee1b2a8ce8"} err="failed to get container status \"5cd959c5599107444c9c499a8263b32c8eaac0aac5fe224b05e8e9ee1b2a8ce8\": rpc error: code = NotFound desc = could not find container \"5cd959c5599107444c9c499a8263b32c8eaac0aac5fe224b05e8e9ee1b2a8ce8\": container with ID starting with 5cd959c5599107444c9c499a8263b32c8eaac0aac5fe224b05e8e9ee1b2a8ce8 not found: ID does not exist"
Aug 01 00:16:58.124346 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:16:58.124323    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/ensemble-graph-01c7a-7f54567bd4-96nwd"]
Aug 01 00:16:58.130400 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:16:58.130378    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/ensemble-graph-01c7a-7f54567bd4-96nwd"]
Aug 01 00:16:58.489416 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:16:58.489383    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="65eab692-528b-4c72-bc0b-4fdb2f5b4678" path="/var/lib/kubelet/pods/65eab692-528b-4c72-bc0b-4fdb2f5b4678/volumes"
Aug 01 00:17:04.968001 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:17:04.967964    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-d0074-predictor-5fc6bf94b7-fxc9g" podUID="9f543e2b-4d86-42f6-9859-dc6dab87f5b2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.45:8080: connect: connection refused"
Aug 01 00:17:14.968135 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:17:14.968100    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-d0074-predictor-5fc6bf94b7-fxc9g"
Aug 01 00:17:17.259378 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:17:17.259342    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-d99eb-8bf569c97-62v27"]
Aug 01 00:17:17.259904 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:17:17.259564    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/sequence-graph-d99eb-8bf569c97-62v27" podUID="fa70dcc9-b8a7-4b3c-a582-290cbec06a9a" containerName="sequence-graph-d99eb" containerID="cri-o://4589c656a83d953a52229d055aae3b3adea404000ab6ec19d06ed243017d68f2" gracePeriod=30
Aug 01 00:17:17.283396 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:17:17.283349    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gttdm_b32e6d42-e4e0-4113-b8ce-f5d5c5dc13f7/konnectivity-agent/0.log" limit=1024
Aug 01 00:17:17.369515 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:17:17.369483    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-d99eb-predictor-5bf7f45f47-77mxx"]
Aug 01 00:17:17.370252 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:17:17.370030    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-d99eb-predictor-5bf7f45f47-77mxx" podUID="2da97b83-5b75-4fb7-912b-0e2dd8d1444d" containerName="kube-rbac-proxy" containerID="cri-o://f284b0f19771ebdc7b42558a59c4771b4ccb1ce292a8ae290d4e7292e535aeb4" gracePeriod=30
Aug 01 00:17:17.370252 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:17:17.370115    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-d99eb-predictor-5bf7f45f47-77mxx" podUID="2da97b83-5b75-4fb7-912b-0e2dd8d1444d" containerName="kserve-container" containerID="cri-o://6e2e0b1a0abe31e3f6866b2990daca0f8b45bb58eaf8cfaaf7d5a3ad01687882" gracePeriod=30
Aug 01 00:17:17.391423 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:17:17.391373    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gttdm_b32e6d42-e4e0-4113-b8ce-f5d5c5dc13f7/konnectivity-agent/0.log" limit=1024
Aug 01 00:17:17.457445 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:17:17.457412    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-bb639-predictor-7fccb6f469-zdmj7"]
Aug 01 00:17:17.457830 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:17:17.457816    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b6eed2b1-cef9-457a-b42d-99effb74a9a4" containerName="kserve-container"
Aug 01 00:17:17.457830 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:17:17.457831    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="b6eed2b1-cef9-457a-b42d-99effb74a9a4" containerName="kserve-container"
Aug 01 00:17:17.457916 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:17:17.457845    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="65eab692-528b-4c72-bc0b-4fdb2f5b4678" containerName="ensemble-graph-01c7a"
Aug 01 00:17:17.457916 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:17:17.457851    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="65eab692-528b-4c72-bc0b-4fdb2f5b4678" containerName="ensemble-graph-01c7a"
Aug 01 00:17:17.457916 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:17:17.457868    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b6eed2b1-cef9-457a-b42d-99effb74a9a4" containerName="kube-rbac-proxy"
Aug 01 00:17:17.457916 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:17:17.457874    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="b6eed2b1-cef9-457a-b42d-99effb74a9a4" containerName="kube-rbac-proxy"
Aug 01 00:17:17.458092 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:17:17.457923    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="65eab692-528b-4c72-bc0b-4fdb2f5b4678" containerName="ensemble-graph-01c7a"
Aug 01 00:17:17.458092 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:17:17.457935    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="b6eed2b1-cef9-457a-b42d-99effb74a9a4" containerName="kube-rbac-proxy"
Aug 01 00:17:17.458092 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:17:17.457942    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="b6eed2b1-cef9-457a-b42d-99effb74a9a4" containerName="kserve-container"
Aug 01 00:17:17.462835 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:17:17.462813    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-bb639-predictor-7fccb6f469-zdmj7"
Aug 01 00:17:17.464809 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:17:17.464782    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-bb639-predictor-serving-cert\""
Aug 01 00:17:17.464943 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:17:17.464835    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-bb639-kube-rbac-proxy-sar-config\""
Aug 01 00:17:17.470321 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:17:17.470292    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-bb639-predictor-7fccb6f469-zdmj7"]
Aug 01 00:17:17.474730 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:17:17.474697    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gttdm_b32e6d42-e4e0-4113-b8ce-f5d5c5dc13f7/konnectivity-agent/0.log" limit=1024
Aug 01 00:17:17.545783 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:17:17.545702    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gttdm_b32e6d42-e4e0-4113-b8ce-f5d5c5dc13f7/konnectivity-agent/0.log" limit=1024
Aug 01 00:17:17.549540 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:17:17.549511    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c3f7ef2d-1ce3-4f69-9660-f559ac064ea7-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-bb639-predictor-7fccb6f469-zdmj7\" (UID: \"c3f7ef2d-1ce3-4f69-9660-f559ac064ea7\") " pod="kserve-ci-e2e-test/success-200-isvc-bb639-predictor-7fccb6f469-zdmj7"
Aug 01 00:17:17.549688 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:17:17.549561    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c3f7ef2d-1ce3-4f69-9660-f559ac064ea7-proxy-tls\") pod \"success-200-isvc-bb639-predictor-7fccb6f469-zdmj7\" (UID: \"c3f7ef2d-1ce3-4f69-9660-f559ac064ea7\") " pod="kserve-ci-e2e-test/success-200-isvc-bb639-predictor-7fccb6f469-zdmj7"
Aug 01 00:17:17.549741 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:17:17.549687    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7hv5q\" (UniqueName: \"kubernetes.io/projected/c3f7ef2d-1ce3-4f69-9660-f559ac064ea7-kube-api-access-7hv5q\") pod \"success-200-isvc-bb639-predictor-7fccb6f469-zdmj7\" (UID: \"c3f7ef2d-1ce3-4f69-9660-f559ac064ea7\") " pod="kserve-ci-e2e-test/success-200-isvc-bb639-predictor-7fccb6f469-zdmj7"
Aug 01 00:17:17.650548 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:17:17.650514    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7hv5q\" (UniqueName: \"kubernetes.io/projected/c3f7ef2d-1ce3-4f69-9660-f559ac064ea7-kube-api-access-7hv5q\") pod \"success-200-isvc-bb639-predictor-7fccb6f469-zdmj7\" (UID: \"c3f7ef2d-1ce3-4f69-9660-f559ac064ea7\") " pod="kserve-ci-e2e-test/success-200-isvc-bb639-predictor-7fccb6f469-zdmj7"
Aug 01 00:17:17.650548 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:17:17.650552    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c3f7ef2d-1ce3-4f69-9660-f559ac064ea7-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-bb639-predictor-7fccb6f469-zdmj7\" (UID: \"c3f7ef2d-1ce3-4f69-9660-f559ac064ea7\") " pod="kserve-ci-e2e-test/success-200-isvc-bb639-predictor-7fccb6f469-zdmj7"
Aug 01 00:17:17.650813 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:17:17.650591    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c3f7ef2d-1ce3-4f69-9660-f559ac064ea7-proxy-tls\") pod \"success-200-isvc-bb639-predictor-7fccb6f469-zdmj7\" (UID: \"c3f7ef2d-1ce3-4f69-9660-f559ac064ea7\") " pod="kserve-ci-e2e-test/success-200-isvc-bb639-predictor-7fccb6f469-zdmj7"
Aug 01 00:17:17.650877 ip-10-0-136-108 kubenswrapper[2578]: E0801 00:17:17.650838    2578 secret.go:189] Couldn't get secret kserve-ci-e2e-test/success-200-isvc-bb639-predictor-serving-cert: secret "success-200-isvc-bb639-predictor-serving-cert" not found
Aug 01 00:17:17.650939 ip-10-0-136-108 kubenswrapper[2578]: E0801 00:17:17.650920    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c3f7ef2d-1ce3-4f69-9660-f559ac064ea7-proxy-tls podName:c3f7ef2d-1ce3-4f69-9660-f559ac064ea7 nodeName:}" failed. No retries permitted until 2026-08-01 00:17:18.150901499 +0000 UTC m=+1862.118652246 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/c3f7ef2d-1ce3-4f69-9660-f559ac064ea7-proxy-tls") pod "success-200-isvc-bb639-predictor-7fccb6f469-zdmj7" (UID: "c3f7ef2d-1ce3-4f69-9660-f559ac064ea7") : secret "success-200-isvc-bb639-predictor-serving-cert" not found
Aug 01 00:17:17.651547 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:17:17.651528    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c3f7ef2d-1ce3-4f69-9660-f559ac064ea7-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-bb639-predictor-7fccb6f469-zdmj7\" (UID: \"c3f7ef2d-1ce3-4f69-9660-f559ac064ea7\") " pod="kserve-ci-e2e-test/success-200-isvc-bb639-predictor-7fccb6f469-zdmj7"
Aug 01 00:17:17.660945 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:17:17.660920    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7hv5q\" (UniqueName: \"kubernetes.io/projected/c3f7ef2d-1ce3-4f69-9660-f559ac064ea7-kube-api-access-7hv5q\") pod \"success-200-isvc-bb639-predictor-7fccb6f469-zdmj7\" (UID: \"c3f7ef2d-1ce3-4f69-9660-f559ac064ea7\") " pod="kserve-ci-e2e-test/success-200-isvc-bb639-predictor-7fccb6f469-zdmj7"
Aug 01 00:17:17.704093 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:17:17.704059    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gttdm_b32e6d42-e4e0-4113-b8ce-f5d5c5dc13f7/konnectivity-agent/0.log" limit=1024
Aug 01 00:17:17.859512 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:17:17.859428    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gttdm_b32e6d42-e4e0-4113-b8ce-f5d5c5dc13f7/konnectivity-agent/0.log" limit=1024
Aug 01 00:17:18.154629 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:17:18.154539    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c3f7ef2d-1ce3-4f69-9660-f559ac064ea7-proxy-tls\") pod \"success-200-isvc-bb639-predictor-7fccb6f469-zdmj7\" (UID: \"c3f7ef2d-1ce3-4f69-9660-f559ac064ea7\") " pod="kserve-ci-e2e-test/success-200-isvc-bb639-predictor-7fccb6f469-zdmj7"
Aug 01 00:17:18.157080 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:17:18.157046    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c3f7ef2d-1ce3-4f69-9660-f559ac064ea7-proxy-tls\") pod \"success-200-isvc-bb639-predictor-7fccb6f469-zdmj7\" (UID: \"c3f7ef2d-1ce3-4f69-9660-f559ac064ea7\") " pod="kserve-ci-e2e-test/success-200-isvc-bb639-predictor-7fccb6f469-zdmj7"
Aug 01 00:17:18.201288 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:17:18.201253    2578 generic.go:356] "Generic (PLEG): container finished" podID="2da97b83-5b75-4fb7-912b-0e2dd8d1444d" containerID="f284b0f19771ebdc7b42558a59c4771b4ccb1ce292a8ae290d4e7292e535aeb4" exitCode=2
Aug 01 00:17:18.201465 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:17:18.201332    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-d99eb-predictor-5bf7f45f47-77mxx" event={"ID":"2da97b83-5b75-4fb7-912b-0e2dd8d1444d","Type":"ContainerDied","Data":"f284b0f19771ebdc7b42558a59c4771b4ccb1ce292a8ae290d4e7292e535aeb4"}
Aug 01 00:17:18.374691 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:17:18.374650    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-bb639-predictor-7fccb6f469-zdmj7"
Aug 01 00:17:18.509769 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:17:18.509743    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-bb639-predictor-7fccb6f469-zdmj7"]
Aug 01 00:17:18.511786 ip-10-0-136-108 kubenswrapper[2578]: W0801 00:17:18.511763    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc3f7ef2d_1ce3_4f69_9660_f559ac064ea7.slice/crio-1ace02dd0d68ef2b73fbae6c4c99525578a6eb54fa3bca8bd08517a46cf54e0d WatchSource:0}: Error finding container 1ace02dd0d68ef2b73fbae6c4c99525578a6eb54fa3bca8bd08517a46cf54e0d: Status 404 returned error can't find the container with id 1ace02dd0d68ef2b73fbae6c4c99525578a6eb54fa3bca8bd08517a46cf54e0d
Aug 01 00:17:19.208438 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:17:19.208402    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-bb639-predictor-7fccb6f469-zdmj7" event={"ID":"c3f7ef2d-1ce3-4f69-9660-f559ac064ea7","Type":"ContainerStarted","Data":"49679d1de7ea7d50e52c0f1105469f895bf6c6bdc2b9aaf8701ee0cdf24af147"}
Aug 01 00:17:19.208620 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:17:19.208467    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-bb639-predictor-7fccb6f469-zdmj7" event={"ID":"c3f7ef2d-1ce3-4f69-9660-f559ac064ea7","Type":"ContainerStarted","Data":"944cfc39854c8c09a42a0269e084482da2c2390c72882af7cce2c9beb0810971"}
Aug 01 00:17:19.208620 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:17:19.208500    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-bb639-predictor-7fccb6f469-zdmj7"
Aug 01 00:17:19.208620 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:17:19.208522    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-bb639-predictor-7fccb6f469-zdmj7" event={"ID":"c3f7ef2d-1ce3-4f69-9660-f559ac064ea7","Type":"ContainerStarted","Data":"1ace02dd0d68ef2b73fbae6c4c99525578a6eb54fa3bca8bd08517a46cf54e0d"}
Aug 01 00:17:19.226352 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:17:19.226265    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/success-200-isvc-bb639-predictor-7fccb6f469-zdmj7" podStartSLOduration=2.226249325 podStartE2EDuration="2.226249325s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-01 00:17:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-01 00:17:19.22551106 +0000 UTC m=+1863.193261837" watchObservedRunningTime="2026-08-01 00:17:19.226249325 +0000 UTC m=+1863.194000083"
Aug 01 00:17:19.908722 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:17:19.908674    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-d99eb-8bf569c97-62v27" podUID="fa70dcc9-b8a7-4b3c-a582-290cbec06a9a" containerName="sequence-graph-d99eb" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Aug 01 00:17:20.213572 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:17:20.213480    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-bb639-predictor-7fccb6f469-zdmj7"
Aug 01 00:17:20.214977 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:17:20.214952    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-bb639-predictor-7fccb6f469-zdmj7" podUID="c3f7ef2d-1ce3-4f69-9660-f559ac064ea7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.46:8080: connect: connection refused"
Aug 01 00:17:20.622563 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:17:20.622534    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-d99eb-predictor-5bf7f45f47-77mxx"
Aug 01 00:17:20.776528 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:17:20.776492    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2da97b83-5b75-4fb7-912b-0e2dd8d1444d-kube-rbac-proxy-sar-config\") pod \"2da97b83-5b75-4fb7-912b-0e2dd8d1444d\" (UID: \"2da97b83-5b75-4fb7-912b-0e2dd8d1444d\") "
Aug 01 00:17:20.776721 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:17:20.776540    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bf9g8\" (UniqueName: \"kubernetes.io/projected/2da97b83-5b75-4fb7-912b-0e2dd8d1444d-kube-api-access-bf9g8\") pod \"2da97b83-5b75-4fb7-912b-0e2dd8d1444d\" (UID: \"2da97b83-5b75-4fb7-912b-0e2dd8d1444d\") "
Aug 01 00:17:20.776721 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:17:20.776570    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2da97b83-5b75-4fb7-912b-0e2dd8d1444d-proxy-tls\") pod \"2da97b83-5b75-4fb7-912b-0e2dd8d1444d\" (UID: \"2da97b83-5b75-4fb7-912b-0e2dd8d1444d\") "
Aug 01 00:17:20.776904 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:17:20.776874    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2da97b83-5b75-4fb7-912b-0e2dd8d1444d-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "2da97b83-5b75-4fb7-912b-0e2dd8d1444d" (UID: "2da97b83-5b75-4fb7-912b-0e2dd8d1444d"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 01 00:17:20.778718 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:17:20.778694    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2da97b83-5b75-4fb7-912b-0e2dd8d1444d-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "2da97b83-5b75-4fb7-912b-0e2dd8d1444d" (UID: "2da97b83-5b75-4fb7-912b-0e2dd8d1444d"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 01 00:17:20.778785 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:17:20.778720    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2da97b83-5b75-4fb7-912b-0e2dd8d1444d-kube-api-access-bf9g8" (OuterVolumeSpecName: "kube-api-access-bf9g8") pod "2da97b83-5b75-4fb7-912b-0e2dd8d1444d" (UID: "2da97b83-5b75-4fb7-912b-0e2dd8d1444d"). InnerVolumeSpecName "kube-api-access-bf9g8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 01 00:17:20.877570 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:17:20.877528    2578 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2da97b83-5b75-4fb7-912b-0e2dd8d1444d-kube-rbac-proxy-sar-config\") on node \"ip-10-0-136-108.ec2.internal\" DevicePath \"\""
Aug 01 00:17:20.877570 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:17:20.877562    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bf9g8\" (UniqueName: \"kubernetes.io/projected/2da97b83-5b75-4fb7-912b-0e2dd8d1444d-kube-api-access-bf9g8\") on node \"ip-10-0-136-108.ec2.internal\" DevicePath \"\""
Aug 01 00:17:20.877570 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:17:20.877577    2578 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2da97b83-5b75-4fb7-912b-0e2dd8d1444d-proxy-tls\") on node \"ip-10-0-136-108.ec2.internal\" DevicePath \"\""
Aug 01 00:17:21.219145 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:17:21.219058    2578 generic.go:356] "Generic (PLEG): container finished" podID="2da97b83-5b75-4fb7-912b-0e2dd8d1444d" containerID="6e2e0b1a0abe31e3f6866b2990daca0f8b45bb58eaf8cfaaf7d5a3ad01687882" exitCode=0
Aug 01 00:17:21.219145 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:17:21.219112    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-d99eb-predictor-5bf7f45f47-77mxx"
Aug 01 00:17:21.219145 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:17:21.219133    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-d99eb-predictor-5bf7f45f47-77mxx" event={"ID":"2da97b83-5b75-4fb7-912b-0e2dd8d1444d","Type":"ContainerDied","Data":"6e2e0b1a0abe31e3f6866b2990daca0f8b45bb58eaf8cfaaf7d5a3ad01687882"}
Aug 01 00:17:21.219705 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:17:21.219211    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-d99eb-predictor-5bf7f45f47-77mxx" event={"ID":"2da97b83-5b75-4fb7-912b-0e2dd8d1444d","Type":"ContainerDied","Data":"8f7f876364b6f7abafd91d6616a182c4ea473e8591481147f6e5dfab4e370595"}
Aug 01 00:17:21.219705 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:17:21.219239    2578 scope.go:117] "RemoveContainer" containerID="f284b0f19771ebdc7b42558a59c4771b4ccb1ce292a8ae290d4e7292e535aeb4"
Aug 01 00:17:21.219840 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:17:21.219820    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-bb639-predictor-7fccb6f469-zdmj7" podUID="c3f7ef2d-1ce3-4f69-9660-f559ac064ea7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.46:8080: connect: connection refused"
Aug 01 00:17:21.229027 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:17:21.229009    2578 scope.go:117] "RemoveContainer" containerID="6e2e0b1a0abe31e3f6866b2990daca0f8b45bb58eaf8cfaaf7d5a3ad01687882"
Aug 01 00:17:21.236936 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:17:21.236916    2578 scope.go:117] "RemoveContainer" containerID="f284b0f19771ebdc7b42558a59c4771b4ccb1ce292a8ae290d4e7292e535aeb4"
Aug 01 00:17:21.237239 ip-10-0-136-108 kubenswrapper[2578]: E0801 00:17:21.237216    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f284b0f19771ebdc7b42558a59c4771b4ccb1ce292a8ae290d4e7292e535aeb4\": container with ID starting with f284b0f19771ebdc7b42558a59c4771b4ccb1ce292a8ae290d4e7292e535aeb4 not found: ID does not exist" containerID="f284b0f19771ebdc7b42558a59c4771b4ccb1ce292a8ae290d4e7292e535aeb4"
Aug 01 00:17:21.237294 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:17:21.237254    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f284b0f19771ebdc7b42558a59c4771b4ccb1ce292a8ae290d4e7292e535aeb4"} err="failed to get container status \"f284b0f19771ebdc7b42558a59c4771b4ccb1ce292a8ae290d4e7292e535aeb4\": rpc error: code = NotFound desc = could not find container \"f284b0f19771ebdc7b42558a59c4771b4ccb1ce292a8ae290d4e7292e535aeb4\": container with ID starting with f284b0f19771ebdc7b42558a59c4771b4ccb1ce292a8ae290d4e7292e535aeb4 not found: ID does not exist"
Aug 01 00:17:21.237294 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:17:21.237279    2578 scope.go:117] "RemoveContainer" containerID="6e2e0b1a0abe31e3f6866b2990daca0f8b45bb58eaf8cfaaf7d5a3ad01687882"
Aug 01 00:17:21.237542 ip-10-0-136-108 kubenswrapper[2578]: E0801 00:17:21.237524    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6e2e0b1a0abe31e3f6866b2990daca0f8b45bb58eaf8cfaaf7d5a3ad01687882\": container with ID starting with 6e2e0b1a0abe31e3f6866b2990daca0f8b45bb58eaf8cfaaf7d5a3ad01687882 not found: ID does not exist" containerID="6e2e0b1a0abe31e3f6866b2990daca0f8b45bb58eaf8cfaaf7d5a3ad01687882"
Aug 01 00:17:21.237583 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:17:21.237548    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6e2e0b1a0abe31e3f6866b2990daca0f8b45bb58eaf8cfaaf7d5a3ad01687882"} err="failed to get container status \"6e2e0b1a0abe31e3f6866b2990daca0f8b45bb58eaf8cfaaf7d5a3ad01687882\": rpc error: code = NotFound desc = could not find container \"6e2e0b1a0abe31e3f6866b2990daca0f8b45bb58eaf8cfaaf7d5a3ad01687882\": container with ID starting with 6e2e0b1a0abe31e3f6866b2990daca0f8b45bb58eaf8cfaaf7d5a3ad01687882 not found: ID does not exist"
Aug 01 00:17:21.240498 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:17:21.240472    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-d99eb-predictor-5bf7f45f47-77mxx"]
Aug 01 00:17:21.244371 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:17:21.244349    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-d99eb-predictor-5bf7f45f47-77mxx"]
Aug 01 00:17:22.489672 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:17:22.489640    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2da97b83-5b75-4fb7-912b-0e2dd8d1444d" path="/var/lib/kubelet/pods/2da97b83-5b75-4fb7-912b-0e2dd8d1444d/volumes"
Aug 01 00:17:24.909293 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:17:24.909254    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-d99eb-8bf569c97-62v27" podUID="fa70dcc9-b8a7-4b3c-a582-290cbec06a9a" containerName="sequence-graph-d99eb" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Aug 01 00:17:26.224425 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:17:26.224397    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-bb639-predictor-7fccb6f469-zdmj7"
Aug 01 00:17:26.224898 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:17:26.224870    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-bb639-predictor-7fccb6f469-zdmj7" podUID="c3f7ef2d-1ce3-4f69-9660-f559ac064ea7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.46:8080: connect: connection refused"
Aug 01 00:17:27.743106 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:17:27.743072    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/splitter-graph-d0074-8f77cc87-zwsk9"]
Aug 01 00:17:27.743584 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:17:27.743463    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2da97b83-5b75-4fb7-912b-0e2dd8d1444d" containerName="kube-rbac-proxy"
Aug 01 00:17:27.743584 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:17:27.743476    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="2da97b83-5b75-4fb7-912b-0e2dd8d1444d" containerName="kube-rbac-proxy"
Aug 01 00:17:27.743584 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:17:27.743491    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2da97b83-5b75-4fb7-912b-0e2dd8d1444d" containerName="kserve-container"
Aug 01 00:17:27.743584 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:17:27.743497    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="2da97b83-5b75-4fb7-912b-0e2dd8d1444d" containerName="kserve-container"
Aug 01 00:17:27.743584 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:17:27.743562    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="2da97b83-5b75-4fb7-912b-0e2dd8d1444d" containerName="kserve-container"
Aug 01 00:17:27.743584 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:17:27.743571    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="2da97b83-5b75-4fb7-912b-0e2dd8d1444d" containerName="kube-rbac-proxy"
Aug 01 00:17:27.747934 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:17:27.747897    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/splitter-graph-d0074-8f77cc87-zwsk9"
Aug 01 00:17:27.749482 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:17:27.749451    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gttdm_b32e6d42-e4e0-4113-b8ce-f5d5c5dc13f7/konnectivity-agent/0.log" limit=1024
Aug 01 00:17:27.749797 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:17:27.749780    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"splitter-graph-d0074-kube-rbac-proxy-sar-config\""
Aug 01 00:17:27.749851 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:17:27.749781    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"splitter-graph-d0074-serving-cert\""
Aug 01 00:17:27.754249 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:17:27.754217    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/splitter-graph-d0074-8f77cc87-zwsk9"]
Aug 01 00:17:27.843152 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:17:27.843113    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a8a3c57c-c5b7-4e0b-9c1f-61c8be085c40-openshift-service-ca-bundle\") pod \"splitter-graph-d0074-8f77cc87-zwsk9\" (UID: \"a8a3c57c-c5b7-4e0b-9c1f-61c8be085c40\") " pod="kserve-ci-e2e-test/splitter-graph-d0074-8f77cc87-zwsk9"
Aug 01 00:17:27.843356 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:17:27.843206    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a8a3c57c-c5b7-4e0b-9c1f-61c8be085c40-proxy-tls\") pod \"splitter-graph-d0074-8f77cc87-zwsk9\" (UID: \"a8a3c57c-c5b7-4e0b-9c1f-61c8be085c40\") " pod="kserve-ci-e2e-test/splitter-graph-d0074-8f77cc87-zwsk9"
Aug 01 00:17:27.859323 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:17:27.859295    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gttdm_b32e6d42-e4e0-4113-b8ce-f5d5c5dc13f7/konnectivity-agent/0.log" limit=1024
Aug 01 00:17:27.944569 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:17:27.944525    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a8a3c57c-c5b7-4e0b-9c1f-61c8be085c40-proxy-tls\") pod \"splitter-graph-d0074-8f77cc87-zwsk9\" (UID: \"a8a3c57c-c5b7-4e0b-9c1f-61c8be085c40\") " pod="kserve-ci-e2e-test/splitter-graph-d0074-8f77cc87-zwsk9"
Aug 01 00:17:27.944723 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:17:27.944620    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a8a3c57c-c5b7-4e0b-9c1f-61c8be085c40-openshift-service-ca-bundle\") pod \"splitter-graph-d0074-8f77cc87-zwsk9\" (UID: \"a8a3c57c-c5b7-4e0b-9c1f-61c8be085c40\") " pod="kserve-ci-e2e-test/splitter-graph-d0074-8f77cc87-zwsk9"
Aug 01 00:17:27.945349 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:17:27.945319    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a8a3c57c-c5b7-4e0b-9c1f-61c8be085c40-openshift-service-ca-bundle\") pod \"splitter-graph-d0074-8f77cc87-zwsk9\" (UID: \"a8a3c57c-c5b7-4e0b-9c1f-61c8be085c40\") " pod="kserve-ci-e2e-test/splitter-graph-d0074-8f77cc87-zwsk9"
Aug 01 00:17:27.947110 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:17:27.947080    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a8a3c57c-c5b7-4e0b-9c1f-61c8be085c40-proxy-tls\") pod \"splitter-graph-d0074-8f77cc87-zwsk9\" (UID: \"a8a3c57c-c5b7-4e0b-9c1f-61c8be085c40\") " pod="kserve-ci-e2e-test/splitter-graph-d0074-8f77cc87-zwsk9"
Aug 01 00:17:28.060016 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:17:28.059914    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/splitter-graph-d0074-8f77cc87-zwsk9"
Aug 01 00:17:28.191477 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:17:28.191400    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/splitter-graph-d0074-8f77cc87-zwsk9"]
Aug 01 00:17:28.193802 ip-10-0-136-108 kubenswrapper[2578]: W0801 00:17:28.193772    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda8a3c57c_c5b7_4e0b_9c1f_61c8be085c40.slice/crio-14ecc60fe50da3dd6830bf836d072295da227c45b33691d46eb132a6bc39b67b WatchSource:0}: Error finding container 14ecc60fe50da3dd6830bf836d072295da227c45b33691d46eb132a6bc39b67b: Status 404 returned error can't find the container with id 14ecc60fe50da3dd6830bf836d072295da227c45b33691d46eb132a6bc39b67b
Aug 01 00:17:28.255806 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:17:28.255757    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/splitter-graph-d0074-8f77cc87-zwsk9" event={"ID":"a8a3c57c-c5b7-4e0b-9c1f-61c8be085c40","Type":"ContainerStarted","Data":"14ecc60fe50da3dd6830bf836d072295da227c45b33691d46eb132a6bc39b67b"}
Aug 01 00:17:29.261590 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:17:29.261551    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/splitter-graph-d0074-8f77cc87-zwsk9" event={"ID":"a8a3c57c-c5b7-4e0b-9c1f-61c8be085c40","Type":"ContainerStarted","Data":"ca02b06ed555a61c1f244a90cf88fbd0d423a6adea2238fea41d0529a8a0b23c"}
Aug 01 00:17:29.262000 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:17:29.261618    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/splitter-graph-d0074-8f77cc87-zwsk9"
Aug 01 00:17:29.278055 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:17:29.278003    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/splitter-graph-d0074-8f77cc87-zwsk9" podStartSLOduration=2.27798882 podStartE2EDuration="2.27798882s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-01 00:17:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-01 00:17:29.276217804 +0000 UTC m=+1873.243968563" watchObservedRunningTime="2026-08-01 00:17:29.27798882 +0000 UTC m=+1873.245739578"
Aug 01 00:17:29.909205 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:17:29.909135    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-d99eb-8bf569c97-62v27" podUID="fa70dcc9-b8a7-4b3c-a582-290cbec06a9a" containerName="sequence-graph-d99eb" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Aug 01 00:17:29.909413 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:17:29.909297    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/sequence-graph-d99eb-8bf569c97-62v27"
Aug 01 00:17:34.909024 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:17:34.908981    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-d99eb-8bf569c97-62v27" podUID="fa70dcc9-b8a7-4b3c-a582-290cbec06a9a" containerName="sequence-graph-d99eb" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Aug 01 00:17:35.271624 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:17:35.271590    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/splitter-graph-d0074-8f77cc87-zwsk9"
Aug 01 00:17:36.225377 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:17:36.225336    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-bb639-predictor-7fccb6f469-zdmj7" podUID="c3f7ef2d-1ce3-4f69-9660-f559ac064ea7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.46:8080: connect: connection refused"
Aug 01 00:17:37.836007 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:17:37.835973    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/splitter-graph-d0074-8f77cc87-zwsk9"]
Aug 01 00:17:37.836493 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:17:37.836215    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/splitter-graph-d0074-8f77cc87-zwsk9" podUID="a8a3c57c-c5b7-4e0b-9c1f-61c8be085c40" containerName="splitter-graph-d0074" containerID="cri-o://ca02b06ed555a61c1f244a90cf88fbd0d423a6adea2238fea41d0529a8a0b23c" gracePeriod=30
Aug 01 00:17:37.854844 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:17:37.854813    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gttdm_b32e6d42-e4e0-4113-b8ce-f5d5c5dc13f7/konnectivity-agent/0.log" limit=1024
Aug 01 00:17:37.959259 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:17:37.959220    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-d0074-predictor-5fc6bf94b7-fxc9g"]
Aug 01 00:17:37.959568 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:17:37.959536    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-d0074-predictor-5fc6bf94b7-fxc9g" podUID="9f543e2b-4d86-42f6-9859-dc6dab87f5b2" containerName="kserve-container" containerID="cri-o://6a5c95b385d5d7a1e496b01533f936a74a51ed07d248af13e062fed1ea55b490" gracePeriod=30
Aug 01 00:17:37.959880 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:17:37.959626    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-d0074-predictor-5fc6bf94b7-fxc9g" podUID="9f543e2b-4d86-42f6-9859-dc6dab87f5b2" containerName="kube-rbac-proxy" containerID="cri-o://fe51d901790f32d56f2fb9efa1a220fdcb0137479b166b5ef865ec1e55b22f01" gracePeriod=30
Aug 01 00:17:37.981752 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:17:37.981721    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gttdm_b32e6d42-e4e0-4113-b8ce-f5d5c5dc13f7/konnectivity-agent/0.log" limit=1024
Aug 01 00:17:38.030535 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:17:38.030499    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-2f013-predictor-bb647f765-5tkvs"]
Aug 01 00:17:38.034320 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:17:38.034298    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-2f013-predictor-bb647f765-5tkvs"
Aug 01 00:17:38.036236 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:17:38.036216    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-2f013-predictor-serving-cert\""
Aug 01 00:17:38.036317 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:17:38.036261    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-2f013-kube-rbac-proxy-sar-config\""
Aug 01 00:17:38.047618 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:17:38.047593    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-2f013-predictor-bb647f765-5tkvs"]
Aug 01 00:17:38.061865 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:17:38.061833    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gttdm_b32e6d42-e4e0-4113-b8ce-f5d5c5dc13f7/konnectivity-agent/0.log" limit=1024
Aug 01 00:17:38.125903 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:17:38.125817    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gttdm_b32e6d42-e4e0-4113-b8ce-f5d5c5dc13f7/konnectivity-agent/0.log" limit=1024
Aug 01 00:17:38.134164 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:17:38.134127    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/915a3a14-b2de-4ab3-ac4a-5215dcecab1f-proxy-tls\") pod \"success-200-isvc-2f013-predictor-bb647f765-5tkvs\" (UID: \"915a3a14-b2de-4ab3-ac4a-5215dcecab1f\") " pod="kserve-ci-e2e-test/success-200-isvc-2f013-predictor-bb647f765-5tkvs"
Aug 01 00:17:38.134364 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:17:38.134205    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qzvjb\" (UniqueName: \"kubernetes.io/projected/915a3a14-b2de-4ab3-ac4a-5215dcecab1f-kube-api-access-qzvjb\") pod \"success-200-isvc-2f013-predictor-bb647f765-5tkvs\" (UID: \"915a3a14-b2de-4ab3-ac4a-5215dcecab1f\") " pod="kserve-ci-e2e-test/success-200-isvc-2f013-predictor-bb647f765-5tkvs"
Aug 01 00:17:38.134364 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:17:38.134267    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/915a3a14-b2de-4ab3-ac4a-5215dcecab1f-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-2f013-predictor-bb647f765-5tkvs\" (UID: \"915a3a14-b2de-4ab3-ac4a-5215dcecab1f\") " pod="kserve-ci-e2e-test/success-200-isvc-2f013-predictor-bb647f765-5tkvs"
Aug 01 00:17:38.235013 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:17:38.234931    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/915a3a14-b2de-4ab3-ac4a-5215dcecab1f-proxy-tls\") pod \"success-200-isvc-2f013-predictor-bb647f765-5tkvs\" (UID: \"915a3a14-b2de-4ab3-ac4a-5215dcecab1f\") " pod="kserve-ci-e2e-test/success-200-isvc-2f013-predictor-bb647f765-5tkvs"
Aug 01 00:17:38.235013 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:17:38.234979    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qzvjb\" (UniqueName: \"kubernetes.io/projected/915a3a14-b2de-4ab3-ac4a-5215dcecab1f-kube-api-access-qzvjb\") pod \"success-200-isvc-2f013-predictor-bb647f765-5tkvs\" (UID: \"915a3a14-b2de-4ab3-ac4a-5215dcecab1f\") " pod="kserve-ci-e2e-test/success-200-isvc-2f013-predictor-bb647f765-5tkvs"
Aug 01 00:17:38.235013 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:17:38.235012    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/915a3a14-b2de-4ab3-ac4a-5215dcecab1f-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-2f013-predictor-bb647f765-5tkvs\" (UID: \"915a3a14-b2de-4ab3-ac4a-5215dcecab1f\") " pod="kserve-ci-e2e-test/success-200-isvc-2f013-predictor-bb647f765-5tkvs"
Aug 01 00:17:38.235777 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:17:38.235754    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/915a3a14-b2de-4ab3-ac4a-5215dcecab1f-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-2f013-predictor-bb647f765-5tkvs\" (UID: \"915a3a14-b2de-4ab3-ac4a-5215dcecab1f\") " pod="kserve-ci-e2e-test/success-200-isvc-2f013-predictor-bb647f765-5tkvs"
Aug 01 00:17:38.237744 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:17:38.237722    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/915a3a14-b2de-4ab3-ac4a-5215dcecab1f-proxy-tls\") pod \"success-200-isvc-2f013-predictor-bb647f765-5tkvs\" (UID: \"915a3a14-b2de-4ab3-ac4a-5215dcecab1f\") " pod="kserve-ci-e2e-test/success-200-isvc-2f013-predictor-bb647f765-5tkvs"
Aug 01 00:17:38.245820 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:17:38.245797    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qzvjb\" (UniqueName: \"kubernetes.io/projected/915a3a14-b2de-4ab3-ac4a-5215dcecab1f-kube-api-access-qzvjb\") pod \"success-200-isvc-2f013-predictor-bb647f765-5tkvs\" (UID: \"915a3a14-b2de-4ab3-ac4a-5215dcecab1f\") " pod="kserve-ci-e2e-test/success-200-isvc-2f013-predictor-bb647f765-5tkvs"
Aug 01 00:17:38.259989 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:17:38.259965    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gttdm_b32e6d42-e4e0-4113-b8ce-f5d5c5dc13f7/konnectivity-agent/0.log" limit=1024
Aug 01 00:17:38.312212 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:17:38.312153    2578 generic.go:356] "Generic (PLEG): container finished" podID="9f543e2b-4d86-42f6-9859-dc6dab87f5b2" containerID="fe51d901790f32d56f2fb9efa1a220fdcb0137479b166b5ef865ec1e55b22f01" exitCode=2
Aug 01 00:17:38.312411 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:17:38.312208    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-d0074-predictor-5fc6bf94b7-fxc9g" event={"ID":"9f543e2b-4d86-42f6-9859-dc6dab87f5b2","Type":"ContainerDied","Data":"fe51d901790f32d56f2fb9efa1a220fdcb0137479b166b5ef865ec1e55b22f01"}
Aug 01 00:17:38.346816 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:17:38.346787    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-2f013-predictor-bb647f765-5tkvs"
Aug 01 00:17:38.479728 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:17:38.479702    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-2f013-predictor-bb647f765-5tkvs"]
Aug 01 00:17:38.481965 ip-10-0-136-108 kubenswrapper[2578]: W0801 00:17:38.481940    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod915a3a14_b2de_4ab3_ac4a_5215dcecab1f.slice/crio-c2c29965a1bf81cd2268f89bf3fae2bc23a44fc79c83f8dafb740d0b656a81ff WatchSource:0}: Error finding container c2c29965a1bf81cd2268f89bf3fae2bc23a44fc79c83f8dafb740d0b656a81ff: Status 404 returned error can't find the container with id c2c29965a1bf81cd2268f89bf3fae2bc23a44fc79c83f8dafb740d0b656a81ff
Aug 01 00:17:39.319211 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:17:39.319164    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-2f013-predictor-bb647f765-5tkvs" event={"ID":"915a3a14-b2de-4ab3-ac4a-5215dcecab1f","Type":"ContainerStarted","Data":"ef5dd385f3b1bb6a1b0bcf09a4ca20bed9b6dd7605504962d19eb40742facddb"}
Aug 01 00:17:39.319604 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:17:39.319238    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-2f013-predictor-bb647f765-5tkvs" event={"ID":"915a3a14-b2de-4ab3-ac4a-5215dcecab1f","Type":"ContainerStarted","Data":"84d623cc225370371f0f4157b5af802f8f0ac52b03ab8d64ebf711397cee823b"}
Aug 01 00:17:39.319604 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:17:39.319260    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-2f013-predictor-bb647f765-5tkvs" event={"ID":"915a3a14-b2de-4ab3-ac4a-5215dcecab1f","Type":"ContainerStarted","Data":"c2c29965a1bf81cd2268f89bf3fae2bc23a44fc79c83f8dafb740d0b656a81ff"}
Aug 01 00:17:39.319604 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:17:39.319328    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-2f013-predictor-bb647f765-5tkvs"
Aug 01 00:17:39.338417 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:17:39.338366    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/success-200-isvc-2f013-predictor-bb647f765-5tkvs" podStartSLOduration=1.33834936 podStartE2EDuration="1.33834936s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-01 00:17:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-01 00:17:39.336360005 +0000 UTC m=+1883.304110766" watchObservedRunningTime="2026-08-01 00:17:39.33834936 +0000 UTC m=+1883.306100118"
Aug 01 00:17:39.908977 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:17:39.908934    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-d99eb-8bf569c97-62v27" podUID="fa70dcc9-b8a7-4b3c-a582-290cbec06a9a" containerName="sequence-graph-d99eb" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Aug 01 00:17:39.961830 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:17:39.961782    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-d0074-predictor-5fc6bf94b7-fxc9g" podUID="9f543e2b-4d86-42f6-9859-dc6dab87f5b2" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.45:8643/healthz\": dial tcp 10.133.0.45:8643: connect: connection refused"
Aug 01 00:17:40.269889 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:17:40.269849    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/splitter-graph-d0074-8f77cc87-zwsk9" podUID="a8a3c57c-c5b7-4e0b-9c1f-61c8be085c40" containerName="splitter-graph-d0074" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Aug 01 00:17:40.324085 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:17:40.324055    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-2f013-predictor-bb647f765-5tkvs"
Aug 01 00:17:40.325319 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:17:40.325292    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-2f013-predictor-bb647f765-5tkvs" podUID="915a3a14-b2de-4ab3-ac4a-5215dcecab1f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.48:8080: connect: connection refused"
Aug 01 00:17:41.215907 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:17:41.215880    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-d0074-predictor-5fc6bf94b7-fxc9g"
Aug 01 00:17:41.263407 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:17:41.263368    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9f543e2b-4d86-42f6-9859-dc6dab87f5b2-proxy-tls\") pod \"9f543e2b-4d86-42f6-9859-dc6dab87f5b2\" (UID: \"9f543e2b-4d86-42f6-9859-dc6dab87f5b2\") "
Aug 01 00:17:41.263608 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:17:41.263443    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fq76m\" (UniqueName: \"kubernetes.io/projected/9f543e2b-4d86-42f6-9859-dc6dab87f5b2-kube-api-access-fq76m\") pod \"9f543e2b-4d86-42f6-9859-dc6dab87f5b2\" (UID: \"9f543e2b-4d86-42f6-9859-dc6dab87f5b2\") "
Aug 01 00:17:41.263608 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:17:41.263482    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9f543e2b-4d86-42f6-9859-dc6dab87f5b2-kube-rbac-proxy-sar-config\") pod \"9f543e2b-4d86-42f6-9859-dc6dab87f5b2\" (UID: \"9f543e2b-4d86-42f6-9859-dc6dab87f5b2\") "
Aug 01 00:17:41.263861 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:17:41.263834    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9f543e2b-4d86-42f6-9859-dc6dab87f5b2-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "9f543e2b-4d86-42f6-9859-dc6dab87f5b2" (UID: "9f543e2b-4d86-42f6-9859-dc6dab87f5b2"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 01 00:17:41.265587 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:17:41.265561    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9f543e2b-4d86-42f6-9859-dc6dab87f5b2-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "9f543e2b-4d86-42f6-9859-dc6dab87f5b2" (UID: "9f543e2b-4d86-42f6-9859-dc6dab87f5b2"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 01 00:17:41.265698 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:17:41.265623    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9f543e2b-4d86-42f6-9859-dc6dab87f5b2-kube-api-access-fq76m" (OuterVolumeSpecName: "kube-api-access-fq76m") pod "9f543e2b-4d86-42f6-9859-dc6dab87f5b2" (UID: "9f543e2b-4d86-42f6-9859-dc6dab87f5b2"). InnerVolumeSpecName "kube-api-access-fq76m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 01 00:17:41.329956 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:17:41.329856    2578 generic.go:356] "Generic (PLEG): container finished" podID="9f543e2b-4d86-42f6-9859-dc6dab87f5b2" containerID="6a5c95b385d5d7a1e496b01533f936a74a51ed07d248af13e062fed1ea55b490" exitCode=0
Aug 01 00:17:41.329956 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:17:41.329906    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-d0074-predictor-5fc6bf94b7-fxc9g"
Aug 01 00:17:41.329956 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:17:41.329925    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-d0074-predictor-5fc6bf94b7-fxc9g" event={"ID":"9f543e2b-4d86-42f6-9859-dc6dab87f5b2","Type":"ContainerDied","Data":"6a5c95b385d5d7a1e496b01533f936a74a51ed07d248af13e062fed1ea55b490"}
Aug 01 00:17:41.330580 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:17:41.329984    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-d0074-predictor-5fc6bf94b7-fxc9g" event={"ID":"9f543e2b-4d86-42f6-9859-dc6dab87f5b2","Type":"ContainerDied","Data":"4f27fe938cb1428763a8c64a6665a452f2dde28d8ddddc07b645879c98923eba"}
Aug 01 00:17:41.330580 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:17:41.330012    2578 scope.go:117] "RemoveContainer" containerID="fe51d901790f32d56f2fb9efa1a220fdcb0137479b166b5ef865ec1e55b22f01"
Aug 01 00:17:41.330787 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:17:41.330760    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-2f013-predictor-bb647f765-5tkvs" podUID="915a3a14-b2de-4ab3-ac4a-5215dcecab1f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.48:8080: connect: connection refused"
Aug 01 00:17:41.339933 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:17:41.339910    2578 scope.go:117] "RemoveContainer" containerID="6a5c95b385d5d7a1e496b01533f936a74a51ed07d248af13e062fed1ea55b490"
Aug 01 00:17:41.348049 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:17:41.348029    2578 scope.go:117] "RemoveContainer" containerID="fe51d901790f32d56f2fb9efa1a220fdcb0137479b166b5ef865ec1e55b22f01"
Aug 01 00:17:41.348613 ip-10-0-136-108 kubenswrapper[2578]: E0801 00:17:41.348441    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fe51d901790f32d56f2fb9efa1a220fdcb0137479b166b5ef865ec1e55b22f01\": container with ID starting with fe51d901790f32d56f2fb9efa1a220fdcb0137479b166b5ef865ec1e55b22f01 not found: ID does not exist" containerID="fe51d901790f32d56f2fb9efa1a220fdcb0137479b166b5ef865ec1e55b22f01"
Aug 01 00:17:41.348613 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:17:41.348485    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fe51d901790f32d56f2fb9efa1a220fdcb0137479b166b5ef865ec1e55b22f01"} err="failed to get container status \"fe51d901790f32d56f2fb9efa1a220fdcb0137479b166b5ef865ec1e55b22f01\": rpc error: code = NotFound desc = could not find container \"fe51d901790f32d56f2fb9efa1a220fdcb0137479b166b5ef865ec1e55b22f01\": container with ID starting with fe51d901790f32d56f2fb9efa1a220fdcb0137479b166b5ef865ec1e55b22f01 not found: ID does not exist"
Aug 01 00:17:41.348613 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:17:41.348519    2578 scope.go:117] "RemoveContainer" containerID="6a5c95b385d5d7a1e496b01533f936a74a51ed07d248af13e062fed1ea55b490"
Aug 01 00:17:41.349010 ip-10-0-136-108 kubenswrapper[2578]: E0801 00:17:41.348981    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6a5c95b385d5d7a1e496b01533f936a74a51ed07d248af13e062fed1ea55b490\": container with ID starting with 6a5c95b385d5d7a1e496b01533f936a74a51ed07d248af13e062fed1ea55b490 not found: ID does not exist" containerID="6a5c95b385d5d7a1e496b01533f936a74a51ed07d248af13e062fed1ea55b490"
Aug 01 00:17:41.349101 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:17:41.349019    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6a5c95b385d5d7a1e496b01533f936a74a51ed07d248af13e062fed1ea55b490"} err="failed to get container status \"6a5c95b385d5d7a1e496b01533f936a74a51ed07d248af13e062fed1ea55b490\": rpc error: code = NotFound desc = could not find container \"6a5c95b385d5d7a1e496b01533f936a74a51ed07d248af13e062fed1ea55b490\": container with ID starting with 6a5c95b385d5d7a1e496b01533f936a74a51ed07d248af13e062fed1ea55b490 not found: ID does not exist"
Aug 01 00:17:41.350613 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:17:41.350591    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-d0074-predictor-5fc6bf94b7-fxc9g"]
Aug 01 00:17:41.355049 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:17:41.355028    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-d0074-predictor-5fc6bf94b7-fxc9g"]
Aug 01 00:17:41.364943 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:17:41.364920    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fq76m\" (UniqueName: \"kubernetes.io/projected/9f543e2b-4d86-42f6-9859-dc6dab87f5b2-kube-api-access-fq76m\") on node \"ip-10-0-136-108.ec2.internal\" DevicePath \"\""
Aug 01 00:17:41.365067 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:17:41.365046    2578 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9f543e2b-4d86-42f6-9859-dc6dab87f5b2-kube-rbac-proxy-sar-config\") on node \"ip-10-0-136-108.ec2.internal\" DevicePath \"\""
Aug 01 00:17:41.365145 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:17:41.365069    2578 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9f543e2b-4d86-42f6-9859-dc6dab87f5b2-proxy-tls\") on node \"ip-10-0-136-108.ec2.internal\" DevicePath \"\""
Aug 01 00:17:42.489577 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:17:42.489538    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9f543e2b-4d86-42f6-9859-dc6dab87f5b2" path="/var/lib/kubelet/pods/9f543e2b-4d86-42f6-9859-dc6dab87f5b2/volumes"
Aug 01 00:17:44.909351 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:17:44.909301    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-d99eb-8bf569c97-62v27" podUID="fa70dcc9-b8a7-4b3c-a582-290cbec06a9a" containerName="sequence-graph-d99eb" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Aug 01 00:17:45.269993 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:17:45.269956    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/splitter-graph-d0074-8f77cc87-zwsk9" podUID="a8a3c57c-c5b7-4e0b-9c1f-61c8be085c40" containerName="splitter-graph-d0074" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Aug 01 00:17:46.225372 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:17:46.225329    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-bb639-predictor-7fccb6f469-zdmj7" podUID="c3f7ef2d-1ce3-4f69-9660-f559ac064ea7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.46:8080: connect: connection refused"
Aug 01 00:17:46.335743 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:17:46.335709    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-2f013-predictor-bb647f765-5tkvs"
Aug 01 00:17:46.336309 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:17:46.336275    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-2f013-predictor-bb647f765-5tkvs" podUID="915a3a14-b2de-4ab3-ac4a-5215dcecab1f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.48:8080: connect: connection refused"
Aug 01 00:17:47.362348 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:17:47.362317    2578 generic.go:356] "Generic (PLEG): container finished" podID="fa70dcc9-b8a7-4b3c-a582-290cbec06a9a" containerID="4589c656a83d953a52229d055aae3b3adea404000ab6ec19d06ed243017d68f2" exitCode=0
Aug 01 00:17:47.362765 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:17:47.362390    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sequence-graph-d99eb-8bf569c97-62v27" event={"ID":"fa70dcc9-b8a7-4b3c-a582-290cbec06a9a","Type":"ContainerDied","Data":"4589c656a83d953a52229d055aae3b3adea404000ab6ec19d06ed243017d68f2"}
Aug 01 00:17:47.413752 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:17:47.413722    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sequence-graph-d99eb-8bf569c97-62v27"
Aug 01 00:17:47.522236 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:17:47.522128    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fa70dcc9-b8a7-4b3c-a582-290cbec06a9a-openshift-service-ca-bundle\") pod \"fa70dcc9-b8a7-4b3c-a582-290cbec06a9a\" (UID: \"fa70dcc9-b8a7-4b3c-a582-290cbec06a9a\") "
Aug 01 00:17:47.522396 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:17:47.522237    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fa70dcc9-b8a7-4b3c-a582-290cbec06a9a-proxy-tls\") pod \"fa70dcc9-b8a7-4b3c-a582-290cbec06a9a\" (UID: \"fa70dcc9-b8a7-4b3c-a582-290cbec06a9a\") "
Aug 01 00:17:47.522516 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:17:47.522491    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fa70dcc9-b8a7-4b3c-a582-290cbec06a9a-openshift-service-ca-bundle" (OuterVolumeSpecName: "openshift-service-ca-bundle") pod "fa70dcc9-b8a7-4b3c-a582-290cbec06a9a" (UID: "fa70dcc9-b8a7-4b3c-a582-290cbec06a9a"). InnerVolumeSpecName "openshift-service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 01 00:17:47.524211 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:17:47.524188    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fa70dcc9-b8a7-4b3c-a582-290cbec06a9a-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "fa70dcc9-b8a7-4b3c-a582-290cbec06a9a" (UID: "fa70dcc9-b8a7-4b3c-a582-290cbec06a9a"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 01 00:17:47.623061 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:17:47.623019    2578 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fa70dcc9-b8a7-4b3c-a582-290cbec06a9a-proxy-tls\") on node \"ip-10-0-136-108.ec2.internal\" DevicePath \"\""
Aug 01 00:17:47.623061 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:17:47.623051    2578 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fa70dcc9-b8a7-4b3c-a582-290cbec06a9a-openshift-service-ca-bundle\") on node \"ip-10-0-136-108.ec2.internal\" DevicePath \"\""
Aug 01 00:17:48.369068 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:17:48.369038    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sequence-graph-d99eb-8bf569c97-62v27"
Aug 01 00:17:48.369502 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:17:48.369038    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sequence-graph-d99eb-8bf569c97-62v27" event={"ID":"fa70dcc9-b8a7-4b3c-a582-290cbec06a9a","Type":"ContainerDied","Data":"c3f9ad04bb99cd83dc1e5122c142e9a513818b158f5dcf02fc635f71b634f113"}
Aug 01 00:17:48.369502 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:17:48.369198    2578 scope.go:117] "RemoveContainer" containerID="4589c656a83d953a52229d055aae3b3adea404000ab6ec19d06ed243017d68f2"
Aug 01 00:17:48.387785 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:17:48.387756    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-d99eb-8bf569c97-62v27"]
Aug 01 00:17:48.391236 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:17:48.391213    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-d99eb-8bf569c97-62v27"]
Aug 01 00:17:48.489988 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:17:48.489956    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fa70dcc9-b8a7-4b3c-a582-290cbec06a9a" path="/var/lib/kubelet/pods/fa70dcc9-b8a7-4b3c-a582-290cbec06a9a/volumes"
Aug 01 00:17:50.270056 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:17:50.270011    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/splitter-graph-d0074-8f77cc87-zwsk9" podUID="a8a3c57c-c5b7-4e0b-9c1f-61c8be085c40" containerName="splitter-graph-d0074" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Aug 01 00:17:50.270478 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:17:50.270137    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/splitter-graph-d0074-8f77cc87-zwsk9"
Aug 01 00:17:55.270564 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:17:55.270517    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/splitter-graph-d0074-8f77cc87-zwsk9" podUID="a8a3c57c-c5b7-4e0b-9c1f-61c8be085c40" containerName="splitter-graph-d0074" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Aug 01 00:17:56.225846 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:17:56.225807    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-bb639-predictor-7fccb6f469-zdmj7" podUID="c3f7ef2d-1ce3-4f69-9660-f559ac064ea7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.46:8080: connect: connection refused"
Aug 01 00:17:56.336625 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:17:56.336581    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-2f013-predictor-bb647f765-5tkvs" podUID="915a3a14-b2de-4ab3-ac4a-5215dcecab1f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.48:8080: connect: connection refused"
Aug 01 00:18:00.269941 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:18:00.269898    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/splitter-graph-d0074-8f77cc87-zwsk9" podUID="a8a3c57c-c5b7-4e0b-9c1f-61c8be085c40" containerName="splitter-graph-d0074" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Aug 01 00:18:05.269894 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:18:05.269854    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/splitter-graph-d0074-8f77cc87-zwsk9" podUID="a8a3c57c-c5b7-4e0b-9c1f-61c8be085c40" containerName="splitter-graph-d0074" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Aug 01 00:18:06.226221 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:18:06.226184    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-bb639-predictor-7fccb6f469-zdmj7"
Aug 01 00:18:06.336226 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:18:06.336190    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-2f013-predictor-bb647f765-5tkvs" podUID="915a3a14-b2de-4ab3-ac4a-5215dcecab1f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.48:8080: connect: connection refused"
Aug 01 00:18:07.988443 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:18:07.988418    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/splitter-graph-d0074-8f77cc87-zwsk9"
Aug 01 00:18:08.099081 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:18:08.098989    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a8a3c57c-c5b7-4e0b-9c1f-61c8be085c40-proxy-tls\") pod \"a8a3c57c-c5b7-4e0b-9c1f-61c8be085c40\" (UID: \"a8a3c57c-c5b7-4e0b-9c1f-61c8be085c40\") "
Aug 01 00:18:08.099081 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:18:08.099052    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a8a3c57c-c5b7-4e0b-9c1f-61c8be085c40-openshift-service-ca-bundle\") pod \"a8a3c57c-c5b7-4e0b-9c1f-61c8be085c40\" (UID: \"a8a3c57c-c5b7-4e0b-9c1f-61c8be085c40\") "
Aug 01 00:18:08.099422 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:18:08.099397    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a8a3c57c-c5b7-4e0b-9c1f-61c8be085c40-openshift-service-ca-bundle" (OuterVolumeSpecName: "openshift-service-ca-bundle") pod "a8a3c57c-c5b7-4e0b-9c1f-61c8be085c40" (UID: "a8a3c57c-c5b7-4e0b-9c1f-61c8be085c40"). InnerVolumeSpecName "openshift-service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 01 00:18:08.101154 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:18:08.101134    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a8a3c57c-c5b7-4e0b-9c1f-61c8be085c40-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "a8a3c57c-c5b7-4e0b-9c1f-61c8be085c40" (UID: "a8a3c57c-c5b7-4e0b-9c1f-61c8be085c40"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 01 00:18:08.200571 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:18:08.200531    2578 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a8a3c57c-c5b7-4e0b-9c1f-61c8be085c40-openshift-service-ca-bundle\") on node \"ip-10-0-136-108.ec2.internal\" DevicePath \"\""
Aug 01 00:18:08.200571 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:18:08.200564    2578 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a8a3c57c-c5b7-4e0b-9c1f-61c8be085c40-proxy-tls\") on node \"ip-10-0-136-108.ec2.internal\" DevicePath \"\""
Aug 01 00:18:08.465965 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:18:08.465870    2578 generic.go:356] "Generic (PLEG): container finished" podID="a8a3c57c-c5b7-4e0b-9c1f-61c8be085c40" containerID="ca02b06ed555a61c1f244a90cf88fbd0d423a6adea2238fea41d0529a8a0b23c" exitCode=0
Aug 01 00:18:08.465965 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:18:08.465918    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/splitter-graph-d0074-8f77cc87-zwsk9"
Aug 01 00:18:08.465965 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:18:08.465933    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/splitter-graph-d0074-8f77cc87-zwsk9" event={"ID":"a8a3c57c-c5b7-4e0b-9c1f-61c8be085c40","Type":"ContainerDied","Data":"ca02b06ed555a61c1f244a90cf88fbd0d423a6adea2238fea41d0529a8a0b23c"}
Aug 01 00:18:08.466266 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:18:08.465985    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/splitter-graph-d0074-8f77cc87-zwsk9" event={"ID":"a8a3c57c-c5b7-4e0b-9c1f-61c8be085c40","Type":"ContainerDied","Data":"14ecc60fe50da3dd6830bf836d072295da227c45b33691d46eb132a6bc39b67b"}
Aug 01 00:18:08.466845 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:18:08.466015    2578 scope.go:117] "RemoveContainer" containerID="ca02b06ed555a61c1f244a90cf88fbd0d423a6adea2238fea41d0529a8a0b23c"
Aug 01 00:18:08.477841 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:18:08.477819    2578 scope.go:117] "RemoveContainer" containerID="ca02b06ed555a61c1f244a90cf88fbd0d423a6adea2238fea41d0529a8a0b23c"
Aug 01 00:18:08.478140 ip-10-0-136-108 kubenswrapper[2578]: E0801 00:18:08.478121    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ca02b06ed555a61c1f244a90cf88fbd0d423a6adea2238fea41d0529a8a0b23c\": container with ID starting with ca02b06ed555a61c1f244a90cf88fbd0d423a6adea2238fea41d0529a8a0b23c not found: ID does not exist" containerID="ca02b06ed555a61c1f244a90cf88fbd0d423a6adea2238fea41d0529a8a0b23c"
Aug 01 00:18:08.478212 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:18:08.478149    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ca02b06ed555a61c1f244a90cf88fbd0d423a6adea2238fea41d0529a8a0b23c"} err="failed to get container status \"ca02b06ed555a61c1f244a90cf88fbd0d423a6adea2238fea41d0529a8a0b23c\": rpc error: code = NotFound desc = could not find container \"ca02b06ed555a61c1f244a90cf88fbd0d423a6adea2238fea41d0529a8a0b23c\": container with ID starting with ca02b06ed555a61c1f244a90cf88fbd0d423a6adea2238fea41d0529a8a0b23c not found: ID does not exist"
Aug 01 00:18:08.490163 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:18:08.490138    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/splitter-graph-d0074-8f77cc87-zwsk9"]
Aug 01 00:18:08.490310 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:18:08.490188    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/splitter-graph-d0074-8f77cc87-zwsk9"]
Aug 01 00:18:10.489069 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:18:10.489033    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a8a3c57c-c5b7-4e0b-9c1f-61c8be085c40" path="/var/lib/kubelet/pods/a8a3c57c-c5b7-4e0b-9c1f-61c8be085c40/volumes"
Aug 01 00:18:16.336900 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:18:16.336860    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-2f013-predictor-bb647f765-5tkvs" podUID="915a3a14-b2de-4ab3-ac4a-5215dcecab1f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.48:8080: connect: connection refused"
Aug 01 00:18:17.544528 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:18:17.544491    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/switch-graph-bb639-658b9f69db-rwxvz"]
Aug 01 00:18:17.544894 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:18:17.544857    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a8a3c57c-c5b7-4e0b-9c1f-61c8be085c40" containerName="splitter-graph-d0074"
Aug 01 00:18:17.544894 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:18:17.544869    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="a8a3c57c-c5b7-4e0b-9c1f-61c8be085c40" containerName="splitter-graph-d0074"
Aug 01 00:18:17.544894 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:18:17.544884    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fa70dcc9-b8a7-4b3c-a582-290cbec06a9a" containerName="sequence-graph-d99eb"
Aug 01 00:18:17.544894 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:18:17.544890    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="fa70dcc9-b8a7-4b3c-a582-290cbec06a9a" containerName="sequence-graph-d99eb"
Aug 01 00:18:17.545030 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:18:17.544901    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9f543e2b-4d86-42f6-9859-dc6dab87f5b2" containerName="kube-rbac-proxy"
Aug 01 00:18:17.545030 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:18:17.544907    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="9f543e2b-4d86-42f6-9859-dc6dab87f5b2" containerName="kube-rbac-proxy"
Aug 01 00:18:17.545030 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:18:17.544918    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9f543e2b-4d86-42f6-9859-dc6dab87f5b2" containerName="kserve-container"
Aug 01 00:18:17.545030 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:18:17.544923    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="9f543e2b-4d86-42f6-9859-dc6dab87f5b2" containerName="kserve-container"
Aug 01 00:18:17.545030 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:18:17.544973    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="9f543e2b-4d86-42f6-9859-dc6dab87f5b2" containerName="kube-rbac-proxy"
Aug 01 00:18:17.545030 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:18:17.545009    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="9f543e2b-4d86-42f6-9859-dc6dab87f5b2" containerName="kserve-container"
Aug 01 00:18:17.545030 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:18:17.545015    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="a8a3c57c-c5b7-4e0b-9c1f-61c8be085c40" containerName="splitter-graph-d0074"
Aug 01 00:18:17.545030 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:18:17.545021    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="fa70dcc9-b8a7-4b3c-a582-290cbec06a9a" containerName="sequence-graph-d99eb"
Aug 01 00:18:17.549633 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:18:17.549607    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gttdm_b32e6d42-e4e0-4113-b8ce-f5d5c5dc13f7/konnectivity-agent/0.log" limit=1024
Aug 01 00:18:17.549757 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:18:17.549647    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/switch-graph-bb639-658b9f69db-rwxvz"
Aug 01 00:18:17.551572 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:18:17.551555    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"switch-graph-bb639-kube-rbac-proxy-sar-config\""
Aug 01 00:18:17.551572 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:18:17.551563    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"switch-graph-bb639-serving-cert\""
Aug 01 00:18:17.555487 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:18:17.555465    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/switch-graph-bb639-658b9f69db-rwxvz"]
Aug 01 00:18:17.687672 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:18:17.687629    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/141f7de1-1db2-49fa-89d9-5cde41749e2b-proxy-tls\") pod \"switch-graph-bb639-658b9f69db-rwxvz\" (UID: \"141f7de1-1db2-49fa-89d9-5cde41749e2b\") " pod="kserve-ci-e2e-test/switch-graph-bb639-658b9f69db-rwxvz"
Aug 01 00:18:17.687855 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:18:17.687701    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/141f7de1-1db2-49fa-89d9-5cde41749e2b-openshift-service-ca-bundle\") pod \"switch-graph-bb639-658b9f69db-rwxvz\" (UID: \"141f7de1-1db2-49fa-89d9-5cde41749e2b\") " pod="kserve-ci-e2e-test/switch-graph-bb639-658b9f69db-rwxvz"
Aug 01 00:18:17.788911 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:18:17.788871    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/141f7de1-1db2-49fa-89d9-5cde41749e2b-proxy-tls\") pod \"switch-graph-bb639-658b9f69db-rwxvz\" (UID: \"141f7de1-1db2-49fa-89d9-5cde41749e2b\") " pod="kserve-ci-e2e-test/switch-graph-bb639-658b9f69db-rwxvz"
Aug 01 00:18:17.789108 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:18:17.788937    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/141f7de1-1db2-49fa-89d9-5cde41749e2b-openshift-service-ca-bundle\") pod \"switch-graph-bb639-658b9f69db-rwxvz\" (UID: \"141f7de1-1db2-49fa-89d9-5cde41749e2b\") " pod="kserve-ci-e2e-test/switch-graph-bb639-658b9f69db-rwxvz"
Aug 01 00:18:17.789108 ip-10-0-136-108 kubenswrapper[2578]: E0801 00:18:17.789033    2578 secret.go:189] Couldn't get secret kserve-ci-e2e-test/switch-graph-bb639-serving-cert: secret "switch-graph-bb639-serving-cert" not found
Aug 01 00:18:17.789235 ip-10-0-136-108 kubenswrapper[2578]: E0801 00:18:17.789118    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/141f7de1-1db2-49fa-89d9-5cde41749e2b-proxy-tls podName:141f7de1-1db2-49fa-89d9-5cde41749e2b nodeName:}" failed. No retries permitted until 2026-08-01 00:18:18.28910069 +0000 UTC m=+1922.256851426 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/141f7de1-1db2-49fa-89d9-5cde41749e2b-proxy-tls") pod "switch-graph-bb639-658b9f69db-rwxvz" (UID: "141f7de1-1db2-49fa-89d9-5cde41749e2b") : secret "switch-graph-bb639-serving-cert" not found
Aug 01 00:18:17.789637 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:18:17.789617    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/141f7de1-1db2-49fa-89d9-5cde41749e2b-openshift-service-ca-bundle\") pod \"switch-graph-bb639-658b9f69db-rwxvz\" (UID: \"141f7de1-1db2-49fa-89d9-5cde41749e2b\") " pod="kserve-ci-e2e-test/switch-graph-bb639-658b9f69db-rwxvz"
Aug 01 00:18:17.944732 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:18:17.944630    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gttdm_b32e6d42-e4e0-4113-b8ce-f5d5c5dc13f7/konnectivity-agent/0.log" limit=1024
Aug 01 00:18:18.294346 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:18:18.294313    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/141f7de1-1db2-49fa-89d9-5cde41749e2b-proxy-tls\") pod \"switch-graph-bb639-658b9f69db-rwxvz\" (UID: \"141f7de1-1db2-49fa-89d9-5cde41749e2b\") " pod="kserve-ci-e2e-test/switch-graph-bb639-658b9f69db-rwxvz"
Aug 01 00:18:18.296812 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:18:18.296788    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/141f7de1-1db2-49fa-89d9-5cde41749e2b-proxy-tls\") pod \"switch-graph-bb639-658b9f69db-rwxvz\" (UID: \"141f7de1-1db2-49fa-89d9-5cde41749e2b\") " pod="kserve-ci-e2e-test/switch-graph-bb639-658b9f69db-rwxvz"
Aug 01 00:18:18.461575 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:18:18.461524    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/switch-graph-bb639-658b9f69db-rwxvz"
Aug 01 00:18:18.590783 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:18:18.590758    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/switch-graph-bb639-658b9f69db-rwxvz"]
Aug 01 00:18:18.593336 ip-10-0-136-108 kubenswrapper[2578]: W0801 00:18:18.593307    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod141f7de1_1db2_49fa_89d9_5cde41749e2b.slice/crio-bc34b8b0f6401ae2f5b7d36d13c5289681e0408df9df79124c997f4fc928aaf1 WatchSource:0}: Error finding container bc34b8b0f6401ae2f5b7d36d13c5289681e0408df9df79124c997f4fc928aaf1: Status 404 returned error can't find the container with id bc34b8b0f6401ae2f5b7d36d13c5289681e0408df9df79124c997f4fc928aaf1
Aug 01 00:18:19.528664 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:18:19.528625    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/switch-graph-bb639-658b9f69db-rwxvz" event={"ID":"141f7de1-1db2-49fa-89d9-5cde41749e2b","Type":"ContainerStarted","Data":"732af120ffb7cb84ea2cab0b1f6cb7e08fffbf4ef09b144789e1246ff78d36d5"}
Aug 01 00:18:19.528894 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:18:19.528780    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/switch-graph-bb639-658b9f69db-rwxvz"
Aug 01 00:18:19.528894 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:18:19.528801    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/switch-graph-bb639-658b9f69db-rwxvz" event={"ID":"141f7de1-1db2-49fa-89d9-5cde41749e2b","Type":"ContainerStarted","Data":"bc34b8b0f6401ae2f5b7d36d13c5289681e0408df9df79124c997f4fc928aaf1"}
Aug 01 00:18:19.543447 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:18:19.543392    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/switch-graph-bb639-658b9f69db-rwxvz" podStartSLOduration=2.543374238 podStartE2EDuration="2.543374238s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-01 00:18:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-01 00:18:19.542634333 +0000 UTC m=+1923.510385096" watchObservedRunningTime="2026-08-01 00:18:19.543374238 +0000 UTC m=+1923.511125040"
Aug 01 00:18:25.539637 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:18:25.539607    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/switch-graph-bb639-658b9f69db-rwxvz"
Aug 01 00:18:26.337122 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:18:26.337093    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-2f013-predictor-bb647f765-5tkvs"
Aug 01 00:18:32.754302 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:18:32.754269    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gttdm_b32e6d42-e4e0-4113-b8ce-f5d5c5dc13f7/konnectivity-agent/0.log" limit=1024
Aug 01 00:18:38.096206 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:18:38.096154    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/splitter-graph-2f013-5f984b6955-7x5nv"]
Aug 01 00:18:38.099777 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:18:38.099755    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/splitter-graph-2f013-5f984b6955-7x5nv"
Aug 01 00:18:38.101521 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:18:38.101490    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"splitter-graph-2f013-kube-rbac-proxy-sar-config\""
Aug 01 00:18:38.101634 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:18:38.101587    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"splitter-graph-2f013-serving-cert\""
Aug 01 00:18:38.106591 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:18:38.106559    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/splitter-graph-2f013-5f984b6955-7x5nv"]
Aug 01 00:18:38.106712 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:18:38.106609    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gttdm_b32e6d42-e4e0-4113-b8ce-f5d5c5dc13f7/konnectivity-agent/0.log" limit=1024
Aug 01 00:18:38.157780 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:18:38.157742    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3061cbb5-a5f7-4911-8b37-d33a4dc48f7b-openshift-service-ca-bundle\") pod \"splitter-graph-2f013-5f984b6955-7x5nv\" (UID: \"3061cbb5-a5f7-4911-8b37-d33a4dc48f7b\") " pod="kserve-ci-e2e-test/splitter-graph-2f013-5f984b6955-7x5nv"
Aug 01 00:18:38.157949 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:18:38.157805    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3061cbb5-a5f7-4911-8b37-d33a4dc48f7b-proxy-tls\") pod \"splitter-graph-2f013-5f984b6955-7x5nv\" (UID: \"3061cbb5-a5f7-4911-8b37-d33a4dc48f7b\") " pod="kserve-ci-e2e-test/splitter-graph-2f013-5f984b6955-7x5nv"
Aug 01 00:18:38.259134 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:18:38.259099    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3061cbb5-a5f7-4911-8b37-d33a4dc48f7b-openshift-service-ca-bundle\") pod \"splitter-graph-2f013-5f984b6955-7x5nv\" (UID: \"3061cbb5-a5f7-4911-8b37-d33a4dc48f7b\") " pod="kserve-ci-e2e-test/splitter-graph-2f013-5f984b6955-7x5nv"
Aug 01 00:18:38.259331 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:18:38.259145    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3061cbb5-a5f7-4911-8b37-d33a4dc48f7b-proxy-tls\") pod \"splitter-graph-2f013-5f984b6955-7x5nv\" (UID: \"3061cbb5-a5f7-4911-8b37-d33a4dc48f7b\") " pod="kserve-ci-e2e-test/splitter-graph-2f013-5f984b6955-7x5nv"
Aug 01 00:18:38.259331 ip-10-0-136-108 kubenswrapper[2578]: E0801 00:18:38.259285    2578 secret.go:189] Couldn't get secret kserve-ci-e2e-test/splitter-graph-2f013-serving-cert: secret "splitter-graph-2f013-serving-cert" not found
Aug 01 00:18:38.259405 ip-10-0-136-108 kubenswrapper[2578]: E0801 00:18:38.259354    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3061cbb5-a5f7-4911-8b37-d33a4dc48f7b-proxy-tls podName:3061cbb5-a5f7-4911-8b37-d33a4dc48f7b nodeName:}" failed. No retries permitted until 2026-08-01 00:18:38.759332391 +0000 UTC m=+1942.727083142 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/3061cbb5-a5f7-4911-8b37-d33a4dc48f7b-proxy-tls") pod "splitter-graph-2f013-5f984b6955-7x5nv" (UID: "3061cbb5-a5f7-4911-8b37-d33a4dc48f7b") : secret "splitter-graph-2f013-serving-cert" not found
Aug 01 00:18:38.259867 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:18:38.259846    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3061cbb5-a5f7-4911-8b37-d33a4dc48f7b-openshift-service-ca-bundle\") pod \"splitter-graph-2f013-5f984b6955-7x5nv\" (UID: \"3061cbb5-a5f7-4911-8b37-d33a4dc48f7b\") " pod="kserve-ci-e2e-test/splitter-graph-2f013-5f984b6955-7x5nv"
Aug 01 00:18:38.608272 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:18:38.608235    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gttdm_b32e6d42-e4e0-4113-b8ce-f5d5c5dc13f7/konnectivity-agent/0.log" limit=1024
Aug 01 00:18:38.763417 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:18:38.763378    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3061cbb5-a5f7-4911-8b37-d33a4dc48f7b-proxy-tls\") pod \"splitter-graph-2f013-5f984b6955-7x5nv\" (UID: \"3061cbb5-a5f7-4911-8b37-d33a4dc48f7b\") " pod="kserve-ci-e2e-test/splitter-graph-2f013-5f984b6955-7x5nv"
Aug 01 00:18:38.765821 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:18:38.765799    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3061cbb5-a5f7-4911-8b37-d33a4dc48f7b-proxy-tls\") pod \"splitter-graph-2f013-5f984b6955-7x5nv\" (UID: \"3061cbb5-a5f7-4911-8b37-d33a4dc48f7b\") " pod="kserve-ci-e2e-test/splitter-graph-2f013-5f984b6955-7x5nv"
Aug 01 00:18:39.011777 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:18:39.011746    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/splitter-graph-2f013-5f984b6955-7x5nv"
Aug 01 00:18:39.140393 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:18:39.140369    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/splitter-graph-2f013-5f984b6955-7x5nv"]
Aug 01 00:18:39.142696 ip-10-0-136-108 kubenswrapper[2578]: W0801 00:18:39.142669    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3061cbb5_a5f7_4911_8b37_d33a4dc48f7b.slice/crio-e6f88c44a6ddf4863d09bac9435654fab4da30192d5ac60c1e39653607eb2119 WatchSource:0}: Error finding container e6f88c44a6ddf4863d09bac9435654fab4da30192d5ac60c1e39653607eb2119: Status 404 returned error can't find the container with id e6f88c44a6ddf4863d09bac9435654fab4da30192d5ac60c1e39653607eb2119
Aug 01 00:18:39.622206 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:18:39.622156    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/splitter-graph-2f013-5f984b6955-7x5nv" event={"ID":"3061cbb5-a5f7-4911-8b37-d33a4dc48f7b","Type":"ContainerStarted","Data":"ecd16b4f4ba7b26835ba169829adaead32003533686058e7ae3266732ee4a3ae"}
Aug 01 00:18:39.622360 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:18:39.622227    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/splitter-graph-2f013-5f984b6955-7x5nv" event={"ID":"3061cbb5-a5f7-4911-8b37-d33a4dc48f7b","Type":"ContainerStarted","Data":"e6f88c44a6ddf4863d09bac9435654fab4da30192d5ac60c1e39653607eb2119"}
Aug 01 00:18:39.622360 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:18:39.622312    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/splitter-graph-2f013-5f984b6955-7x5nv"
Aug 01 00:18:39.637698 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:18:39.637648    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/splitter-graph-2f013-5f984b6955-7x5nv" podStartSLOduration=1.637631937 podStartE2EDuration="1.637631937s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-01 00:18:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-01 00:18:39.63589289 +0000 UTC m=+1943.603643652" watchObservedRunningTime="2026-08-01 00:18:39.637631937 +0000 UTC m=+1943.605382720"
Aug 01 00:18:45.633233 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:18:45.633202    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/splitter-graph-2f013-5f984b6955-7x5nv"
Aug 01 00:19:23.769210 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:19:23.769117    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gttdm_b32e6d42-e4e0-4113-b8ce-f5d5c5dc13f7/konnectivity-agent/0.log" limit=1024
Aug 01 00:19:23.819401 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:19:23.819353    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gttdm_b32e6d42-e4e0-4113-b8ce-f5d5c5dc13f7/konnectivity-agent/0.log" limit=1024
Aug 01 00:24:35.788115 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:24:35.788074    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gttdm_b32e6d42-e4e0-4113-b8ce-f5d5c5dc13f7/konnectivity-agent/0.log" limit=1024
Aug 01 00:25:12.553397 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:25:12.553315    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gttdm_b32e6d42-e4e0-4113-b8ce-f5d5c5dc13f7/konnectivity-agent/0.log" limit=1024
Aug 01 00:26:52.773817 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:26:52.773782    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/splitter-graph-2f013-5f984b6955-7x5nv"]
Aug 01 00:26:52.774367 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:26:52.773989    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/splitter-graph-2f013-5f984b6955-7x5nv" podUID="3061cbb5-a5f7-4911-8b37-d33a4dc48f7b" containerName="splitter-graph-2f013" containerID="cri-o://ecd16b4f4ba7b26835ba169829adaead32003533686058e7ae3266732ee4a3ae" gracePeriod=30
Aug 01 00:26:52.823705 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:26:52.823667    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gttdm_b32e6d42-e4e0-4113-b8ce-f5d5c5dc13f7/konnectivity-agent/0.log" limit=1024
Aug 01 00:26:52.902192 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:26:52.902138    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-2f013-predictor-bb647f765-5tkvs"]
Aug 01 00:26:52.902485 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:26:52.902458    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-2f013-predictor-bb647f765-5tkvs" podUID="915a3a14-b2de-4ab3-ac4a-5215dcecab1f" containerName="kserve-container" containerID="cri-o://84d623cc225370371f0f4157b5af802f8f0ac52b03ab8d64ebf711397cee823b" gracePeriod=30
Aug 01 00:26:52.902577 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:26:52.902520    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-2f013-predictor-bb647f765-5tkvs" podUID="915a3a14-b2de-4ab3-ac4a-5215dcecab1f" containerName="kube-rbac-proxy" containerID="cri-o://ef5dd385f3b1bb6a1b0bcf09a4ca20bed9b6dd7605504962d19eb40742facddb" gracePeriod=30
Aug 01 00:26:52.934908 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:26:52.934880    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gttdm_b32e6d42-e4e0-4113-b8ce-f5d5c5dc13f7/konnectivity-agent/0.log" limit=1024
Aug 01 00:26:52.994213 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:26:52.994154    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gttdm_b32e6d42-e4e0-4113-b8ce-f5d5c5dc13f7/konnectivity-agent/0.log" limit=1024
Aug 01 00:26:53.908685 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:26:53.908653    2578 generic.go:356] "Generic (PLEG): container finished" podID="915a3a14-b2de-4ab3-ac4a-5215dcecab1f" containerID="ef5dd385f3b1bb6a1b0bcf09a4ca20bed9b6dd7605504962d19eb40742facddb" exitCode=2
Aug 01 00:26:53.909076 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:26:53.908726    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-2f013-predictor-bb647f765-5tkvs" event={"ID":"915a3a14-b2de-4ab3-ac4a-5215dcecab1f","Type":"ContainerDied","Data":"ef5dd385f3b1bb6a1b0bcf09a4ca20bed9b6dd7605504962d19eb40742facddb"}
Aug 01 00:26:55.630860 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:26:55.630820    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/splitter-graph-2f013-5f984b6955-7x5nv" podUID="3061cbb5-a5f7-4911-8b37-d33a4dc48f7b" containerName="splitter-graph-2f013" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Aug 01 00:26:56.138711 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:26:56.138686    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-2f013-predictor-bb647f765-5tkvs"
Aug 01 00:26:56.238858 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:26:56.238828    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qzvjb\" (UniqueName: \"kubernetes.io/projected/915a3a14-b2de-4ab3-ac4a-5215dcecab1f-kube-api-access-qzvjb\") pod \"915a3a14-b2de-4ab3-ac4a-5215dcecab1f\" (UID: \"915a3a14-b2de-4ab3-ac4a-5215dcecab1f\") "
Aug 01 00:26:56.239048 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:26:56.238894    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/915a3a14-b2de-4ab3-ac4a-5215dcecab1f-proxy-tls\") pod \"915a3a14-b2de-4ab3-ac4a-5215dcecab1f\" (UID: \"915a3a14-b2de-4ab3-ac4a-5215dcecab1f\") "
Aug 01 00:26:56.239048 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:26:56.238959    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/915a3a14-b2de-4ab3-ac4a-5215dcecab1f-kube-rbac-proxy-sar-config\") pod \"915a3a14-b2de-4ab3-ac4a-5215dcecab1f\" (UID: \"915a3a14-b2de-4ab3-ac4a-5215dcecab1f\") "
Aug 01 00:26:56.239340 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:26:56.239313    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/915a3a14-b2de-4ab3-ac4a-5215dcecab1f-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "915a3a14-b2de-4ab3-ac4a-5215dcecab1f" (UID: "915a3a14-b2de-4ab3-ac4a-5215dcecab1f"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 01 00:26:56.240912 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:26:56.240888    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/915a3a14-b2de-4ab3-ac4a-5215dcecab1f-kube-api-access-qzvjb" (OuterVolumeSpecName: "kube-api-access-qzvjb") pod "915a3a14-b2de-4ab3-ac4a-5215dcecab1f" (UID: "915a3a14-b2de-4ab3-ac4a-5215dcecab1f"). InnerVolumeSpecName "kube-api-access-qzvjb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 01 00:26:56.240991 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:26:56.240888    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/915a3a14-b2de-4ab3-ac4a-5215dcecab1f-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "915a3a14-b2de-4ab3-ac4a-5215dcecab1f" (UID: "915a3a14-b2de-4ab3-ac4a-5215dcecab1f"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 01 00:26:56.339892 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:26:56.339854    2578 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/915a3a14-b2de-4ab3-ac4a-5215dcecab1f-proxy-tls\") on node \"ip-10-0-136-108.ec2.internal\" DevicePath \"\""
Aug 01 00:26:56.339892 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:26:56.339884    2578 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/915a3a14-b2de-4ab3-ac4a-5215dcecab1f-kube-rbac-proxy-sar-config\") on node \"ip-10-0-136-108.ec2.internal\" DevicePath \"\""
Aug 01 00:26:56.339892 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:26:56.339895    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qzvjb\" (UniqueName: \"kubernetes.io/projected/915a3a14-b2de-4ab3-ac4a-5215dcecab1f-kube-api-access-qzvjb\") on node \"ip-10-0-136-108.ec2.internal\" DevicePath \"\""
Aug 01 00:26:56.923437 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:26:56.923344    2578 generic.go:356] "Generic (PLEG): container finished" podID="915a3a14-b2de-4ab3-ac4a-5215dcecab1f" containerID="84d623cc225370371f0f4157b5af802f8f0ac52b03ab8d64ebf711397cee823b" exitCode=0
Aug 01 00:26:56.923437 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:26:56.923394    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-2f013-predictor-bb647f765-5tkvs"
Aug 01 00:26:56.923941 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:26:56.923431    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-2f013-predictor-bb647f765-5tkvs" event={"ID":"915a3a14-b2de-4ab3-ac4a-5215dcecab1f","Type":"ContainerDied","Data":"84d623cc225370371f0f4157b5af802f8f0ac52b03ab8d64ebf711397cee823b"}
Aug 01 00:26:56.923941 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:26:56.923505    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-2f013-predictor-bb647f765-5tkvs" event={"ID":"915a3a14-b2de-4ab3-ac4a-5215dcecab1f","Type":"ContainerDied","Data":"c2c29965a1bf81cd2268f89bf3fae2bc23a44fc79c83f8dafb740d0b656a81ff"}
Aug 01 00:26:56.923941 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:26:56.923556    2578 scope.go:117] "RemoveContainer" containerID="ef5dd385f3b1bb6a1b0bcf09a4ca20bed9b6dd7605504962d19eb40742facddb"
Aug 01 00:26:56.931941 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:26:56.931917    2578 scope.go:117] "RemoveContainer" containerID="84d623cc225370371f0f4157b5af802f8f0ac52b03ab8d64ebf711397cee823b"
Aug 01 00:26:56.939485 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:26:56.939460    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-2f013-predictor-bb647f765-5tkvs"]
Aug 01 00:26:56.941050 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:26:56.941031    2578 scope.go:117] "RemoveContainer" containerID="ef5dd385f3b1bb6a1b0bcf09a4ca20bed9b6dd7605504962d19eb40742facddb"
Aug 01 00:26:56.941345 ip-10-0-136-108 kubenswrapper[2578]: E0801 00:26:56.941318    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ef5dd385f3b1bb6a1b0bcf09a4ca20bed9b6dd7605504962d19eb40742facddb\": container with ID starting with ef5dd385f3b1bb6a1b0bcf09a4ca20bed9b6dd7605504962d19eb40742facddb not found: ID does not exist" containerID="ef5dd385f3b1bb6a1b0bcf09a4ca20bed9b6dd7605504962d19eb40742facddb"
Aug 01 00:26:56.941450 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:26:56.941355    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ef5dd385f3b1bb6a1b0bcf09a4ca20bed9b6dd7605504962d19eb40742facddb"} err="failed to get container status \"ef5dd385f3b1bb6a1b0bcf09a4ca20bed9b6dd7605504962d19eb40742facddb\": rpc error: code = NotFound desc = could not find container \"ef5dd385f3b1bb6a1b0bcf09a4ca20bed9b6dd7605504962d19eb40742facddb\": container with ID starting with ef5dd385f3b1bb6a1b0bcf09a4ca20bed9b6dd7605504962d19eb40742facddb not found: ID does not exist"
Aug 01 00:26:56.941450 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:26:56.941386    2578 scope.go:117] "RemoveContainer" containerID="84d623cc225370371f0f4157b5af802f8f0ac52b03ab8d64ebf711397cee823b"
Aug 01 00:26:56.941676 ip-10-0-136-108 kubenswrapper[2578]: E0801 00:26:56.941654    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"84d623cc225370371f0f4157b5af802f8f0ac52b03ab8d64ebf711397cee823b\": container with ID starting with 84d623cc225370371f0f4157b5af802f8f0ac52b03ab8d64ebf711397cee823b not found: ID does not exist" containerID="84d623cc225370371f0f4157b5af802f8f0ac52b03ab8d64ebf711397cee823b"
Aug 01 00:26:56.941726 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:26:56.941685    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"84d623cc225370371f0f4157b5af802f8f0ac52b03ab8d64ebf711397cee823b"} err="failed to get container status \"84d623cc225370371f0f4157b5af802f8f0ac52b03ab8d64ebf711397cee823b\": rpc error: code = NotFound desc = could not find container \"84d623cc225370371f0f4157b5af802f8f0ac52b03ab8d64ebf711397cee823b\": container with ID starting with 84d623cc225370371f0f4157b5af802f8f0ac52b03ab8d64ebf711397cee823b not found: ID does not exist"
Aug 01 00:26:56.942326 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:26:56.942309    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-2f013-predictor-bb647f765-5tkvs"]
Aug 01 00:26:58.489032 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:26:58.488994    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="915a3a14-b2de-4ab3-ac4a-5215dcecab1f" path="/var/lib/kubelet/pods/915a3a14-b2de-4ab3-ac4a-5215dcecab1f/volumes"
Aug 01 00:27:00.630746 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:27:00.630708    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/splitter-graph-2f013-5f984b6955-7x5nv" podUID="3061cbb5-a5f7-4911-8b37-d33a4dc48f7b" containerName="splitter-graph-2f013" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Aug 01 00:27:05.630981 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:27:05.630930    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/splitter-graph-2f013-5f984b6955-7x5nv" podUID="3061cbb5-a5f7-4911-8b37-d33a4dc48f7b" containerName="splitter-graph-2f013" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Aug 01 00:27:05.631388 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:27:05.631049    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/splitter-graph-2f013-5f984b6955-7x5nv"
Aug 01 00:27:10.631235 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:27:10.631199    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/splitter-graph-2f013-5f984b6955-7x5nv" podUID="3061cbb5-a5f7-4911-8b37-d33a4dc48f7b" containerName="splitter-graph-2f013" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Aug 01 00:27:15.630489 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:27:15.630446    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/splitter-graph-2f013-5f984b6955-7x5nv" podUID="3061cbb5-a5f7-4911-8b37-d33a4dc48f7b" containerName="splitter-graph-2f013" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Aug 01 00:27:20.630804 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:27:20.630762    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/splitter-graph-2f013-5f984b6955-7x5nv" podUID="3061cbb5-a5f7-4911-8b37-d33a4dc48f7b" containerName="splitter-graph-2f013" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Aug 01 00:27:23.046650 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:27:23.046559    2578 generic.go:356] "Generic (PLEG): container finished" podID="3061cbb5-a5f7-4911-8b37-d33a4dc48f7b" containerID="ecd16b4f4ba7b26835ba169829adaead32003533686058e7ae3266732ee4a3ae" exitCode=0
Aug 01 00:27:23.046650 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:27:23.046627    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/splitter-graph-2f013-5f984b6955-7x5nv" event={"ID":"3061cbb5-a5f7-4911-8b37-d33a4dc48f7b","Type":"ContainerDied","Data":"ecd16b4f4ba7b26835ba169829adaead32003533686058e7ae3266732ee4a3ae"}
Aug 01 00:27:23.420416 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:27:23.420388    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/splitter-graph-2f013-5f984b6955-7x5nv"
Aug 01 00:27:23.583963 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:27:23.583930    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3061cbb5-a5f7-4911-8b37-d33a4dc48f7b-proxy-tls\") pod \"3061cbb5-a5f7-4911-8b37-d33a4dc48f7b\" (UID: \"3061cbb5-a5f7-4911-8b37-d33a4dc48f7b\") "
Aug 01 00:27:23.584140 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:27:23.584031    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3061cbb5-a5f7-4911-8b37-d33a4dc48f7b-openshift-service-ca-bundle\") pod \"3061cbb5-a5f7-4911-8b37-d33a4dc48f7b\" (UID: \"3061cbb5-a5f7-4911-8b37-d33a4dc48f7b\") "
Aug 01 00:27:23.584396 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:27:23.584372    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3061cbb5-a5f7-4911-8b37-d33a4dc48f7b-openshift-service-ca-bundle" (OuterVolumeSpecName: "openshift-service-ca-bundle") pod "3061cbb5-a5f7-4911-8b37-d33a4dc48f7b" (UID: "3061cbb5-a5f7-4911-8b37-d33a4dc48f7b"). InnerVolumeSpecName "openshift-service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 01 00:27:23.585990 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:27:23.585969    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3061cbb5-a5f7-4911-8b37-d33a4dc48f7b-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "3061cbb5-a5f7-4911-8b37-d33a4dc48f7b" (UID: "3061cbb5-a5f7-4911-8b37-d33a4dc48f7b"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 01 00:27:23.685326 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:27:23.685237    2578 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3061cbb5-a5f7-4911-8b37-d33a4dc48f7b-proxy-tls\") on node \"ip-10-0-136-108.ec2.internal\" DevicePath \"\""
Aug 01 00:27:23.685326 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:27:23.685265    2578 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3061cbb5-a5f7-4911-8b37-d33a4dc48f7b-openshift-service-ca-bundle\") on node \"ip-10-0-136-108.ec2.internal\" DevicePath \"\""
Aug 01 00:27:24.052468 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:27:24.052437    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/splitter-graph-2f013-5f984b6955-7x5nv"
Aug 01 00:27:24.052958 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:27:24.052485    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/splitter-graph-2f013-5f984b6955-7x5nv" event={"ID":"3061cbb5-a5f7-4911-8b37-d33a4dc48f7b","Type":"ContainerDied","Data":"e6f88c44a6ddf4863d09bac9435654fab4da30192d5ac60c1e39653607eb2119"}
Aug 01 00:27:24.052958 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:27:24.052555    2578 scope.go:117] "RemoveContainer" containerID="ecd16b4f4ba7b26835ba169829adaead32003533686058e7ae3266732ee4a3ae"
Aug 01 00:27:24.070940 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:27:24.070916    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/splitter-graph-2f013-5f984b6955-7x5nv"]
Aug 01 00:27:24.074114 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:27:24.074092    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/splitter-graph-2f013-5f984b6955-7x5nv"]
Aug 01 00:27:24.489901 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:27:24.489857    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3061cbb5-a5f7-4911-8b37-d33a4dc48f7b" path="/var/lib/kubelet/pods/3061cbb5-a5f7-4911-8b37-d33a4dc48f7b/volumes"
Aug 01 00:30:58.810951 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:30:58.810920    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gttdm_b32e6d42-e4e0-4113-b8ce-f5d5c5dc13f7/konnectivity-agent/0.log" limit=1024
Aug 01 00:34:15.285147 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:34:15.285042    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gttdm_b32e6d42-e4e0-4113-b8ce-f5d5c5dc13f7/konnectivity-agent/0.log" limit=1024
Aug 01 00:34:36.860365 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:34:36.860330    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/switch-graph-bb639-658b9f69db-rwxvz"]
Aug 01 00:34:36.860828 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:34:36.860591    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/switch-graph-bb639-658b9f69db-rwxvz" podUID="141f7de1-1db2-49fa-89d9-5cde41749e2b" containerName="switch-graph-bb639" containerID="cri-o://732af120ffb7cb84ea2cab0b1f6cb7e08fffbf4ef09b144789e1246ff78d36d5" gracePeriod=30
Aug 01 00:34:36.878698 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:34:36.878667    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gttdm_b32e6d42-e4e0-4113-b8ce-f5d5c5dc13f7/konnectivity-agent/0.log" limit=1024
Aug 01 00:34:37.008767 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:34:37.008732    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-bb639-predictor-7fccb6f469-zdmj7"]
Aug 01 00:34:37.009059 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:34:37.009014    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-bb639-predictor-7fccb6f469-zdmj7" podUID="c3f7ef2d-1ce3-4f69-9660-f559ac064ea7" containerName="kserve-container" containerID="cri-o://944cfc39854c8c09a42a0269e084482da2c2390c72882af7cce2c9beb0810971" gracePeriod=30
Aug 01 00:34:37.009226 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:34:37.009082    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-bb639-predictor-7fccb6f469-zdmj7" podUID="c3f7ef2d-1ce3-4f69-9660-f559ac064ea7" containerName="kube-rbac-proxy" containerID="cri-o://49679d1de7ea7d50e52c0f1105469f895bf6c6bdc2b9aaf8701ee0cdf24af147" gracePeriod=30
Aug 01 00:34:37.029964 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:34:37.029941    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gttdm_b32e6d42-e4e0-4113-b8ce-f5d5c5dc13f7/konnectivity-agent/0.log" limit=1024
Aug 01 00:34:37.102339 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:34:37.102307    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gttdm_b32e6d42-e4e0-4113-b8ce-f5d5c5dc13f7/konnectivity-agent/0.log" limit=1024
Aug 01 00:34:38.033902 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:34:38.033861    2578 generic.go:356] "Generic (PLEG): container finished" podID="c3f7ef2d-1ce3-4f69-9660-f559ac064ea7" containerID="49679d1de7ea7d50e52c0f1105469f895bf6c6bdc2b9aaf8701ee0cdf24af147" exitCode=2
Aug 01 00:34:38.034307 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:34:38.033928    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-bb639-predictor-7fccb6f469-zdmj7" event={"ID":"c3f7ef2d-1ce3-4f69-9660-f559ac064ea7","Type":"ContainerDied","Data":"49679d1de7ea7d50e52c0f1105469f895bf6c6bdc2b9aaf8701ee0cdf24af147"}
Aug 01 00:34:38.604976 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:34:38.604945    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gttdm_b32e6d42-e4e0-4113-b8ce-f5d5c5dc13f7/konnectivity-agent/0.log" limit=1024
Aug 01 00:34:38.705198 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:34:38.705135    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gttdm_b32e6d42-e4e0-4113-b8ce-f5d5c5dc13f7/konnectivity-agent/0.log" limit=1024
Aug 01 00:34:40.355571 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:34:40.355542    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-bb639-predictor-7fccb6f469-zdmj7"
Aug 01 00:34:40.455868 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:34:40.455765    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7hv5q\" (UniqueName: \"kubernetes.io/projected/c3f7ef2d-1ce3-4f69-9660-f559ac064ea7-kube-api-access-7hv5q\") pod \"c3f7ef2d-1ce3-4f69-9660-f559ac064ea7\" (UID: \"c3f7ef2d-1ce3-4f69-9660-f559ac064ea7\") "
Aug 01 00:34:40.455868 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:34:40.455826    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c3f7ef2d-1ce3-4f69-9660-f559ac064ea7-kube-rbac-proxy-sar-config\") pod \"c3f7ef2d-1ce3-4f69-9660-f559ac064ea7\" (UID: \"c3f7ef2d-1ce3-4f69-9660-f559ac064ea7\") "
Aug 01 00:34:40.455868 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:34:40.455861    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c3f7ef2d-1ce3-4f69-9660-f559ac064ea7-proxy-tls\") pod \"c3f7ef2d-1ce3-4f69-9660-f559ac064ea7\" (UID: \"c3f7ef2d-1ce3-4f69-9660-f559ac064ea7\") "
Aug 01 00:34:40.456302 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:34:40.456276    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c3f7ef2d-1ce3-4f69-9660-f559ac064ea7-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "c3f7ef2d-1ce3-4f69-9660-f559ac064ea7" (UID: "c3f7ef2d-1ce3-4f69-9660-f559ac064ea7"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 01 00:34:40.457996 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:34:40.457960    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c3f7ef2d-1ce3-4f69-9660-f559ac064ea7-kube-api-access-7hv5q" (OuterVolumeSpecName: "kube-api-access-7hv5q") pod "c3f7ef2d-1ce3-4f69-9660-f559ac064ea7" (UID: "c3f7ef2d-1ce3-4f69-9660-f559ac064ea7"). InnerVolumeSpecName "kube-api-access-7hv5q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 01 00:34:40.457996 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:34:40.457982    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c3f7ef2d-1ce3-4f69-9660-f559ac064ea7-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "c3f7ef2d-1ce3-4f69-9660-f559ac064ea7" (UID: "c3f7ef2d-1ce3-4f69-9660-f559ac064ea7"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 01 00:34:40.538220 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:34:40.538166    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-bb639-658b9f69db-rwxvz" podUID="141f7de1-1db2-49fa-89d9-5cde41749e2b" containerName="switch-graph-bb639" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Aug 01 00:34:40.557284 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:34:40.557255    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7hv5q\" (UniqueName: \"kubernetes.io/projected/c3f7ef2d-1ce3-4f69-9660-f559ac064ea7-kube-api-access-7hv5q\") on node \"ip-10-0-136-108.ec2.internal\" DevicePath \"\""
Aug 01 00:34:40.557284 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:34:40.557278    2578 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c3f7ef2d-1ce3-4f69-9660-f559ac064ea7-kube-rbac-proxy-sar-config\") on node \"ip-10-0-136-108.ec2.internal\" DevicePath \"\""
Aug 01 00:34:40.557284 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:34:40.557288    2578 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c3f7ef2d-1ce3-4f69-9660-f559ac064ea7-proxy-tls\") on node \"ip-10-0-136-108.ec2.internal\" DevicePath \"\""
Aug 01 00:34:41.049368 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:34:41.049330    2578 generic.go:356] "Generic (PLEG): container finished" podID="c3f7ef2d-1ce3-4f69-9660-f559ac064ea7" containerID="944cfc39854c8c09a42a0269e084482da2c2390c72882af7cce2c9beb0810971" exitCode=0
Aug 01 00:34:41.049554 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:34:41.049385    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-bb639-predictor-7fccb6f469-zdmj7"
Aug 01 00:34:41.049554 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:34:41.049404    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-bb639-predictor-7fccb6f469-zdmj7" event={"ID":"c3f7ef2d-1ce3-4f69-9660-f559ac064ea7","Type":"ContainerDied","Data":"944cfc39854c8c09a42a0269e084482da2c2390c72882af7cce2c9beb0810971"}
Aug 01 00:34:41.049554 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:34:41.049475    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-bb639-predictor-7fccb6f469-zdmj7" event={"ID":"c3f7ef2d-1ce3-4f69-9660-f559ac064ea7","Type":"ContainerDied","Data":"1ace02dd0d68ef2b73fbae6c4c99525578a6eb54fa3bca8bd08517a46cf54e0d"}
Aug 01 00:34:41.049554 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:34:41.049503    2578 scope.go:117] "RemoveContainer" containerID="49679d1de7ea7d50e52c0f1105469f895bf6c6bdc2b9aaf8701ee0cdf24af147"
Aug 01 00:34:41.057968 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:34:41.057945    2578 scope.go:117] "RemoveContainer" containerID="944cfc39854c8c09a42a0269e084482da2c2390c72882af7cce2c9beb0810971"
Aug 01 00:34:41.065231 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:34:41.065204    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-bb639-predictor-7fccb6f469-zdmj7"]
Aug 01 00:34:41.066488 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:34:41.066467    2578 scope.go:117] "RemoveContainer" containerID="49679d1de7ea7d50e52c0f1105469f895bf6c6bdc2b9aaf8701ee0cdf24af147"
Aug 01 00:34:41.066768 ip-10-0-136-108 kubenswrapper[2578]: E0801 00:34:41.066747    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"49679d1de7ea7d50e52c0f1105469f895bf6c6bdc2b9aaf8701ee0cdf24af147\": container with ID starting with 49679d1de7ea7d50e52c0f1105469f895bf6c6bdc2b9aaf8701ee0cdf24af147 not found: ID does not exist" containerID="49679d1de7ea7d50e52c0f1105469f895bf6c6bdc2b9aaf8701ee0cdf24af147"
Aug 01 00:34:41.066831 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:34:41.066777    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"49679d1de7ea7d50e52c0f1105469f895bf6c6bdc2b9aaf8701ee0cdf24af147"} err="failed to get container status \"49679d1de7ea7d50e52c0f1105469f895bf6c6bdc2b9aaf8701ee0cdf24af147\": rpc error: code = NotFound desc = could not find container \"49679d1de7ea7d50e52c0f1105469f895bf6c6bdc2b9aaf8701ee0cdf24af147\": container with ID starting with 49679d1de7ea7d50e52c0f1105469f895bf6c6bdc2b9aaf8701ee0cdf24af147 not found: ID does not exist"
Aug 01 00:34:41.066831 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:34:41.066796    2578 scope.go:117] "RemoveContainer" containerID="944cfc39854c8c09a42a0269e084482da2c2390c72882af7cce2c9beb0810971"
Aug 01 00:34:41.067047 ip-10-0-136-108 kubenswrapper[2578]: E0801 00:34:41.067031    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"944cfc39854c8c09a42a0269e084482da2c2390c72882af7cce2c9beb0810971\": container with ID starting with 944cfc39854c8c09a42a0269e084482da2c2390c72882af7cce2c9beb0810971 not found: ID does not exist" containerID="944cfc39854c8c09a42a0269e084482da2c2390c72882af7cce2c9beb0810971"
Aug 01 00:34:41.067085 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:34:41.067050    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"944cfc39854c8c09a42a0269e084482da2c2390c72882af7cce2c9beb0810971"} err="failed to get container status \"944cfc39854c8c09a42a0269e084482da2c2390c72882af7cce2c9beb0810971\": rpc error: code = NotFound desc = could not find container \"944cfc39854c8c09a42a0269e084482da2c2390c72882af7cce2c9beb0810971\": container with ID starting with 944cfc39854c8c09a42a0269e084482da2c2390c72882af7cce2c9beb0810971 not found: ID does not exist"
Aug 01 00:34:41.071039 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:34:41.071019    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-bb639-predictor-7fccb6f469-zdmj7"]
Aug 01 00:34:42.488713 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:34:42.488679    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c3f7ef2d-1ce3-4f69-9660-f559ac064ea7" path="/var/lib/kubelet/pods/c3f7ef2d-1ce3-4f69-9660-f559ac064ea7/volumes"
Aug 01 00:34:45.537734 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:34:45.537696    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-bb639-658b9f69db-rwxvz" podUID="141f7de1-1db2-49fa-89d9-5cde41749e2b" containerName="switch-graph-bb639" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Aug 01 00:34:50.538664 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:34:50.538626    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-bb639-658b9f69db-rwxvz" podUID="141f7de1-1db2-49fa-89d9-5cde41749e2b" containerName="switch-graph-bb639" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Aug 01 00:34:50.539120 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:34:50.538737    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/switch-graph-bb639-658b9f69db-rwxvz"
Aug 01 00:34:54.414187 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:34:54.414153    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_switch-graph-bb639-658b9f69db-rwxvz_141f7de1-1db2-49fa-89d9-5cde41749e2b/switch-graph-bb639/0.log"
Aug 01 00:34:55.205116 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:34:55.205088    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_switch-graph-bb639-658b9f69db-rwxvz_141f7de1-1db2-49fa-89d9-5cde41749e2b/switch-graph-bb639/0.log"
Aug 01 00:34:55.538408 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:34:55.538366    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-bb639-658b9f69db-rwxvz" podUID="141f7de1-1db2-49fa-89d9-5cde41749e2b" containerName="switch-graph-bb639" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Aug 01 00:34:55.997786 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:34:55.997758    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_switch-graph-bb639-658b9f69db-rwxvz_141f7de1-1db2-49fa-89d9-5cde41749e2b/switch-graph-bb639/0.log"
Aug 01 00:34:56.766832 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:34:56.766805    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_switch-graph-bb639-658b9f69db-rwxvz_141f7de1-1db2-49fa-89d9-5cde41749e2b/switch-graph-bb639/0.log"
Aug 01 00:34:57.539028 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:34:57.538996    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_switch-graph-bb639-658b9f69db-rwxvz_141f7de1-1db2-49fa-89d9-5cde41749e2b/switch-graph-bb639/0.log"
Aug 01 00:34:58.310893 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:34:58.310855    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_switch-graph-bb639-658b9f69db-rwxvz_141f7de1-1db2-49fa-89d9-5cde41749e2b/switch-graph-bb639/0.log"
Aug 01 00:34:59.074351 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:34:59.074319    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_switch-graph-bb639-658b9f69db-rwxvz_141f7de1-1db2-49fa-89d9-5cde41749e2b/switch-graph-bb639/0.log"
Aug 01 00:34:59.840108 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:34:59.840081    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_switch-graph-bb639-658b9f69db-rwxvz_141f7de1-1db2-49fa-89d9-5cde41749e2b/switch-graph-bb639/0.log"
Aug 01 00:35:00.538495 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:35:00.538452    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-bb639-658b9f69db-rwxvz" podUID="141f7de1-1db2-49fa-89d9-5cde41749e2b" containerName="switch-graph-bb639" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Aug 01 00:35:00.616601 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:35:00.616559    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_switch-graph-bb639-658b9f69db-rwxvz_141f7de1-1db2-49fa-89d9-5cde41749e2b/switch-graph-bb639/0.log"
Aug 01 00:35:01.414781 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:35:01.414755    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_switch-graph-bb639-658b9f69db-rwxvz_141f7de1-1db2-49fa-89d9-5cde41749e2b/switch-graph-bb639/0.log"
Aug 01 00:35:02.198193 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:35:02.198157    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_switch-graph-bb639-658b9f69db-rwxvz_141f7de1-1db2-49fa-89d9-5cde41749e2b/switch-graph-bb639/0.log"
Aug 01 00:35:02.990141 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:35:02.990118    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_switch-graph-bb639-658b9f69db-rwxvz_141f7de1-1db2-49fa-89d9-5cde41749e2b/switch-graph-bb639/0.log"
Aug 01 00:35:03.799081 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:35:03.799056    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_switch-graph-bb639-658b9f69db-rwxvz_141f7de1-1db2-49fa-89d9-5cde41749e2b/switch-graph-bb639/0.log"
Aug 01 00:35:04.609701 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:35:04.609675    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_switch-graph-bb639-658b9f69db-rwxvz_141f7de1-1db2-49fa-89d9-5cde41749e2b/switch-graph-bb639/0.log"
Aug 01 00:35:05.537823 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:35:05.537780    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-bb639-658b9f69db-rwxvz" podUID="141f7de1-1db2-49fa-89d9-5cde41749e2b" containerName="switch-graph-bb639" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Aug 01 00:35:07.013962 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:35:07.013938    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/switch-graph-bb639-658b9f69db-rwxvz"
Aug 01 00:35:07.069153 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:35:07.069120    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/141f7de1-1db2-49fa-89d9-5cde41749e2b-proxy-tls\") pod \"141f7de1-1db2-49fa-89d9-5cde41749e2b\" (UID: \"141f7de1-1db2-49fa-89d9-5cde41749e2b\") "
Aug 01 00:35:07.069153 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:35:07.069158    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/141f7de1-1db2-49fa-89d9-5cde41749e2b-openshift-service-ca-bundle\") pod \"141f7de1-1db2-49fa-89d9-5cde41749e2b\" (UID: \"141f7de1-1db2-49fa-89d9-5cde41749e2b\") "
Aug 01 00:35:07.069551 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:35:07.069517    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/141f7de1-1db2-49fa-89d9-5cde41749e2b-openshift-service-ca-bundle" (OuterVolumeSpecName: "openshift-service-ca-bundle") pod "141f7de1-1db2-49fa-89d9-5cde41749e2b" (UID: "141f7de1-1db2-49fa-89d9-5cde41749e2b"). InnerVolumeSpecName "openshift-service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 01 00:35:07.071203 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:35:07.071165    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/141f7de1-1db2-49fa-89d9-5cde41749e2b-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "141f7de1-1db2-49fa-89d9-5cde41749e2b" (UID: "141f7de1-1db2-49fa-89d9-5cde41749e2b"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 01 00:35:07.169786 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:35:07.169691    2578 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/141f7de1-1db2-49fa-89d9-5cde41749e2b-proxy-tls\") on node \"ip-10-0-136-108.ec2.internal\" DevicePath \"\""
Aug 01 00:35:07.169786 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:35:07.169729    2578 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/141f7de1-1db2-49fa-89d9-5cde41749e2b-openshift-service-ca-bundle\") on node \"ip-10-0-136-108.ec2.internal\" DevicePath \"\""
Aug 01 00:35:07.174002 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:35:07.173973    2578 generic.go:356] "Generic (PLEG): container finished" podID="141f7de1-1db2-49fa-89d9-5cde41749e2b" containerID="732af120ffb7cb84ea2cab0b1f6cb7e08fffbf4ef09b144789e1246ff78d36d5" exitCode=0
Aug 01 00:35:07.174159 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:35:07.174009    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/switch-graph-bb639-658b9f69db-rwxvz"
Aug 01 00:35:07.174159 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:35:07.174048    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/switch-graph-bb639-658b9f69db-rwxvz" event={"ID":"141f7de1-1db2-49fa-89d9-5cde41749e2b","Type":"ContainerDied","Data":"732af120ffb7cb84ea2cab0b1f6cb7e08fffbf4ef09b144789e1246ff78d36d5"}
Aug 01 00:35:07.174159 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:35:07.174104    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/switch-graph-bb639-658b9f69db-rwxvz" event={"ID":"141f7de1-1db2-49fa-89d9-5cde41749e2b","Type":"ContainerDied","Data":"bc34b8b0f6401ae2f5b7d36d13c5289681e0408df9df79124c997f4fc928aaf1"}
Aug 01 00:35:07.174159 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:35:07.174130    2578 scope.go:117] "RemoveContainer" containerID="732af120ffb7cb84ea2cab0b1f6cb7e08fffbf4ef09b144789e1246ff78d36d5"
Aug 01 00:35:07.183800 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:35:07.183783    2578 scope.go:117] "RemoveContainer" containerID="732af120ffb7cb84ea2cab0b1f6cb7e08fffbf4ef09b144789e1246ff78d36d5"
Aug 01 00:35:07.184042 ip-10-0-136-108 kubenswrapper[2578]: E0801 00:35:07.184024    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"732af120ffb7cb84ea2cab0b1f6cb7e08fffbf4ef09b144789e1246ff78d36d5\": container with ID starting with 732af120ffb7cb84ea2cab0b1f6cb7e08fffbf4ef09b144789e1246ff78d36d5 not found: ID does not exist" containerID="732af120ffb7cb84ea2cab0b1f6cb7e08fffbf4ef09b144789e1246ff78d36d5"
Aug 01 00:35:07.184097 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:35:07.184049    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"732af120ffb7cb84ea2cab0b1f6cb7e08fffbf4ef09b144789e1246ff78d36d5"} err="failed to get container status \"732af120ffb7cb84ea2cab0b1f6cb7e08fffbf4ef09b144789e1246ff78d36d5\": rpc error: code = NotFound desc = could not find container \"732af120ffb7cb84ea2cab0b1f6cb7e08fffbf4ef09b144789e1246ff78d36d5\": container with ID starting with 732af120ffb7cb84ea2cab0b1f6cb7e08fffbf4ef09b144789e1246ff78d36d5 not found: ID does not exist"
Aug 01 00:35:07.194394 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:35:07.194373    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/switch-graph-bb639-658b9f69db-rwxvz"]
Aug 01 00:35:07.200160 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:35:07.200139    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/switch-graph-bb639-658b9f69db-rwxvz"]
Aug 01 00:35:08.113884 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:35:08.113833    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gttdm_b32e6d42-e4e0-4113-b8ce-f5d5c5dc13f7/konnectivity-agent/0.log" limit=1024
Aug 01 00:35:08.284232 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:35:08.284202    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gttdm_b32e6d42-e4e0-4113-b8ce-f5d5c5dc13f7/konnectivity-agent/0.log" limit=1024
Aug 01 00:35:08.339703 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:35:08.339673    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gttdm_b32e6d42-e4e0-4113-b8ce-f5d5c5dc13f7/konnectivity-agent/0.log" limit=1024
Aug 01 00:35:08.489262 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:35:08.489233    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="141f7de1-1db2-49fa-89d9-5cde41749e2b" path="/var/lib/kubelet/pods/141f7de1-1db2-49fa-89d9-5cde41749e2b/volumes"
Aug 01 00:35:11.088477 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:35:11.088448    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-xlv68_9d113c99-25bd-49a4-b8e2-f5c46f3ecc02/global-pull-secret-syncer/0.log"
Aug 01 00:35:11.124285 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:35:11.124256    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-gttdm_b32e6d42-e4e0-4113-b8ce-f5d5c5dc13f7/konnectivity-agent/0.log"
Aug 01 00:35:11.230220 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:35:11.230195    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-136-108.ec2.internal_cc577a92ce0dd873599a996e54a72791/haproxy/0.log"
Aug 01 00:35:13.760435 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:35:13.760396    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gttdm_b32e6d42-e4e0-4113-b8ce-f5d5c5dc13f7/konnectivity-agent/0.log" limit=1024
Aug 01 00:35:13.815928 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:35:13.815875    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gttdm_b32e6d42-e4e0-4113-b8ce-f5d5c5dc13f7/konnectivity-agent/0.log" limit=1024
Aug 01 00:35:15.850403 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:35:15.850371    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_d3918f5e-491a-4456-9438-783cf36e291e/alertmanager/0.log"
Aug 01 00:35:15.872720 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:35:15.872684    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_d3918f5e-491a-4456-9438-783cf36e291e/config-reloader/0.log"
Aug 01 00:35:15.894551 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:35:15.894522    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_d3918f5e-491a-4456-9438-783cf36e291e/kube-rbac-proxy-web/0.log"
Aug 01 00:35:15.918282 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:35:15.918257    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_d3918f5e-491a-4456-9438-783cf36e291e/kube-rbac-proxy/0.log"
Aug 01 00:35:15.941045 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:35:15.941020    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_d3918f5e-491a-4456-9438-783cf36e291e/kube-rbac-proxy-metric/0.log"
Aug 01 00:35:15.977466 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:35:15.977426    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_d3918f5e-491a-4456-9438-783cf36e291e/prom-label-proxy/0.log"
Aug 01 00:35:16.026754 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:35:16.026728    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_d3918f5e-491a-4456-9438-783cf36e291e/init-config-reloader/0.log"
Aug 01 00:35:16.189467 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:35:16.189396    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-5b86cb7b86-269w6_8cec3464-a28f-45b2-bb1e-1464eb049d79/metrics-server/0.log"
Aug 01 00:35:16.216455 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:35:16.216427    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-6dffbbd67-lzqqq_8a2dfa37-6798-4b86-be3b-166e942af34d/monitoring-plugin/0.log"
Aug 01 00:35:16.402707 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:35:16.402675    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-wxg27_49955a37-a47a-48ee-826c-6aaa95715852/node-exporter/0.log"
Aug 01 00:35:16.422580 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:35:16.422553    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-wxg27_49955a37-a47a-48ee-826c-6aaa95715852/kube-rbac-proxy/0.log"
Aug 01 00:35:16.444286 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:35:16.444224    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-wxg27_49955a37-a47a-48ee-826c-6aaa95715852/init-textfile/0.log"
Aug 01 00:35:16.733321 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:35:16.733221    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-758d7cb874-knbql_c01686d9-cf94-4729-88fc-8c6894967f98/prometheus-operator/0.log"
Aug 01 00:35:16.751038 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:35:16.751015    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-758d7cb874-knbql_c01686d9-cf94-4729-88fc-8c6894967f98/kube-rbac-proxy/0.log"
Aug 01 00:35:16.804297 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:35:16.804273    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-697d9cdbc4-srjhx_cbbb8e62-6d70-43f7-8851-f29b45d31862/telemeter-client/0.log"
Aug 01 00:35:16.829101 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:35:16.829080    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-697d9cdbc4-srjhx_cbbb8e62-6d70-43f7-8851-f29b45d31862/reload/0.log"
Aug 01 00:35:16.849305 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:35:16.849285    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-697d9cdbc4-srjhx_cbbb8e62-6d70-43f7-8851-f29b45d31862/kube-rbac-proxy/0.log"
Aug 01 00:35:16.879805 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:35:16.879781    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-6fb556c5d9-8dt4b_a2d3ccdb-0424-404c-a0b6-d72f56fbbb03/thanos-query/0.log"
Aug 01 00:35:16.899806 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:35:16.899777    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-6fb556c5d9-8dt4b_a2d3ccdb-0424-404c-a0b6-d72f56fbbb03/kube-rbac-proxy-web/0.log"
Aug 01 00:35:16.925740 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:35:16.925718    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-6fb556c5d9-8dt4b_a2d3ccdb-0424-404c-a0b6-d72f56fbbb03/kube-rbac-proxy/0.log"
Aug 01 00:35:16.956008 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:35:16.955986    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-6fb556c5d9-8dt4b_a2d3ccdb-0424-404c-a0b6-d72f56fbbb03/prom-label-proxy/0.log"
Aug 01 00:35:16.988663 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:35:16.988642    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-6fb556c5d9-8dt4b_a2d3ccdb-0424-404c-a0b6-d72f56fbbb03/kube-rbac-proxy-rules/0.log"
Aug 01 00:35:17.013016 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:35:17.012995    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-6fb556c5d9-8dt4b_a2d3ccdb-0424-404c-a0b6-d72f56fbbb03/kube-rbac-proxy-metrics/0.log"
Aug 01 00:35:19.164541 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:35:19.164513    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-66d866d495-nhllr_2e86968e-4498-462f-baa8-da63b80d8ae7/console/0.log"
Aug 01 00:35:19.601546 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:35:19.601512    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-pmldc/perf-node-gather-daemonset-t6vz9"]
Aug 01 00:35:19.601886 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:35:19.601873    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="915a3a14-b2de-4ab3-ac4a-5215dcecab1f" containerName="kube-rbac-proxy"
Aug 01 00:35:19.601931 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:35:19.601888    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="915a3a14-b2de-4ab3-ac4a-5215dcecab1f" containerName="kube-rbac-proxy"
Aug 01 00:35:19.601931 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:35:19.601898    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="141f7de1-1db2-49fa-89d9-5cde41749e2b" containerName="switch-graph-bb639"
Aug 01 00:35:19.601931 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:35:19.601903    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="141f7de1-1db2-49fa-89d9-5cde41749e2b" containerName="switch-graph-bb639"
Aug 01 00:35:19.601931 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:35:19.601912    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c3f7ef2d-1ce3-4f69-9660-f559ac064ea7" containerName="kube-rbac-proxy"
Aug 01 00:35:19.601931 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:35:19.601919    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="c3f7ef2d-1ce3-4f69-9660-f559ac064ea7" containerName="kube-rbac-proxy"
Aug 01 00:35:19.601931 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:35:19.601932    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="915a3a14-b2de-4ab3-ac4a-5215dcecab1f" containerName="kserve-container"
Aug 01 00:35:19.602117 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:35:19.601938    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="915a3a14-b2de-4ab3-ac4a-5215dcecab1f" containerName="kserve-container"
Aug 01 00:35:19.602117 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:35:19.601949    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c3f7ef2d-1ce3-4f69-9660-f559ac064ea7" containerName="kserve-container"
Aug 01 00:35:19.602117 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:35:19.601955    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="c3f7ef2d-1ce3-4f69-9660-f559ac064ea7" containerName="kserve-container"
Aug 01 00:35:19.602117 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:35:19.601960    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3061cbb5-a5f7-4911-8b37-d33a4dc48f7b" containerName="splitter-graph-2f013"
Aug 01 00:35:19.602117 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:35:19.601967    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="3061cbb5-a5f7-4911-8b37-d33a4dc48f7b" containerName="splitter-graph-2f013"
Aug 01 00:35:19.602117 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:35:19.602029    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="915a3a14-b2de-4ab3-ac4a-5215dcecab1f" containerName="kserve-container"
Aug 01 00:35:19.602117 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:35:19.602038    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="3061cbb5-a5f7-4911-8b37-d33a4dc48f7b" containerName="splitter-graph-2f013"
Aug 01 00:35:19.602117 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:35:19.602046    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="c3f7ef2d-1ce3-4f69-9660-f559ac064ea7" containerName="kube-rbac-proxy"
Aug 01 00:35:19.602117 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:35:19.602053    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="141f7de1-1db2-49fa-89d9-5cde41749e2b" containerName="switch-graph-bb639"
Aug 01 00:35:19.602117 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:35:19.602059    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="915a3a14-b2de-4ab3-ac4a-5215dcecab1f" containerName="kube-rbac-proxy"
Aug 01 00:35:19.602117 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:35:19.602065    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="c3f7ef2d-1ce3-4f69-9660-f559ac064ea7" containerName="kserve-container"
Aug 01 00:35:19.606536 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:35:19.606518    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-pmldc/perf-node-gather-daemonset-t6vz9"
Aug 01 00:35:19.608601 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:35:19.608579    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-pmldc\"/\"kube-root-ca.crt\""
Aug 01 00:35:19.609095 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:35:19.609075    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-pmldc\"/\"default-dockercfg-t8758\""
Aug 01 00:35:19.609219 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:35:19.609094    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-pmldc\"/\"openshift-service-ca.crt\""
Aug 01 00:35:19.613667 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:35:19.613604    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-pmldc/perf-node-gather-daemonset-t6vz9"]
Aug 01 00:35:19.673823 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:35:19.673782    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5faccf6d-3a01-4a18-a78a-bd9d7321c9d3-sys\") pod \"perf-node-gather-daemonset-t6vz9\" (UID: \"5faccf6d-3a01-4a18-a78a-bd9d7321c9d3\") " pod="openshift-must-gather-pmldc/perf-node-gather-daemonset-t6vz9"
Aug 01 00:35:19.674008 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:35:19.673840    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/5faccf6d-3a01-4a18-a78a-bd9d7321c9d3-proc\") pod \"perf-node-gather-daemonset-t6vz9\" (UID: \"5faccf6d-3a01-4a18-a78a-bd9d7321c9d3\") " pod="openshift-must-gather-pmldc/perf-node-gather-daemonset-t6vz9"
Aug 01 00:35:19.674008 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:35:19.673925    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/5faccf6d-3a01-4a18-a78a-bd9d7321c9d3-podres\") pod \"perf-node-gather-daemonset-t6vz9\" (UID: \"5faccf6d-3a01-4a18-a78a-bd9d7321c9d3\") " pod="openshift-must-gather-pmldc/perf-node-gather-daemonset-t6vz9"
Aug 01 00:35:19.674008 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:35:19.673976    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/5faccf6d-3a01-4a18-a78a-bd9d7321c9d3-lib-modules\") pod \"perf-node-gather-daemonset-t6vz9\" (UID: \"5faccf6d-3a01-4a18-a78a-bd9d7321c9d3\") " pod="openshift-must-gather-pmldc/perf-node-gather-daemonset-t6vz9"
Aug 01 00:35:19.674114 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:35:19.674033    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wxc6d\" (UniqueName: \"kubernetes.io/projected/5faccf6d-3a01-4a18-a78a-bd9d7321c9d3-kube-api-access-wxc6d\") pod \"perf-node-gather-daemonset-t6vz9\" (UID: \"5faccf6d-3a01-4a18-a78a-bd9d7321c9d3\") " pod="openshift-must-gather-pmldc/perf-node-gather-daemonset-t6vz9"
Aug 01 00:35:19.775265 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:35:19.775224    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/5faccf6d-3a01-4a18-a78a-bd9d7321c9d3-proc\") pod \"perf-node-gather-daemonset-t6vz9\" (UID: \"5faccf6d-3a01-4a18-a78a-bd9d7321c9d3\") " pod="openshift-must-gather-pmldc/perf-node-gather-daemonset-t6vz9"
Aug 01 00:35:19.775464 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:35:19.775276    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/5faccf6d-3a01-4a18-a78a-bd9d7321c9d3-podres\") pod \"perf-node-gather-daemonset-t6vz9\" (UID: \"5faccf6d-3a01-4a18-a78a-bd9d7321c9d3\") " pod="openshift-must-gather-pmldc/perf-node-gather-daemonset-t6vz9"
Aug 01 00:35:19.775464 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:35:19.775309    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/5faccf6d-3a01-4a18-a78a-bd9d7321c9d3-lib-modules\") pod \"perf-node-gather-daemonset-t6vz9\" (UID: \"5faccf6d-3a01-4a18-a78a-bd9d7321c9d3\") " pod="openshift-must-gather-pmldc/perf-node-gather-daemonset-t6vz9"
Aug 01 00:35:19.775464 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:35:19.775342    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wxc6d\" (UniqueName: \"kubernetes.io/projected/5faccf6d-3a01-4a18-a78a-bd9d7321c9d3-kube-api-access-wxc6d\") pod \"perf-node-gather-daemonset-t6vz9\" (UID: \"5faccf6d-3a01-4a18-a78a-bd9d7321c9d3\") " pod="openshift-must-gather-pmldc/perf-node-gather-daemonset-t6vz9"
Aug 01 00:35:19.775464 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:35:19.775360    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/5faccf6d-3a01-4a18-a78a-bd9d7321c9d3-proc\") pod \"perf-node-gather-daemonset-t6vz9\" (UID: \"5faccf6d-3a01-4a18-a78a-bd9d7321c9d3\") " pod="openshift-must-gather-pmldc/perf-node-gather-daemonset-t6vz9"
Aug 01 00:35:19.775464 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:35:19.775374    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5faccf6d-3a01-4a18-a78a-bd9d7321c9d3-sys\") pod \"perf-node-gather-daemonset-t6vz9\" (UID: \"5faccf6d-3a01-4a18-a78a-bd9d7321c9d3\") " pod="openshift-must-gather-pmldc/perf-node-gather-daemonset-t6vz9"
Aug 01 00:35:19.775464 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:35:19.775449    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5faccf6d-3a01-4a18-a78a-bd9d7321c9d3-sys\") pod \"perf-node-gather-daemonset-t6vz9\" (UID: \"5faccf6d-3a01-4a18-a78a-bd9d7321c9d3\") " pod="openshift-must-gather-pmldc/perf-node-gather-daemonset-t6vz9"
Aug 01 00:35:19.775763 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:35:19.775488    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/5faccf6d-3a01-4a18-a78a-bd9d7321c9d3-podres\") pod \"perf-node-gather-daemonset-t6vz9\" (UID: \"5faccf6d-3a01-4a18-a78a-bd9d7321c9d3\") " pod="openshift-must-gather-pmldc/perf-node-gather-daemonset-t6vz9"
Aug 01 00:35:19.775763 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:35:19.775531    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/5faccf6d-3a01-4a18-a78a-bd9d7321c9d3-lib-modules\") pod \"perf-node-gather-daemonset-t6vz9\" (UID: \"5faccf6d-3a01-4a18-a78a-bd9d7321c9d3\") " pod="openshift-must-gather-pmldc/perf-node-gather-daemonset-t6vz9"
Aug 01 00:35:19.783144 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:35:19.783114    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wxc6d\" (UniqueName: \"kubernetes.io/projected/5faccf6d-3a01-4a18-a78a-bd9d7321c9d3-kube-api-access-wxc6d\") pod \"perf-node-gather-daemonset-t6vz9\" (UID: \"5faccf6d-3a01-4a18-a78a-bd9d7321c9d3\") " pod="openshift-must-gather-pmldc/perf-node-gather-daemonset-t6vz9"
Aug 01 00:35:19.917613 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:35:19.917518    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-pmldc/perf-node-gather-daemonset-t6vz9"
Aug 01 00:35:20.048123 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:35:20.048093    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-pmldc/perf-node-gather-daemonset-t6vz9"]
Aug 01 00:35:20.050685 ip-10-0-136-108 kubenswrapper[2578]: W0801 00:35:20.050657    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod5faccf6d_3a01_4a18_a78a_bd9d7321c9d3.slice/crio-7842973cf878842fe71bb82b051f43fca27f4159281a8eae96b24ec4fbbe7f22 WatchSource:0}: Error finding container 7842973cf878842fe71bb82b051f43fca27f4159281a8eae96b24ec4fbbe7f22: Status 404 returned error can't find the container with id 7842973cf878842fe71bb82b051f43fca27f4159281a8eae96b24ec4fbbe7f22
Aug 01 00:35:20.238446 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:35:20.238412    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-pmldc/perf-node-gather-daemonset-t6vz9" event={"ID":"5faccf6d-3a01-4a18-a78a-bd9d7321c9d3","Type":"ContainerStarted","Data":"197a1f061a3015016d10e41a047f611f4871ddea1d44359da10be10c79db4ce7"}
Aug 01 00:35:20.238926 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:35:20.238461    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-pmldc/perf-node-gather-daemonset-t6vz9" event={"ID":"5faccf6d-3a01-4a18-a78a-bd9d7321c9d3","Type":"ContainerStarted","Data":"7842973cf878842fe71bb82b051f43fca27f4159281a8eae96b24ec4fbbe7f22"}
Aug 01 00:35:20.238926 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:35:20.238501    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-pmldc/perf-node-gather-daemonset-t6vz9"
Aug 01 00:35:20.255610 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:35:20.255554    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-pmldc/perf-node-gather-daemonset-t6vz9" podStartSLOduration=1.255536021 podStartE2EDuration="1.255536021s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-08-01 00:35:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-01 00:35:20.253746584 +0000 UTC m=+2944.221497342" watchObservedRunningTime="2026-08-01 00:35:20.255536021 +0000 UTC m=+2944.223286778"
Aug 01 00:35:20.618262 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:35:20.618152    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-cjvb8_759fe27f-5715-4033-bb09-a5f5a21d7c53/dns/0.log"
Aug 01 00:35:20.638543 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:35:20.638515    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-cjvb8_759fe27f-5715-4033-bb09-a5f5a21d7c53/kube-rbac-proxy/0.log"
Aug 01 00:35:20.726511 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:35:20.726482    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-wr2rk_67d8b75c-284a-4bdb-a560-65d472fb1515/dns-node-resolver/0.log"
Aug 01 00:35:21.245500 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:35:21.245472    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-9hxzf_51ec46bf-6903-4794-adc9-3f796e976899/node-ca/0.log"
Aug 01 00:35:22.364322 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:35:22.364288    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-7ftsc_2bcd7dcd-ab90-47bd-8747-5f1fafe9960c/serve-healthcheck-canary/0.log"
Aug 01 00:35:22.816235 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:35:22.816205    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-v2wkd_194c7e46-9a88-4078-a687-63112c8ccab5/kube-rbac-proxy/0.log"
Aug 01 00:35:22.834473 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:35:22.834451    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-v2wkd_194c7e46-9a88-4078-a687-63112c8ccab5/exporter/0.log"
Aug 01 00:35:22.853388 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:35:22.853361    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-v2wkd_194c7e46-9a88-4078-a687-63112c8ccab5/extractor/0.log"
Aug 01 00:35:25.156139 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:35:25.156108    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_kserve-controller-manager-674d959fb8-w5sb4_685ff127-eee9-4acf-bb9b-9a5607dc5d20/manager/0.log"
Aug 01 00:35:25.182660 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:35:25.182632    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_llmisvc-controller-manager-654c557856-8d8w7_74f227fb-7bdc-4402-9140-a7704e0afa63/manager/0.log"
Aug 01 00:35:25.201536 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:35:25.201509    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_model-serving-api-7b954ffc95-sgfgx_72bf9f60-5250-489f-b905-aae303ca3304/server/0.log"
Aug 01 00:35:25.296624 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:35:25.296592    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_odh-model-controller-8588677cfb-l6cv4_f7748b8b-f773-43ac-9a99-79ab1be97aef/manager/0.log"
Aug 01 00:35:26.252547 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:35:26.252517    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-pmldc/perf-node-gather-daemonset-t6vz9"
Aug 01 00:35:30.417750 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:35:30.417714    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-h8nq8_dfb5011f-b591-4203-87ce-b1c6d6ad16ce/kube-multus-additional-cni-plugins/0.log"
Aug 01 00:35:30.438454 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:35:30.438429    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-h8nq8_dfb5011f-b591-4203-87ce-b1c6d6ad16ce/egress-router-binary-copy/0.log"
Aug 01 00:35:30.462125 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:35:30.462089    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-h8nq8_dfb5011f-b591-4203-87ce-b1c6d6ad16ce/cni-plugins/0.log"
Aug 01 00:35:30.481668 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:35:30.481642    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-h8nq8_dfb5011f-b591-4203-87ce-b1c6d6ad16ce/bond-cni-plugin/0.log"
Aug 01 00:35:30.500927 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:35:30.500893    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-h8nq8_dfb5011f-b591-4203-87ce-b1c6d6ad16ce/routeoverride-cni/0.log"
Aug 01 00:35:30.522345 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:35:30.522319    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-h8nq8_dfb5011f-b591-4203-87ce-b1c6d6ad16ce/whereabouts-cni-bincopy/0.log"
Aug 01 00:35:30.542892 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:35:30.542819    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-h8nq8_dfb5011f-b591-4203-87ce-b1c6d6ad16ce/whereabouts-cni/0.log"
Aug 01 00:35:30.911081 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:35:30.911053    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-hlpzq_a82bd3b0-bcad-4511-a3c4-4e4bdd5d6798/kube-multus/0.log"
Aug 01 00:35:31.040826 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:35:31.040802    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-qs7p7_087b0a99-ae24-49e2-ba59-55651731fd60/network-metrics-daemon/0.log"
Aug 01 00:35:31.058714 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:35:31.058684    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-qs7p7_087b0a99-ae24-49e2-ba59-55651731fd60/kube-rbac-proxy/0.log"
Aug 01 00:35:32.199919 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:35:32.199880    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-m9px5_8d650e41-5af5-435a-81ab-aaf3898a490c/ovn-controller/0.log"
Aug 01 00:35:32.215725 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:35:32.215698    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-m9px5_8d650e41-5af5-435a-81ab-aaf3898a490c/ovn-acl-logging/0.log"
Aug 01 00:35:32.237830 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:35:32.237805    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-m9px5_8d650e41-5af5-435a-81ab-aaf3898a490c/kube-rbac-proxy-node/0.log"
Aug 01 00:35:32.260449 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:35:32.260421    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-m9px5_8d650e41-5af5-435a-81ab-aaf3898a490c/kube-rbac-proxy-ovn-metrics/0.log"
Aug 01 00:35:32.278953 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:35:32.278928    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-m9px5_8d650e41-5af5-435a-81ab-aaf3898a490c/northd/0.log"
Aug 01 00:35:32.298221 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:35:32.298189    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-m9px5_8d650e41-5af5-435a-81ab-aaf3898a490c/nbdb/0.log"
Aug 01 00:35:32.319396 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:35:32.319345    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-m9px5_8d650e41-5af5-435a-81ab-aaf3898a490c/sbdb/0.log"
Aug 01 00:35:32.363073 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:35:32.363038    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-m9px5_8d650e41-5af5-435a-81ab-aaf3898a490c/ovnkube-controller/0.log"
Aug 01 00:35:33.376327 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:35:33.376289    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-diagnostics_network-check-target-8d9rr_0503a101-ddee-440a-ae03-e76fe30685e7/network-check-target-container/0.log"
Aug 01 00:35:34.269745 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:35:34.269717    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-operator_iptables-alerter-xssj7_fe1c93d5-e51c-4bea-b395-b782a588e7e0/iptables-alerter/0.log"
Aug 01 00:35:34.885567 ip-10-0-136-108 kubenswrapper[2578]: I0801 00:35:34.885540    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-node-tuning-operator_tuned-p7bwg_93fe11db-24ef-40d3-a042-7be6f62bd6f6/tuned/0.log"